Toga: final releases....

[Shaun]

Toga Testing and Final releases.


I have been testing Toga for Thomas and now that he has finished developing Toga I have been testing the various version and also trying modified settings before releasing the best versions.


Toga II Match results (CCRL time controls)

40/4

Code: Select all

        Engine                  ELO   +    -   Score  AvOp  Games
Toga II 1.3 Beta1 HT75 TEHPon   2888  +15  -15  55.5%  -40.7  1599

Toga II 1.3x4 HT80              2874  +19  -19  49.9%   +0.3   955
Toga II 1.3x4                   2870  +14  -14  51.0%   -6.9  1780
Toga II 1.3x1                   2869  +10  -10  47.9%  +12.7  3377
Toga II 1.3 Beta1               2868  +10  -10  48.9%   +6.5  3968
Toga II 1.3x4 egbb              2867  +20  -20  56.9%  -48.7   870
Toga II 1.3 Beta3b              2865  +11  -11  46.9%  +21.6  2856
Toga II 1.3x4b egbb HT80        2864  +23  -23  49.4%   +6.4   638
Toga II 1.2.1a                  2860   +8   -8  48.1%   +9.5  5987
Toga II 1.3x4 TEHPoff           2860  +21  -21  56.1%  -40.5   799
Toga II 1.3x4b egbb             2855  +17  -17  52.9%  -21.5  1201
Toga II 1.2 Beta3a              2854  +13  -13  45.7%  +30.7  2275
Toga II 1.3 Beta1 HT75          2853  +19  -19  50.5%   -2.9   989
Fruit 2.2.1                     2850  +18  -18  56.7%  -47.4  1031
Toga II 1.2                     2850  +24  -24  57.9%  -57.8   618
Toga II 1.3x4 HT70              2849  +21  -21  47.7%  +13.5   785
Toga II 1.2 Beta3c              2847  +28  -28  57.7%  -54.4   447
Toga II 1.2 Beta3b              2845  +28  -28  57.0%  -52.2   449
Toga II 1.2 Beta3               2839  +28  -28  56.6%  -46.0   448
Toga II 1.1a                    2824  +33  -32  73.5% -177.0   375

Comments

Toga II 1.3 Beta1 HT75 TEHPon outperforms the default settings and seems nearly 30 elo stronger than 1.2.1a.

For Blitz the EGBBs seem to hurt performance and the EGBB fix in 1.3x4b seems to have hurt performance slightly more.

A higher HT value (more pruning) seems relatively better at blitz compared to slower time controls.

I want to test 1.3 Beta1 TEHPon at some point for 40/4


40/12

Code: Select all

            Engine               ELO   +    -   Score  AvOp  Games
Toga II 1.3 Beta1                2857  +14  -14  54.8%  -35.5  1701
Toga II 1.3x4b egbb HT70 TEHPoff 2856  +17  -17  50.8%   -4.8  1187

Toga II 1.3x4 HT70               2849  +20  -20  59.1%  -66.1   845
Toga II 1.3x4                    2844  +20  -20  58.2%  -61.7   825
Toga II 1.3x1                    2839  +23  -23  51.8%  -13.0   576
Toga II 1.2.1a                   2837  +18  -18  56.7%  -45.6  1064
Toga II 1.3 Beta3b               2835  +19  -19  56.3%  -43.9   903
Toga II 1.3x4 HT80               2833  +20  -20  57.0%  -50.2   827
Toga II 1.2 Beta3                2826  +27  -27  58.5%  -56.7   451
Toga II 1.2 Beta3a               2825  +20  -20  56.1%  -42.8   841
Fruit 2.2.1                      2823  +26  -26  49.1%   +4.7   476
Toga II 1.2 32-bit               2812  +27  -27  56.2%  -42.6   446
Toga II 1.2 Beta3b               2807  +26  -26  53.6%  -26.7   480
Toga II 1.2 Beta3c               2804  +27  -27  54.9%  -34.5   448

Comments

Toga 1.3 Beta 1 and 1.3x4b HT70 TEHPoff all but tie and are both about 20 elo stronger than 1.2.1a

Turning off TEHP seems to benefit - the reverse from the Blitz results.

At some point I will test Toga II 1.3 Beta1 HT75 TEHPon.



40/40

Code: Select all

              Engine                 ELO   +    -   Score  AvOp  Games
Toga II 1.3 Beta 1                   2916  +31  -30  62.9%  -88.6   352

Toga II 1.3 Beta3b                   2894  +33  -33  57.6%  -49.9   297
Toga II 1.3x4b HT70 TEHPoff          2893  +98  -96  55.0%  -28.3    30
Toga II 1.2.1a                       2862  +11  -11  49.3%   +2.0  2805
Toga II 1.2                          2847  +31  -31  56.8%  -46.9   339
Fruit 2.2.1                          2840  +13  -13  52.3%  -15.3  2068
Toga II 1.1a                         2825  +18  -18  55.2%  -33.7  1014
Fruit 2.1                            2793  +38  -37  59.2%  -70.1   233

Coments

Toga 1.3 Beta 1 seems to be about 50 elo stronger than 1.2.1a

More games will be played for both Toga II 1.3 Beta 1 and Toga II 1.3x4b HT70 TEHPoff



Toga II endgame test suite results.

These are with the Eigenmann_Endspiel_Test 100 positions.


P4 2.6ghz 768mb - All tests 256mb Hash


Toga II 1.3x4b

Code: Select all

       without  with 5egbbs    with 5egbbs    with 5egbbs
                (16mb cache)   (32mb cache)  (256mb cache)
30sec   38/100     37/100         37/100          36/100
90sec   52/100     57/100         57/100          56/100 
5min    70/100     71/100         71/100          71/100
15min   76/100     80/100         80/100          80/100
60min   83/100     85/100                         85/100

Toga II 1.3x4b (settings)

with 5egbbs 16mb cache

Code: Select all

                             HT70       HT70
        HT80       HT70     TEHPoff   (512mb) 
30sec   35/100    38/100    34/100     38/100
90sec   58/100    57/100    54/100     56/100
5min    74/100    74/100    75/100     76/100
15min   81/100    81/100    78/100     82/100
60min   84/100    86/100    85/100     89/100

Toga II 1.3x4

Code: Select all

       without  with 5egbbs(16mb cache)
30sec   39/100     38/100
90sec   52/100     52/100
5min    72/100     72/100
15min   76/100     76/100
60min   82/100     85/100

Toga II 1.3beta1 (settings)

Code: Select all

                    HT75
       default     TEHPon
30sec   32/100     33/100
90sec   50/100     49/100
5min    59/100     66/100
15min   71/100     74/100
60min   79/100     80/100

Comments

Even when compared to Toga 1.3x4/1.3x4b without egbbs 1.3beta1 does not perform as well in end game positions!

The tests show that 1.3x4b with egbbs ouperforms 1.3x4 suggesting that the fix works as expected*

*1.3x4 and 1.3x4b with egbbs given 60 minutes per possition scored identically however 1.3x4b found the possitions earlier. I double checked as I was surprised at the results.



I plan to release Toga II 1.3beta1 renamed Toga II 1.3.1 (for blitz adjust the settings as above) and Toga II 1.3x4b as Toga II 1.3.4 with the default settings adjusted to HT70 TEHPoff as my priority is slower time controls.


One thing that we can see from the results above is that settings at blitz are not always correct for longer time controls and I would be interested if more pruning benefited other engines at longer time controls - somthing to try..

I also wonder if knowlege either more/less would give different results at different time controls.


I am looking for someone to compile* the final versions and also to host the packages - any offers to shaun.br3w3r(at)totalis3.co.uk (replace (at) with @ and 3's to e's).

*I had planned to do this but I have not even had time to install a compiler yet - I am not a C developer so I don't have one installed already.

Thanks again to Thomas for his development of Fruit, to Fabien for Fruit in the first place and anybody else who has helped either of them!!!

Shaun Brewer


P.S. If you can't wait to Toga to be re-compiled/hosted somewhere drop me an email at the address above and I will forward you the current packages.


P.P.S If anyone wants to carry on the Toga development I will always make time for testing!

[Henrik Dinesen]

Thx Shaun,

Seems that final conclusions is somewhat limited, though the testing is more throughout than we see most of the time.

Regarding compilation and maybe hosting, is Bryan Hofmann not interested?

[Peter Skinner]

I can compile it and host if Shawn likes.

[Shaun]

Thx Shaun,

Seems that final conclusions is somewhat limited, though the testing is more throughout than we see most of the time.

Regarding compilation and maybe hosting, is Bryan Hofmann not interested?

Hi Henrik,

Its been fun!!!

Yes conclusions are not easy in computer chess unless you play thousands and thousands of games or the strength increase is dramatic.

However at 40/4

Toga II 1.3 Beta1 HT75 TEHPon 32-bit is stronger than 1.2.1a 100% according to Bayeselo

at 40/12

Toga II 1.3 Beta1 is stronger 95.5%

at 40/40

Toga II 1.3 Beta1 is stronger 99.9%

All the best

Shaun

[Shaun]

I can compile it and host if Shawn likes.

Hi Peter,

I will drop you a mail tonight

Thanks!!!

Shaun

P.S. Can you drop me an email - also just to let you know the link to your crafty build site in your sig seems broken - I tried to look there for an email address

[Peter Skinner]

The link to my Crafty site is:

http://www.webkikr.net

I am getting rid of all the .info domains I purchased. They simply got to much spam or were labeled as spam on other servers and my mail never got through.

I will send you an email though.

[ernest]

Comments

Even when compared to Toga 1.3x4/1.3x4b without egbbs 1.3beta1 does not perform as well in end game positions!

The tests show that 1.3x4b with egbbs ouperforms 1.3x4 suggesting that the fix works as expected*

*1.3x4 and 1.3x4b with egbbs given 60 minutes per possition scored identically however 1.3x4b found the possitions earlier. I double checked as I was surprised at the results.

Hi Shaun,
I did not know that there was a fix to the egbb problem of 1.3x4. Who did that? Can you give a position where 1.3x4 has the problem and 1.3x4b has not?
Other question: are the 1.3x4 versions the only 1.3 versions which work with egbb?

[Shaun]

Comments

Even when compared to Toga 1.3x4/1.3x4b without egbbs 1.3beta1 does not perform as well in end game positions!

The tests show that 1.3x4b with egbbs ouperforms 1.3x4 suggesting that the fix works as expected*

*1.3x4 and 1.3x4b with egbbs given 60 minutes per possition scored identically however 1.3x4b found the possitions earlier. I double checked as I was surprised at the results.

Hi Shaun,
I did not know that there was a fix to the egbb problem of 1.3x4. Who did that? Can you give a position where 1.3x4 has the problem and 1.3x4b has not?
Other question: are the 1.3x4 versions the only 1.3 versions which work with egbb?

Hi Ernest,

the fix was coded by Thomas with help from Daniel.

I do not have example possitions where 1.3x4 fails but in my end game tests 1.3x4b outperformed 1.3x4.

1.3x4 was the first version to support EGBBs

Shaun

[Werner]

Hi Shaun,
where, when ?

[Shaun]

I have my first windows compiles and will validate them tomorrow.

Shaun