/MT v. /MD: A quick Firenzina compile test

Discussion of computer chess matches and engine tournaments.

Moderator: Ras

Gusev
Posts: 1476
Joined: Mon Jan 28, 2013 2:51 pm

/MT v. /MD: A quick Firenzina compile test

Post by Gusev »

The /MT compile of Firenzina 2.2.2 xTreme that I released yesterday (03/01/2013) at http://dgusev.cs.edinboro.edu/GameDev/C ... nzina.html tied a 500-game 1/1 blitz match with the initial /MD compile released (02/25/2013), +75-75=350.
Engine Score Fi Fi S-B
1: Firenzina_2-2-2_xTreme_popcnt_sse4-2_intel13_md_x64 250.0/500 ··················································································································································································································
··················································································································································································································
················································································
=110==0=====1==11==0=101===1=====0===011===========0=======1=11====000==0=0=====1====01==0=01=====10
====11101=1======0=======0===0=100===0=0======1=====1======10=1=====1==1=00==00===0==0===0=======0==
1=1=====101=====1=11=0====1====0==================1=====0====11=======0=0=0==10=1====0=======0=010=
0=0=10=1====1=101=01=1===1011====10==0====1=1====1=0====0100========1=====1===0=====0=====01=========
1===101==0====10========1010=====0===0=======00=====10=0=0=========0====111=00======1=========11==1= 62500.
1: Firenzina_2-2-2_xTreme_popcnt_sse4-2_intel13_mt_x64 250.0/500 =001==1=====0==00==1=010===0=====1===100===========1=======0=00====111==1=1=====0====10==1=10=====01
====00010=0======1=======1===1=011===1=1======0=====0======01=0=====0==0=11==11===1==1===1=======1==
0=0=====010=====0=00=1====0====1==================0=====1====00=======1=1=1==01=0====1=======1=101=
1=1=01=0====0=010=10=0===0100====01==1====0=0====0=1====1011========0=====0===1=====1=====10=========
0===010==1====01========0101=====1===1=======11=====01=1=1=========1====000=11======0=========00==0= ··················································································································································································································
··················································································································································································································
················································································ 62500.

500 games played / Tournament is finished
Name of the tournament: FirenzinaMatch2MDvMT_KLO250_1-1
Site/ Country: DMITRI-ASUS, United States
Level: Blitz 1/1
Hardware: Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz with 5.9 GB Memory
Operating system: Windows 7 Professional Professional Service Pack 1 (Build 7601) 64 bit

Some drawish result WAS expected, I am just shocked to see such round numbers! The test suite was KLO250. Now I am ready to try more suggested compiler options in addition to the /MT option. (/GS- was already in, and most runtime checks were off all along.)
Gusev
Posts: 1476
Joined: Mon Jan 28, 2013 2:51 pm

/MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile test

Post by Gusev »

I have completed a 200-game 3/3 blitz run of the newest optimized MT_SGVAD compile of Firenzina 2.2.2 xTreme, a clone of Fire 2.2 xTreme by Kranium (Norman Schmidt), against the released MT compile. The test suite was KLO100. SGVAD stands for Schmidt-Gusev-Velasco-Ablett-Dart. It's an abbreviation of last names of those involved in coming up with this set of parameters.

The MT_SGVAD beat MT, 103.5:96.5 (+27-20=153). According to the online ELO calculator, this represents a gain of slightly more than 30 ELO points. We may be near the ceiling of what can be achieved by playing with compiler settings. At this point, I intend to release this baby and proceed with making and testing generic x64 and x32 compiles for Windows.
Tennison
Posts: 183
Joined: Sat Nov 26, 2011 2:02 pm

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by Tennison »

103.5/200 : this is not a +30 but a +12 ...

And with only 200 games you are far from being sure it's a plus.

I have made, with Toga, lot of tests were after 1000 games it was a +20 but after 5000 games it was a +2 or even a -10.

You must test with more games (maybe shorter time control like 1'+1").
Gusev
Posts: 1476
Joined: Mon Jan 28, 2013 2:51 pm

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by Gusev »

103.5/200 : this is not a +30 but a +12 ...
Can you recommend a good ELO calculator?

In this case, the new compile cannot be worse than the old one, because (a) kN/sec are visibly higher, (b) the code didn't change, and (c) Four pros (Schmidt, Velasco, Ablett, and Dart) gave me advice on how to compile Firenzina better. I greatly appreciate that!
gladius
Posts: 568
Joined: Tue Dec 12, 2006 10:10 am
Full name: Gary Linscott

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by gladius »

Gusev wrote:
103.5/200 : this is not a +30 but a +12 ...
Can you recommend a good ELO calculator?

In this case, the new compile cannot be worse than the old one, because (a) kN/sec are visibly higher, (b) the code didn't change, and (c) Four pros (Schmidt, Velasco, Ablett, and Dart) gave me advice on how to compile Firenzina better. I greatly appreciate that!
I'm a bit biased, but try: http://forwardcoding.com/projects/ajaxchess/rating.html.
Gusev
Posts: 1476
Joined: Mon Jan 28, 2013 2:51 pm

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by Gusev »

This one gives +12, and so does this other one that I just found: http://www.3dkingdoms.com/chess/elo.htm
I was looking at the Elo rating change (starting at 3120 for both compiles and setting K=10), not for Elo difference. I used this calculator: http://www.kosteniuk.com/EloCalc/elo.php However, here also, if I plug in 3132 and 3120, then it expects 103.4 points out of 200 for the stronger player.
Tennison
Posts: 183
Joined: Sat Nov 26, 2011 2:02 pm

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by Tennison »

Here is a very useful elocalc : Chess Elo Rating Difference Calculator.

Normally more nps seems to be a good sign.
But sometimes some "optimizations" can let the engine to play less good.

To be sure, play more and more games (with different time controls).

;-)
Gusev
Posts: 1476
Joined: Mon Jan 28, 2013 2:51 pm

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by Gusev »

To be sure, play more and more games (with different time controls).
Will do, after I take care of more generic compiles. Thank you for the correction!
User avatar
geots
Posts: 4790
Joined: Sat Mar 11, 2006 12:42 am

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by geots »

Gusev wrote:I have completed a 200-game 3/3 blitz run of the newest optimized MT_SGVAD compile of Firenzina 2.2.2 xTreme, a clone of Fire 2.2 xTreme by Kranium (Norman Schmidt), against the released MT compile. The test suite was KLO100. SGVAD stands for Schmidt-Gusev-Velasco-Ablett-Dart. It's an abbreviation of last names of those involved in coming up with this set of parameters.

The MT_SGVAD beat MT, 103.5:96.5 (+27-20=153). According to the online ELO calculator, this represents a gain of slightly more than 30 ELO points. We may be near the ceiling of what can be achieved by playing with compiler settings. At this point, I intend to release this baby and proceed with making and testing generic x64 and x32 compiles for Windows.









Dmitri, if it were me I would do it the easy way. You have 2 %s: .5175 and .4825. Put your system's calculator on "scientific" and then do this:

.5175 / .4825 = log * 400 = 12, with 12 being the elo difference.


There are a number of ways- this is just the simplest for me.



george
jd1
Posts: 269
Joined: Wed Oct 24, 2012 2:07 am

Re: /MT_SGVAD v. /MT: Another quick Firenzina 2.2.2 compile

Post by jd1 »

Gusev wrote: In this case, the new compile cannot be worse than the old one, because (a) kN/sec are visibly higher

If the new compile's search tree is identical to the original Fire 2.2 compile (i.e. node count, pv, selective depth, root move ordering etc. - all of which may be seen from UCI output - are always identical), then the elo gain is most easily and accurately measured by taking the percentage difference in speed (kN/s).

If you run a testset of positions, it should be possible to get the approximate speed increase and hence the elo gain.

Jerry