Running a new testing base

[MikeB]

Just a big old fashioned huge round robin tc 10/.17 this is phase 1 in process

Ponder off , single cpu more details later

Code: Select all

Current time : 12:31:28
   1 Stockfish 120316 64 POPC  3390   24   24   748   84%  3076   23% 
   2 Stockfish 7 64 POPCNT     3356   23   23   753   80%  3079   25% 
   3 Komodo 9.42 64-bit        3337   23   23   758   78%  3080   21% 
   4 Komodo 9.3 64-bit         3320   22   22   747   77%  3082   24% 
   5 Stockfish  160226 64bit   3100   20   20   747   50%  3100   26% 
   6 Senpai 1.0                3075   20   20   748   47%  3102   26% 
   7 Texel 1.05 64-bit         3022   20   20   748   41%  3107   25% 
   8 Hakkapeliitta 3.0         3005   21   21   748   39%  3108   22% 
   9 Crafty-25.1b              2971   21   21   758   34%  3111   26% 
  10 Deep HIARCS 14 WCSC       2966   21   21   748   34%  3111   23% 
  11 Crafty-25.1bT11           2964   21   21   758   34%  3111   24% 
  12 Deep Shredder 12 x64      2918   22   22   747   29%  3115   21% 
  13 Arasan 18.3               2877   23   23   748   24%  3119   19% 

[MikeB]

Just a big old fashioned huge round robin tc 10/.17 this is phase 1 of 10 n process.

Ponder off , single cpu more details later

Code: Select all

Current time : 12:31:28
   1 Stockfish 120316 64 POPC  3390   24   24   748   84%  3076   23% 
   2 Stockfish 7 64 POPCNT     3356   23   23   753   80%  3079   25% 
   3 Komodo 9.42 64-bit        3337   23   23   758   78%  3080   21% 
   4 Komodo 9.3 64-bit         3320   22   22   747   77%  3082   24% 
   5 Stockfish  160226 64bit   3100   20   20   747   50%  3100   26% 
   6 Senpai 1.0                3075   20   20   748   47%  3102   26% 
   7 Texel 1.05 64-bit         3022   20   20   748   41%  3107   25% 
   8 Hakkapeliitta 3.0         3005   21   21   748   39%  3108   22% 
   9 Crafty-25.1b              2971   21   21   758   34%  3111   26% 
  10 Deep HIARCS 14 WCSC       2966   21   21   748   34%  3111   23% 
  11 Crafty-25.1bT11           2964   21   21   758   34%  3111   24% 
  12 Deep Shredder 12 x64      2918   22   22   747   29%  3115   21% 
  13 Arasan 18.3               2877   23   23   748   24%  3119   19% 

Realtime updates every 5 minutes are here.

The endgame will be a 5000 game RR with all 13 engines - to use as baseline for future crafty engine testing. The one of a kind opening book, was crafted from CCLR 40/40 games where all opponents were rated over 3200, game did not end in draw and the opening score was between 45 to 65 centipawns. Each and every one of these 2500 opening positions is played twice with alternating colors. No EGTB, no pondering , standard 256M hash.
Each phase covers 250 positions, 500 games, each engine will have played 6000 games at the end of phase I, 60,000 games at the end of phrase 10. Will take about two weeks to complete, check back often.

Code: Select all

System:
  System Version:	OS X 10.11.4 (15E65)
  Kernel Version:	Darwin 15.4.0
  Model Name:	Mac Pro
  Model Identifier:	"Mid 2010"
  Processor Name:	6-Core Intel Xeon
  Processor Speed:	3.46 GHz
  Number of Processors:	2
  Total Number of Cores:	12
  L2 Cache (per Core):	256 KB
  L3 Cache (per Processor):	12 MB
  Memory:	48 GB
  Video Card: AMD Radeon HD 7970
  Monitor: HP 27xw:1920 x 1080 @ 60Hz (1080p)	
  Processor Interconnect Speed:	6.4 GT/s
  Storage: 1.2 TB SSD; 3TB SATA
  SF Benchmark using 24 Threads : ~ 25M+/nps
  SF Benchmark using 1 CPU : ~ 2M/nps
  Crafty Benchmark using 24 Threads :  60M/nps
  Crafty Benchmark using 1 CPU: 6M/nps
  Cost (Used Aug 2015)) =$2199

[jdart]

You need more cores. I can do 60,000 hyper-bullet games in about 8 hours.

--Jon

[MikeB]

You need more cores. I can do 60,000 hyper-bullet games in about 8 hours.

--Jon

Cool - can I quote you when I explained to my wife why we now have 3 Mac Pros? How many cores you have and what time control you use? When each engines plays 60,000, with 13 engines, of course that is 390.000 games to be played in total and the Mac Pro can handles about 30,000+/24 hours - so it should be completed 13 days. Your setup would handled that in about in about 2 days - 6x faster. Nice setup.

Edit: Math correction

[jdart]

1 dual Opteron (24 cores)
1 dual Xeon 2670 (16 cores)
1 dual Xeon 5650 (12 cores)
1 dual Xeon 5620 (8 cores)

All the Xeon systems were bought used on eBay. The Opteron system I built a couple years ago.

60 cores total. Time control is 0:04+0.1.

--Jon

[MikeB]

1 dual Opteron (24 cores)
1 dual Xeon 2670 (16 cores)
1 dual Xeon 5650 (12 cores)
1 dual Xeon 5620 (8 cores)

All the Xeon systems were bought used on eBay. The Opteron system I built a couple years ago.

60 cores total. Time control is 0:04+0.1.

--Jon

We're thinking about going to shorter micro tc. Do you test Arasan vs Arasan or do you Aarsan vs stable of engines? No ill effects going with tc 4/.1? Assuming you do have your own engine communication script/code?

[jdart]

I use 5 opponents. I have tried a few time control variants but the one I am using is pretty fast and reliable (no engines have losses on time). But not all engines are reliable at these very long matches/fast time control.

I use cutechess for matches. My software setup is briefly described here: http://arasanchess.org/blogs/jun15.html (but the description of the tuner is out of date).

--Jon

[MikeB]

Phase 2A has started, a 7800 game run that will take about 6 hours - same link as before for realtime updates. Also added a link that will just show the Phase 2A results.

ALL Realtime updates every 5 minutes.

Phase 2A Only - Realtime updates every 5 minutes.

[MikeB]

Phase 2B will start soon, a 15600 game run that will take about 15 hours - same link as before for realtime updates. Also included a link that will just show the Phase 2B results.

ALL Games Realtime updates every 5 minutes

Phase 2B Games Only - Realtime updates every 5 minutes

It should start within the next 15 minutes - check back often.

[MikeB]

Phase 2C will start soon, another 15600 game run that will take about 15 hours - same link as before for realtime updates. Also included a link that will just show the Phase 2B results.

ALL Games Realtime updates every 5 minutes

Phase 2C Games Only - Realtime updates every 5 minutes

It should start within the next 15 minutes - check back often.