AlphaZero Chess is not that strong ...

[Laskos]

54MB is the size of the tree, and would only be 40% hashful if all nodes of the tree were different. Which would imply the hash table is a write-only data sink, never used for anything other than burning a few memory cycles. More typically a tree size of 54MB would mean 10% hashful.

From what I can see, hashfull reported for this SF on 1 thread on my PC increases by 16-17 MB per second on late opening positions, at about 1.8M NPS. At least in the several starting seconds.

[Laskos]

54MB is the size of the tree, and would only be 40% hashful if all nodes of the tree were different. Which would imply the hash table is a write-only data sink, never used for anything other than burning a few memory cycles. More typically a tree size of 54MB would mean 10% hashful.

From what I can see, hashfull reported for this SF on 1 thread on my PC increases by 16-17 MB per second on late opening positions, at about 1.8M NPS. At least in the several starting seconds.

It seems that the effect of hash tables is much higher than I expected. 1 thread of SF dev, 3 seconds per move, with about 2.0M NPS, about 50 MB hash filled, so for 1 MB hash, and overload of 50, for 128 MB hash 40% hashfull - optimal.

After more than 300 games, the effect seems impressive:

Code: Select all

Games Completed = 312 of 1000 (Avg game length = 316.077 sec)
Settings = Gauntlet/0MB/3000ms per move/M 400cp for 3 moves, D 70 moves/EPD:C:\LittleBlitzer\2moves_v1.epd(32000)
Time = 24775 sec elapsed, 54632 sec remaining
 1.  SF 1 MB                  	136.5/312	19-58-235  	(L: m=0 t=0 i=0 a=58)	(D: r=145 i=7 f=0 s=0 a=83)	(tpm=3006.2 d=26.18 nps=1966384)
 2.  SF 128 MB                	175.5/312	58-19-235  	(L: m=0 t=0 i=0 a=19)	(D: r=145 i=7 f=0 s=0 a=83)	(tpm=3006.6 d=27.27 nps=2023613)

44 Elo points at 75% draw rate is an important difference, win/loss ratio is about 3. Depth with optimal hash is about 1 ply larger. It's much more than I expected. I will leave over New Year's Night to finish the test to 1000 games.

[M ANSARI]

The biggest proof that A0 is very strong is in the move choices that other engines seem to think are not good, but turn out to be very deep. Remember that SF is an extremely fine tuned engine while A0 is in the infancy stage and most likely will get much stronger.

It would be nice to have an Open match with each opposing team running its own version. No excuses just a 20 game match with opening books allowed and EGTB's allowed and a Classic LTC. I would think that A0 would be just as impressive, although SF might get more draws. I wish we could see all the drawn games that were played. There might be some very valuable information there. I would like to know if SF ever had an edge in those games.

[Jesse Gersenson]

Based on the Komodo recommendation of a 40% fill rate per move, I estimate about 128 GB is about right assuming a reasonably fast 64 core machine.

So, if someone has a 64 core machine (or even 32 core), we could try this:

a. machine 1 uses1 GB Hash, Stockfish 8, no opening book. No Syzygy.
b. machine 2 uses the latest Stockfish, 64 GB or 128 GB of hash, with Syzygy. And a decent opening book.

It is unclear how they kept the programs from constantly playing the first few moves the same. We could use some very small opening book for the run.

Just use fixed time of 60 seconds per move. Assuming a typical games lasts 60 moves, then it would take 2 hours per games, or 200 hours total. I am interested in this enough to donate some to cover machine rental on Amazon EC2 or some suitable machine.

Mark

A good task for a computer science grad student.

Someone post a link to a suitable opening book. With that, I will ask someone running a CS department, and with an interest in computer chess, to run the test.

[Jesse Gersenson]

I've got someone interested in running the test. I need a suitable opening book.

Suggestions?

[Adam Hair]

I've got someone interested in running the test. I need a suitable opening book.

Suggestions?

Here are the Polyglot books bundled with Scid vs PC:

http://www.mediafire.com/file/2z80v280gbbm6dq/books.7z

There are probably better books out there for this test, but they tend to be private or in Chessbase format.

If a short book is needed for handicapped Stockfish in order to avoid the same opening moves from being played, here is a 1 move Polyglot book I created from Norm Pollock's GM pgns using only openings found 100 times or more in the pgns:

http://www.mediafire.com/file/3bbka50iy ... _1move.bin

[Ovyron]

I've got someone interested in running the test. I need a suitable opening book.

Suggestions?

I don't think there's anything stronger yet than the branch Brainfish with the Cerebellum Light Library:

http://www.zipproth.de/ (click on Brainfish, then Brainfish for Windows)

I matched it up against all the free Bin books I could find, it convincingly destroyed them.

[Vinvin]

I've got someone interested in running the test. I need a suitable opening book.

Suggestions?

Very good !
Brainfish and the Cerebellum book is the best opening system. May be it's already too good for the test (+45 for fast games) : http://www.sp-cc.de/files/stockfishaktuell.jpg
I was looking for books from Sedat Canbaz but all his files disappeared : http://www.talkchess.com/forum/viewtopic.php?t=54634

[Guenther]

I've got someone interested in running the test. I need a suitable opening book.

Suggestions?

Very good ! Brainfish and the Cerebellum book is the best opening system.
May be it's already too good for the test (+45 for fast games) : http://www.sp-cc.de/files/stockfishaktuell.jpg
I was looking for books from Sedat Canbaz but all his files disappeared : http://www.talkchess.com/forum/viewtopic.php?t=54634

Of course asmfish + cerebellum is a bit stronger (note that cerebellum light is released as a plain polyglot book meanwhile), OTH asmfish wasn't updated for a while though.

[Ovyron]

Of course asmfish + cerebellum is a bit stronger (note that cerebellum light is released as a plain polyglot book meanwhile)

I tried the polyglot version of Cerebellum, unfortunately, in some openings it just goes for three-fold repetition.