TalkChess.com

I organized a big tournament with 33 engines, including some derivatives of Stockfish and some networks for Leela.
(2m+2s permanent-brain on - AMD-Ryzen7-2700X & RTX2080 - 5-moves book)

The Ranking



I evaluated the whole tournament chess-wise and summarized it in a report (with download possibility of all PGN-files):
https://glarean-magazin.ch/2019/08/24/s ... ogramming/

(You can use the "Translate" button at the bottom left for an automatic english translation of the report).

One of the most interesting positions from these tournament games is this one here:

[d]r3r1k1/3b1ppp/8/1p1p4/nPpP1BP1/4P2P/P4P2/1NQ3K1 b – – 0 24
Here Leela played 24... c3!

An amazing move that the Alpha Beta engines don't see...

Greetings: Walter

I’m particularly impressed by the performance of Stoofvlees and Scorpio in your tournament.

Alpha beta seems a bit in a dead end street,

Glarean wrote: ↑Sat Aug 24, 2019 6:17 pm I organized a big tournament with 33 engines, including some derivatives of Stockfish and some networks for Leela.
(2m+2s permanent-brain on - AMD-Ryzen7-2700X & RTX2080 - 5-moves book)

The Ranking



I evaluated the whole tournament chess-wise and summarized it in a report (with download possibility of all PGN-files):
https://glarean-magazin.ch/2019/08/24/s ... ogramming/

(You can use the "Translate" button at the bottom left for an automatic english translation of the report).

One of the most interesting positions from these tournament games is this one here:

[d]r3r1k1/3b1ppp/8/1p1p4/nPpP1BP1/4P2P/P4P2/1NQ3K1 b – – 0 24
Here Leela played 24... c3!

An amazing move that the Alpha Beta engines don't see...

Greetings: Walter

Stockfish can find the move!

New game Line
r3r1k1/3b1ppp/8/1p1p4/nPpP1BP1/4P2P/P4P2/1NQ3K1 b - - 0 1

Analysis by Stockfish 230819 64 POPCNT:

1...c3 2.Na3 Rec8 3.Nc2 Nb2 4.a3 h5 5.f3 hxg4 6.hxg4 Nc4 7.Kf2 f6 8.Qd1 g5 9.Bg3 Be8 10.f4 Nd2 11.Kg2 Ne4 12.Qb1 Kg7 13.Qa1 Bd7 14.f5 Rh8 15.Ne1 Ra7 16.Nc2 Rha8 17.Qc1 Rc8 18.Qf1 Rg8 19.Qa1 Rh8 20.Ne1 Kf7 21.Nc2 Bc6 22.Ne1 Kg8 23.Nd3 Rah7 24.Nf2 c2 25.Qc1
Black is clearly better: -+ (-2.07) Depth: 36/82 00:02:18 4308MN, tb=18119
(, 24.08.2019)

>Stockfish can find the move!
>Analysis by Stockfish 230819 64 POPCNT:
>1...c3 2.Na3 Rec8 3.Nc2 Nb2 4.a3 h5 5.f3 hxg4 6.hxg4 Nc4 7.Kf2 f6 8.Qd1 g5 9.Bg3 Be8 10.f4 Nd2 11.Kg2 Ne4 12.Qb1 Kg7 13.Qa1 Bd7 14.f5 Rh8 15.Ne1 Ra7 16.Nc2 Rha8 17.Qc1 Rc8 18.Qf1 Rg8 19.Qa1 Rh8 20.Ne1 Kf7 21.Nc2 Bc6 22.Ne1 Kg8 23.Nd3 Rah7 24.Nf2 c2 25.Qc1
>Black is clearly better: -+ (-2.07) Depth: 36/82 00:02:18 4308MN, tb=18119

Yeah, of course - but only after 2 minutes. Too late!

See Leela (0 sec!):

Analysis by Lc0 0.22 (NW42595):

24...c3 25.Dc2 Ta6 26.Kg2 h6 27.Kg3 Tc8 28.f3 Tc4 29.Sa3 Txb4 30.Dd3 Ta8 31.Sc2 Tb1 32.e4 Tc8 33.exd5 Sb6 34.d6 Sd5 35.De4 Sxf4 36.Kxf4 Tb2 37.a3 Te8 38.Dd3 Tc8
Schwarz steht deutlich besser: -/+ (-1.42) Tiefe: 6/19 00:00:00
Schwarz steht deutlich besser: -+ (-1.65) Tiefe: 14/40 00:00:55 1929kN

Leela doesn't calculate, Leela knows...

Greetings: Walter

.

dkappe wrote: ↑Sat Aug 24, 2019 6:42 pm I’m particularly impressed by the performance of Stoofvlees and Scorpio in your tournament.

You're right, of course there are about 100 other interesting engines...

mclane wrote: ↑Sat Aug 24, 2019 6:47 pm Alpha beta seems a bit in a dead end street,

I also believe that the future belongs to artificial intelligence, also in chess!
But this can take quite a long time...

im completely unsure though wether 1...Ra6/Rec8 (maybe 1...Re6 as well and even 1...h5 possibly) are really worse here...at least for Ra6/Rec8 the h5 idea (in mwyoung's SF-line) seems to work as well after a really quick overview from my side with bad hardware in use...maybe a SF-multiPV=5 search on strong hardware with decent amount of time could bring clearence...from my pov 1...c3 seems clearest as well though at first sight !?

Glarean wrote: ↑Sat Aug 24, 2019 8:19 pm >Stockfish can find the move!
>Analysis by Stockfish 230819 64 POPCNT:
>1...c3 2.Na3 Rec8 3.Nc2 Nb2 4.a3 h5 5.f3 hxg4 6.hxg4 Nc4 7.Kf2 f6 8.Qd1 g5 9.Bg3 Be8 10.f4 Nd2 11.Kg2 Ne4 12.Qb1 Kg7 13.Qa1 Bd7 14.f5 Rh8 15.Ne1 Ra7 16.Nc2 Rha8 17.Qc1 Rc8 18.Qf1 Rg8 19.Qa1 Rh8 20.Ne1 Kf7 21.Nc2 Bc6 22.Ne1 Kg8 23.Nd3 Rah7 24.Nf2 c2 25.Qc1
>Black is clearly better: -+ (-2.07) Depth: 36/82 00:02:18 4308MN, tb=18119

Yeah, of course - but only after 2 minutes. Too late!

See Leela (0 sec!):

Analysis by Lc0 0.22 (NW42595):

24...c3 25.Dc2 Ta6 26.Kg2 h6 27.Kg3 Tc8 28.f3 Tc4 29.Sa3 Txb4 30.Dd3 Ta8 31.Sc2 Tb1 32.e4 Tc8 33.exd5 Sb6 34.d6 Sd5 35.De4 Sxf4 36.Kxf4 Tb2 37.a3 Te8 38.Dd3 Tc8
Schwarz steht deutlich besser: -/+ (-1.42) Tiefe: 6/19 00:00:00
Schwarz steht deutlich besser: -+ (-1.65) Tiefe: 14/40 00:00:55 1929kN

Leela doesn't calculate, Leela knows...

Greetings: Walter

.

It was faster then 2 min. That is when I clip the analysis.
But of course I thought you tested a bunch of A/B engines and said.
"An amazing move that the Alpha Beta engines don't see..."

Yes, Lc0 sees the move instantly!!!

I can't help but mention one issue I have with your article. You compare depth and nodes of Leela with other engines, which mean completely different things in MCTS and AB engines.

Depth in AB engines refers to the number of iterative deepening steps that have been done. In MCTS there is no such thing as an iterative deepening step and thus the number is instead arbitrarily defined by each respective engine author. They are thus completely impossible to compare.

Nodes in different engines refer to different things. There are different ways you could count nodes eg: eval function calls, make move calls, top of AB search, top of AB and QS search, right before entering QS, and so on. All of them have pros and cons, all of them result in different numbers and to my knowledge there is still no standard.

Due to the iterative deepening and many other things, positions occur in search many times in a classical engine and varying amounts of time are spent on them. Eg if my own engine, Winter, plays in TCEC on 43 threads and reaches an average of depth 30 on each of its threads, the root node will have been called at least 30 * 43 = 1'290 times. Things get even more extreme when you consider search techniques like null move reductions, pvs frameworks, null move reductions, singularity extension checks and so on. Node counts cannot be compared between classical engines, let alone between MCTS NN and classical engines. The sooner people understand this the better.

All this being said, I think having "Leela ratio" is ok. It compares a CPU benchmark with a CPU + GPU benchmark. This does not mean the two numbers being compared actually measure the same thing, they don't, but for the Leela Ratio this is not relevant as long as everyone is doing the same thing. Leela ratio is for hardware comparisons, not engine comparisons.