Laskos wrote: ↑Sun Oct 28, 2018 12:29 pm
In at most a month I will have RTX 2070 GPU working on my desktop PC. I am just waiting to have a more diversified offer, maybe better cooling and maybe better prices. RTX 2070 gives a huge boost,
a factor of 4 (!!!) compared to my GTX 1060 6GB. NVidia by allowing fp16 support, just blew out even my most optimistic expectations for AI/Machine Learning. As I am impatient to see how Leela would fare in new conditions against my top CPU engines, with a bit old i7 4790 4-core 3.8 GHz CPU, which is not a slouch, I played in varied conditions Lc0 ID11261 against SF_dev, by far the best CPU engine in 4:1 time handicap conditions with my "old" GTX 1060. So,
Lc0 is at 240''+ 4'' time control on GTX 1060, SF_dev is on 4 i7 cores at 60''+ 1''.
The comparison is not fair, emulated GPU is state-of-the-art latest generation, more expensive and consumes more power than my a bit old CPU. Next step in several months will be to have an 8 core CPU AMD machine.
Leela Ratio at equal time control would be about 0.5, at this 4x time control, Leela Ratio is equivalent to ~2.0.
Lc0 is v18.1, ID11261.
Strength-wise, from 4-mover opening book of Adam Hair of regular openings, SF_dev and Lc0 are fairly equal:
- Regular openings, 4-moves:
Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 11 - 10 - 29 [0.510] 50
Elo difference: 6.95 +/- 63.00
Finished match
Having this equal result in the general play, I was curious how they compare in different conditions.
From the balanced middlegame positions, SF_dev is much stronger:
- Middlegame balanced
Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 15 - 1 - 34 [0.640] 50
Elo difference: 99.95 +/- 51.76
Finished match
The result is very skewed, and it means that if Lc0 doesn't get a serious advantage until the middlegame, it will either draw or lose, almost never win.
Another interesting aspect would be how Lc0 behaves in unfamiliar Queenless Chess:
[d]rnb1kbnr/pppppppp/8/8/8/8/PPPPPPPP/RNB1KBNR w KQkq - 0 1
- Queenless Chess:
Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 7 - 1 - 42 [0.560] 50
Elo difference: 41.89 +/- 39.83
Finished match
The result is again skewed, SF_dev winning decisively, with Lc0 having difficulty winning any game. Very drawish too, without Queens.
But not always unfamiliar positions disfavor Lc0. In this low draw-rate variant:
[d]3rqknr/4bpp1/4bpp1/1PP1npp1/1PPN1pp1/1PPB4/1PPB4/RNKQR3 w - - 0 1
The fight already starts in the opening and usually ends in midgame, and Lc0 beats heavily SF_dev:
- Variant
Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 14 - 32 - 4 [0.320] 50
Elo difference: -130.94 +/- 102.19
Finished match
Probably, Lc0 is the strongest engine in this variant (say, with Leela Ratio of about 1 and not too short TC).
And finally, and endgame variant, which is borderline Black Win / Draw, as expected SF_dev beats heavily Lc0
[d]1nn1k1n1/4p3/8/8/8/PPPPPPPP/8/4K3 w - - 0 1
- Disbalance:
Score of SF_dev_Syzygy vs lc0_v18.1_Syzygy: 24 - 3 - 23 [0.710] 50
Elo difference: 155.54 +/- 71.95
Finished match
All in all, late midgames and endgames are the main weakness of Lc0, this can explain most of the results in under- or over- performance observed.