LCZero: Progress and Scaling. Relation to CCRL Elo

[AlvaroBegue]

I ran some matches between RuyDos and LCZero at 1 minute + 1 second/move time control. I have an i7-4790K with a GTX980. This should be comparable to CCRL 40/4 time control. RuyDos is ~2780 in that list (the version I used might be 2795 or so; we'll see when version 1.1.6 makes it to the list).

My estimate is that LCZero is 50 to 200 points stronger than RuyDos, which is almost identical to Kai's estimate.

[Laskos]

Taking into account a 3-4% slowdown of the net, I guess that the improvement since ID227 is in the range of 25 Elo points, maybe even a bit more. Have yet to test at fixed time control.

In my gauntlet, the strongest 128-sized network that I tried was 195 which got a score of 108.5/200 with an elo of 2667.3
The last 2 networks I tested (already 195 in size) are 232 and 236 which got 117.5 and 117/200 respectively, with elos of 2700.8 and 2698.9

That's already ~35 elo more (with the same TC and the same hardware) against 195 and ~45 more than 227, and it looks like 239 is somewhat stronger, so I would guess it's a bit more than 25, more like 60 maybe? I'll test 240 later tonight (or 241, depending on external factors such us life ) and we'll see where we stand right now.

My result for ID241 versus ID227 (the first larger net) is about 30 +/- 15 Elo points improvement at fixed time control. I cannot lower much the error bars, as we need the error of the difference in matches, which is roughly sqrt(2) times the error in the match. Your error bars are large.

[Laskos]

Taking into account a 3-4% slowdown of the net, I guess that the improvement since ID227 is in the range of 25 Elo points, maybe even a bit more. Have yet to test at fixed time control.

In my gauntlet, the strongest 128-sized network that I tried was 195 which got a score of 108.5/200 with an elo of 2667.3
The last 2 networks I tested (already 195 in size) are 232 and 236 which got 117.5 and 117/200 respectively, with elos of 2700.8 and 2698.9

That's already ~35 elo more (with the same TC and the same hardware) against 195 and ~45 more than 227, and it looks like 239 is somewhat stronger, so I would guess it's a bit more than 25, more like 60 maybe? I'll test 240 later tonight (or 241, depending on external factors such us life ) and we'll see where we stand right now.

My result for ID241 versus ID227 (the first larger net) is about 30 +/- 15 Elo points improvement at fixed time control. I cannot lower much the error bars, as we need the error of the difference in matches, which is roughly sqrt(2) times the error in the match. Your error bars are large.

All the recent previous tests were with v0.7 binary. Testing with v0.8 at fixed time gives much better results. 70 +/- 20 Elo points improvement compared to v0.7 binary using the same ID241 net. So, all in all, ID241 v0.8 compared to ID227 v0.7 is about 100 +/- 20 Elo points better. I don't know where from this search improvement comes from, maybe that FPU bug. I used no TBs.

[jkiliani]

All the recent previous tests were with v0.7 binary. Testing with v0.8 at fixed time gives much better results. 70 +/- 20 Elo points improvement compared to v0.7 binary using the same ID241 net. So, all in all, ID241 v0.8 compared to ID227 v0.7 is about 100 +/- 20 Elo points better. I don't know where from this search improvement comes from, maybe that FPU bug. I used no TBs.

Almost certainly this improvement comes from the change in PUCT, which makes the search deeper but explores low policy moves less than before. Gian-Carlo Pascutto made the tuning test that revealed this improvement potential a while ago, and after some further testing this change was merged with 0.8. The rules of the FPU bug is minor compared to that.

[Werewolf]

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

[peter]

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

At WACNEW suite with network 240 now 144 of 300 solutions again only instead of 151 with the latest one I tried about a week ago.
Both with 24 threads of 3GHz CPU and 10"/move.

[Albert Silver]

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

At WACNEW suite with network 240 now 144 of 300 solutions again only instead of 151 with the latest one I tried about a week ago.
Both with 24 threads of 3GHz CPU and 10"/move.

Is there a PGN I could download with that suite?

[peter]

Hi Albert!

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

At WACNEW suite with network 240 now 144 of 300 solutions again only instead of 151 with the latest one I tried about a week ago.
Both with 24 threads of 3GHz CPU and 10"/move.

Is there a PGN I could download with that suite?

http://www.talkchess.com/forum/viewtopi ... 14&t=67239
Vincent Lejeune started this thread then here with first results:
http://www.talkchess.com/forum/viewtopi ... 11&t=67239

[Albert Silver]

Hi Albert!

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

At WACNEW suite with network 240 now 144 of 300 solutions again only instead of 151 with the latest one I tried about a week ago.
Both with 24 threads of 3GHz CPU and 10"/move.

Is there a PGN I could download with that suite?

http://www.talkchess.com/forum/viewtopi ... 14&t=67239
Vincent Lejeune started this thread then here with first results:
http://www.talkchess.com/forum/viewtopi ... 11&t=67239

That's in EPD. Is there a way to process tests in Arena?

[peter]

Hi Albert!

Does that mean LCZero will now be weaker at tactics if the strange moves are given less time?

At WACNEW suite with network 240 now 144 of 300 solutions again only instead of 151 with the latest one I tried about a week ago.
Both with 24 threads of 3GHz CPU and 10"/move.

Is there a PGN I could download with that suite?

http://www.talkchess.com/forum/viewtopi ... 14&t=67239
Vincent Lejeune started this thread then here with first results:
http://www.talkchess.com/forum/viewtopi ... 11&t=67239

That's in EPD. Is there a way to process tests in Arena?

There's an option in Arena "Automatic Analysis"-"List of EPD/PGN files" in menue "Engines".