Something goes wrong with lc0 since yesterday?

[yanquis1972]

One month 50 (real) Elo and 4 million training games later. 1 training game would keep my pc busy whole day. So 4 million days later would be 11000 years. If your pc runs 100 times faster then you only need 110 years to get this fantastic result of 50 Elo points.

what is your 'PC'?? pre-pentium, for sure...

[Henk]

I-5 3210M CPU 2.50GHZ. Bought in 2013 or so

When I start my computer I already have to wait half an hour before I can use it. This was not so one year ago. I don't know what happened.
Maybe visual studio 2017 ? Before I used 2015 version.

[George Tsavdaris]

40/2' against single core is fine, but the most strictly correct simulation of CCRL conditions with lc0 on GPU would probably be to leave the lc0 at the same 40/4', as lc0 speed on GPU is very weakly dependent on CPU (2 threads), and for AB engines use 40/2' on a reasonable i7 core. Anyway, we get a picture, I use the same 40/2' bench for both lc0 and AB engines as you use. My GPU is Nvidia 1060 6GB.

Despite what you said here, i used 40/2 for Leela also, in the following gaunlet since i find it unfair to offer Leela double time.
The following 2 gaunlets are for the latest test net, but the way it evolved last day is AMAZING!

AB engines use 1 CPU i5-6500 3.4 GHz that i tested to be 2.1 times faster than CCRL's hardware so to have equivalence of the CCRL 40/4 Elo i used a 40/2 time control.
Leela is using Lc0 with GTX 1070 Ti with 40/2 time control also. I tested 2 testing nets, 10076 and the 1 day later(where they dropped LR) 10080.

Test10 10076 after 100 games had in the following gaunlet a 2970 ±45.4 CCRL 40/4 Elo performance and
Test10 10080 after 63 games so far has in the same gaunlet a 3243 ±75.1 CCRL 40/4 Elo performance!

Lc0 Test10 10076 after 100 games:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10076   2969.7     ±45.4              100 (+26,=41,-33)    46.5 %

   vs.                        :  games (  +,  =,  -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     20 (  3,  8,  9),  35.0 :  -127.3,  23.2,    0.0
   Pedone 1.7                 :     20 (  5, 11,  4),  52.5 :   -43.3,  23.2,    3.1
   Wasp 3.0                   :     20 (  4, 11,  5),  47.5 :   -17.3,  23.2,   22.7
   Rybka 2.3.2a 64 bit        :     20 (  5,  4, 11),  35.0 :    +6.7,  23.2,   61.3
   Spike 1.4                  :     20 (  9,  7,  4),  62.5 :   +51.7,  23.2,   98.7

Lc0 Test10 10080 after 63 games so far:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10080   3242.9     ±75.1              63 (+44,=12,-7),     79.4 %

   vs.                        :  games (  +,  =, -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     13 (  8,  4, 1),  76.9 :  +145.9,  38.3,  100.0
   Pedone 1.7                 :     12 (  7,  3, 2),  70.8 :  +229.9,  38.3,  100.0
   Wasp 3.0                   :     13 ( 12,  1, 0),  96.2 :  +255.9,  38.3,  100.0
   Rybka 2.3.2a 64 bit        :     13 (  8,  2, 3),  69.2 :  +279.9,  38.3,  100.0
   Spike 1.4                  :     12 (  9,  2, 1),  83.3 :  +324.9,  38.3,  100.0

[crem]

40/2' against single core is fine, but the most strictly correct simulation of CCRL conditions with lc0 on GPU would probably be to leave the lc0 at the same 40/4', as lc0 speed on GPU is very weakly dependent on CPU (2 threads), and for AB engines use 40/2' on a reasonable i7 core. Anyway, we get a picture, I use the same 40/2' bench for both lc0 and AB engines as you use. My GPU is Nvidia 1060 6GB.

Despite what you said here, i used 40/2 for Leela also, in the following gaunlet since i find it unfair to offer Leela double time.
The following 2 gaunlets are for the latest test net, but the way it evolved last day is AMAZING!

The learning rate was changed at network 10077 (as planned), so progress should indeed be much faster now.

[Werewolf]

40/2' against single core is fine, but the most strictly correct simulation of CCRL conditions with lc0 on GPU would probably be to leave the lc0 at the same 40/4', as lc0 speed on GPU is very weakly dependent on CPU (2 threads), and for AB engines use 40/2' on a reasonable i7 core. Anyway, we get a picture, I use the same 40/2' bench for both lc0 and AB engines as you use. My GPU is Nvidia 1060 6GB.

Despite what you said here, i used 40/2 for Leela also, in the following gaunlet since i find it unfair to offer Leela double time.
The following 2 gaunlets are for the latest test net, but the way it evolved last day is AMAZING!

The learning rate was changed at network 10077 (as planned), so progress should indeed be much faster now.

So the learning rate has been reduced and this speeds up learning? Hmm. One day I might understand it.

[Robert Pope]

So the learning rate has been reduced and this speeds up learning? Hmm. One day I might understand it.

Here's how I understand it:

With a high learning rate, you have a good chance of overshooting the nearest "best" values, but hopefully you are generally moving through any locally good but not absolutely best settings, and getting generally close to the absolutely best area.

With a lower learning rate, you will progress directly towards the nearest best settings, but there is a risk that this is only a local minimum, and not the absolute best set of settings. So it will generally show more steady improvement but is more likely to hit a dead-end.

I'm never going to find the highest mountain in America if I start in Iowa and limit my steps to the next highest hill in sight.

[Werewolf]

Thanks Robert, that makes sense.

[Laskos]

40/2' against single core is fine, but the most strictly correct simulation of CCRL conditions with lc0 on GPU would probably be to leave the lc0 at the same 40/4', as lc0 speed on GPU is very weakly dependent on CPU (2 threads), and for AB engines use 40/2' on a reasonable i7 core. Anyway, we get a picture, I use the same 40/2' bench for both lc0 and AB engines as you use. My GPU is Nvidia 1060 6GB.

Despite what you said here, i used 40/2 for Leela also, in the following gaunlet since i find it unfair to offer Leela double time.
The following 2 gaunlets are for the latest test net, but the way it evolved last day is AMAZING!

AB engines use 1 CPU i5-6500 3.4 GHz that i tested to be 2.1 times faster than CCRL's hardware so to have equivalence of the CCRL 40/4 Elo i used a 40/2 time control.
Leela is using Lc0 with GTX 1070 Ti with 40/2 time control also. I tested 2 testing nets, 10076 and the 1 day later(where they dropped LR) 10080.

Test10 10076 after 100 games had in the following gaunlet a 2970 ±45.4 CCRL 40/4 Elo performance and
Test10 10080 after 63 games so far has in the same gaunlet a 3243 ±75.1 CCRL 40/4 Elo performance!

Lc0 Test10 10076 after 100 games:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10076   2969.7     ±45.4              100 (+26,=41,-33)    46.5 %

   vs.                        :  games (  +,  =,  -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     20 (  3,  8,  9),  35.0 :  -127.3,  23.2,    0.0
   Pedone 1.7                 :     20 (  5, 11,  4),  52.5 :   -43.3,  23.2,    3.1
   Wasp 3.0                   :     20 (  4, 11,  5),  47.5 :   -17.3,  23.2,   22.7
   Rybka 2.3.2a 64 bit        :     20 (  5,  4, 11),  35.0 :    +6.7,  23.2,   61.3
   Spike 1.4                  :     20 (  9,  7,  4),  62.5 :   +51.7,  23.2,   98.7

Lc0 Test10 10080 after 63 games so far:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10080   3242.9     ±75.1              63 (+44,=12,-7),     79.4 %

   vs.                        :  games (  +,  =, -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     13 (  8,  4, 1),  76.9 :  +145.9,  38.3,  100.0
   Pedone 1.7                 :     12 (  7,  3, 2),  70.8 :  +229.9,  38.3,  100.0
   Wasp 3.0                   :     13 ( 12,  1, 0),  96.2 :  +255.9,  38.3,  100.0
   Rybka 2.3.2a 64 bit        :     13 (  8,  2, 3),  69.2 :  +279.9,  38.3,  100.0
   Spike 1.4                  :     12 (  9,  2, 1),  83.3 :  +324.9,  38.3,  100.0

Good, I haven't followed it recently, thanks for the update.
ID10084 comes in my test at only about 80 Elo points behind ID495 net of the main branch, which seems the best overall as of now, or some 3250 or a bit more CCRL 40/4' Elo points on GTX 1060. On your 1070 Ti it is probably 3300 or so. Huge improvements on the testserver since LR changed!

[JJJ]

40/2' against single core is fine, but the most strictly correct simulation of CCRL conditions with lc0 on GPU would probably be to leave the lc0 at the same 40/4', as lc0 speed on GPU is very weakly dependent on CPU (2 threads), and for AB engines use 40/2' on a reasonable i7 core. Anyway, we get a picture, I use the same 40/2' bench for both lc0 and AB engines as you use. My GPU is Nvidia 1060 6GB.

Despite what you said here, i used 40/2 for Leela also, in the following gaunlet since i find it unfair to offer Leela double time.
The following 2 gaunlets are for the latest test net, but the way it evolved last day is AMAZING!

AB engines use 1 CPU i5-6500 3.4 GHz that i tested to be 2.1 times faster than CCRL's hardware so to have equivalence of the CCRL 40/4 Elo i used a 40/2 time control.
Leela is using Lc0 with GTX 1070 Ti with 40/2 time control also. I tested 2 testing nets, 10076 and the 1 day later(where they dropped LR) 10080.

Test10 10076 after 100 games had in the following gaunlet a 2970 ±45.4 CCRL 40/4 Elo performance and
Test10 10080 after 63 games so far has in the same gaunlet a 3243 ±75.1 CCRL 40/4 Elo performance!

Lc0 Test10 10076 after 100 games:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10076   2969.7     ±45.4              100 (+26,=41,-33)    46.5 %

   vs.                        :  games (  +,  =,  -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     20 (  3,  8,  9),  35.0 :  -127.3,  23.2,    0.0
   Pedone 1.7                 :     20 (  5, 11,  4),  52.5 :   -43.3,  23.2,    3.1
   Wasp 3.0                   :     20 (  4, 11,  5),  47.5 :   -17.3,  23.2,   22.7
   Rybka 2.3.2a 64 bit        :     20 (  5,  4, 11),  35.0 :    +6.7,  23.2,   61.3
   Spike 1.4                  :     20 (  9,  7,  4),  62.5 :   +51.7,  23.2,   98.7

Lc0 Test10 10080 after 63 games so far:

Code: Select all

  Program          CCRL Elo   Error(cl 95%)          Games            Score 
Lc0 Test10 10080   3242.9     ±75.1              63 (+44,=12,-7),     79.4 %

   vs.                        :  games (  +,  =, -),   (%) :    Diff,    SD, CFS (%)
   Laser 1.5                  :     13 (  8,  4, 1),  76.9 :  +145.9,  38.3,  100.0
   Pedone 1.7                 :     12 (  7,  3, 2),  70.8 :  +229.9,  38.3,  100.0
   Wasp 3.0                   :     13 ( 12,  1, 0),  96.2 :  +255.9,  38.3,  100.0
   Rybka 2.3.2a 64 bit        :     13 (  8,  2, 3),  69.2 :  +279.9,  38.3,  100.0
   Spike 1.4                  :     12 (  9,  2, 1),  83.3 :  +324.9,  38.3,  100.0

Good, I haven't followed it recently, thanks for the update.
ID10084 comes in my test at only about 80 Elo points behind ID495 net of the main branch, which seems the best overall as of now, or some 3250 or a bit more CCRL 40/4' Elo points on GTX 1060. On your 1070 Ti it is probably 3300 or so. Huge improvements on the testserver since LR changed!

That's good news, that's mean at this rate we might see a new strongest id in few days.

[Laskos]

These days I probed lc0 ID495 (main branch) for behavior on certain openings against Gaviota 1.0 regular engine (on two cores for randomization). I played overall several thousands ultra-fast games from Noomen topical opening suite.

Here I present over- and under- performance (over- with green, under- with red) in Elo compared to general performance for most common openings:

Ruy Lopez: -46
French: -43
Sicilian: -46
Queen's Pawn: +75
King's Indian: -31
Nimzo Indian: +64
Reti: +31

The 2SD error margins are about 25 Elo points.