CCRL 40/40, 40/4 and FRC lists updated (16th February 2019)

[Modern Times]

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.

Well, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.

[Rebel]

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.

Well, for ratings lists the engine that plays it's first game for the list must be the same engine that plays game number 1,000. Otherwise things get screwed up. Of course humans learn all the time, but this is computerchess.

Already said the same.

So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.

[hgm]

No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.

There is much more you can do with learning than just a book in disguise, by head:

#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.

#2. Evaluation of Move Sequences.

#3. Evaluation of Pawn Structures.

#4. The Rybka MIT technique.

Imagine a game that went as follows:

Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx

The learning algorithm may decide to concentrate on move 15-20 and play something else.

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.

Not sure what all that means. In the rebel 12 example you seem to store data on individual positions. That is by definition position learning. It is this that is forbidden in CCRL testing.

It is not forbidden to tune your eval, or distribute the engine over several files (e.g. DLLs). It is absolutely insane to demand engines playing without their eval. This is on the same level as allowing Stockfish to participate only if its .exe file is not used.

[hgm]

So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.

That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.

[Rebel]

No, I don't think that the file with NN parameters can be called a 'learn file' anymore than stockfish.exe can be called a learn file. They are just eval and search parameters that happen to be located in a separate file for convenience, rather than statically linked in the executable. LC0 has not done any position learning, which is what 'learn file' traditionally refers to: just a book that contains individual position and their evaluation or recommended best move.

There is much more you can do with learning than just a book in disguise, by head:

#1. The rebel 12 example I gave. In particular speeding up the search and often calculate one ply deeper.

#2. Evaluation of Move Sequences.

#3. Evaluation of Pawn Structures.

#4. The Rybka MIT technique.

Imagine a game that went as follows:

Move 1-10 : book
Move 11 : score +0.20
Move 15 : score +0.25
Move 20 : score -0.10
Move 25 : score -0.80
Move 30 : score -3.xx

The learning algorithm may decide to concentrate on move 15-20 and play something else.

Learning is an unexplored area and as long as it is on the boycott list of rating lists it will remain that way.

Not sure what all that means.

Just ask what's not clear.

In the rebel 12 example you seem to store data on individual positions. That is by definition position learning. It is this that is forbidden in CCRL testing.

Tell me what's so wrong on the following example:

[d]r5k1/2p1b1p1/6bp/p4P2/q3pP2/2PnB2P/PP1N4/KR3Q1R b - - bm Nb4;

Code: Select all

00:00:26  16.04  -0.52  1..Nb4 
00:00:48  16.04  -0.52  1..Nb4 
00:01:13  16.04   2.53  1..Nb4 2.cxb4 axb4 3.a3 Bf7 4.Qd1 b3 5.Nc4 Qxc4 6.Qc1 Qc2 

and via the learn file:

Code: Select all

00:00:50  19.00  1.46  1..Nb4 2.cxb4 axb4 3.a3 Bf6 4.Bd4 Bf7 5.Bxf6 
                          Qc2 6.Nc4 Bxc4 7.Rh2 Qb3 8.Qxc4 Qxc4 9.Rg2 
                          bxa3 10.b3 Qd5 11.Rxg7 Kf8 12.Rxc7 Qxf5 13.Be5 

3 plies deeper.

Why should this be forbidden, it's blocking progress.

It is not forbidden to tune your eval, or distribute the engine over several files (e.g. DLLs). It is absolutely insane to demand engines playing without their eval. This is on the same level as allowing Stockfish to participate only if its .exe file is not used.

In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we?

[Modern Times]

So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.

That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.

Agreed.

[hgm]

Just ask what's not clear.

What is 'valuation of move sequences'? What does MIT stand for, and what does it do?

Tell me what's so wrong on the following example:
...
Why should this be forbidden, it's blocking progress.

How do you expect anyone to see that from the output? It is what goes on inside that matters. So tell us what you did, and we can tell you whether it should be forbidden and why.

In the above example search and eval are in full use and regarding AZ and LZ we both know for what the "Z" stands for, don't we?

Yeah, and when Lance Armstrong was winning the Tour de France his muscles and lungs were also in full use. It were the other things that he used that got him into trouble. Likewise in the example above. That the engine can ride uphill faster doesn't mean it is clean...

One of us must be missing something. The "Z" indicates the engine was not tuned on any games that it did not generate itself. Why do you think that is funny?

[Rebel]

So why not announce a rule change in your FAQ? Now that Lc0 is on the brink to overtake the crown of Stockfish give traditional engines the means to learn in return with the above limitation and maybe a few more.

That is no rule change. It has always been allowed to train/tune engines, and submit the improved version for testing. But it would get a new version number, of course, to not confuse it with the older, less trained/tuned version, or contaminate its performance with the results of that previous one.

Agreed.

Under the current rules LZ has an unfair advantage.

[hgm]

So you say, but so far you have not offered a single valid argument why that would be. The argument that its evaluation resides in a different file doesn't impress anyone.

In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.

[Modern Times]

In fact the current CCRL rules give LC0 a tremendously unfair disadvantage, because the standard hardware does not include a GPU.

As was pointed out to us, we say we are a standard hardware list but we are not. We have 1CPU and 4CPU engines on there and they play each other to some extent so that the list is joined up. So a "GPU" could be just another variation of that. So the list would be "1CPU", "4CPU" and "GPU". I think CEGT have in fact set aside a couple of standard reference machines for GPU testing for a 2 year minimum period. From their postings, I think the time control for matches is set in the normal way based on the speed of the CPU, and then the GPU is what it is as the standard reference GPU. That seems completely clear and transparent to me.