My failed attempt to change TCEC NN clone rules

[dkappe]

Oh yeah, if we stick to "2 of 3" rule, I agree that Lc0's NN is surely not "100% unique" relatively to A0 net, it is either a clone or close to being a clone (surely in early Leela it was a clone). Or at least it should bring a strong notion of "not-uniqueness" to either engine or NN part of "2 of 3" rule.

Given that most of NN-based attempts mostly reimplement DeepMind's A0 (although probably Stoofvlees is an exception), where to draw the line between "unique" and "non-unique" is highly opinion-dependent..

For me,
- If an engine implemented NN from a0 papers and intends to move it away from initial architecture as engine develops, it's "unique enough" (although it's already a shaky ground, it's more like an exception only because everyone implements the same thing from a0 paper).

- If an engine took a code from another engine and intends to follow changes by syncing, it is not "unique".

The line is somewhere in the middle, and it's probably not possible to define it in TCEC rules.

I’ve been through the Allie code and the Scorpio code (and the lc0 code, god help us). They all use essentially the same A0 network types, as that is dictated by the PUCT search algorithm. Scorpio and Allie both use a somewhat different search from lc0 and a0, with different backup strategies and various enhancements over the original a0 pseudo code. The lc0 CUDA code that Allie uses (and credits) isn’t particularly innovative although it is technically challenging, especially for someone unfamiliar with GPU programming. Allie would be well served to swap out the lc0 code for a TensorRT INT8 implementation.

[dkappe]

It’s the almost impossible task of catching up with all the work already done and being, lc0. It’s the reason the other NNs are essentially just piggybacks on LC0, because their developers simply can’t do the mass of work involved. It would be like trying to catch Stockfish with something based on TSCP, all by yourself, in limited time, and with a moving target, and then claiming originality. Can’t be done. lc0 engine is the deal. everything else is simplicity in comparison. Piggybackers are clones. Plus they like to spin BS stories. Don’t fall for it.

I think you overestimate how difficult a nn engine is to implement (vs training a net). To facilitate trying new search techniques, I wrote leela lite, a nn engine written in python. The core PUCT algorithm is less that 100 lines of code.

[Milos]

It’s the almost impossible task of catching up with all the work already done and being, lc0. It’s the reason the other NNs are essentially just piggybacks on LC0, because their developers simply can’t do the mass of work involved. It would be like trying to catch Stockfish with something based on TSCP, all by yourself, in limited time, and with a moving target, and then claiming originality. Can’t be done. lc0 engine is the deal. everything else is simplicity in comparison. Piggybackers are clones. Plus they like to spin BS stories. Don’t fall for it.

I think you overestimate how difficult a nn engine is to implement (vs training a net). To facilitate trying new search techniques, I wrote leela lite, a nn engine written in python. The core PUCT algorithm is less that 100 lines of code.

And your reimplemented engine runs "only" 30x slower than Lc0 which is at least 400Elo with the same net. Wow, such an achievement.

[dkappe]

It’s the almost impossible task of catching up with all the work already done and being, lc0. It’s the reason the other NNs are essentially just piggybacks on LC0, because their developers simply can’t do the mass of work involved. It would be like trying to catch Stockfish with something based on TSCP, all by yourself, in limited time, and with a moving target, and then claiming originality. Can’t be done. lc0 engine is the deal. everything else is simplicity in comparison. Piggybackers are clones. Plus they like to spin BS stories. Don’t fall for it.

I think you overestimate how difficult a nn engine is to implement (vs training a net). To facilitate trying new search techniques, I wrote leela lite, a nn engine written in python. The core PUCT algorithm is less that 100 lines of code.

And your reimplemented engine runs "only" 30x slower than Lc0 which is at least 400Elo with the same net. Wow, such an achievement.

If the speed bothers you, I’m working on a Julia version that should resolve that issue. But I think you miss the point: the code for an NN engine only seems complex because it is unfamiliar. Once you understand it, implement a version in your favorite speedy language.

[dkappe]

And your reimplemented engine runs "only" 30x slower than Lc0 which is at least 400Elo with the same net. Wow, such an achievement.

Ah, and it’s not a reimplementation of lc0. No way could it be simple and concise if it imitated the spaghetti code of lc0.

[crem]

If the speed bothers you, I’m working on a Julia version that should resolve that issue.

Will it be even cleaner and faster than your engine in Golang that you've been writing since May 2018, and seemingly actively developed at least until April 2019 (and was "finishing" in February 2019)?

But yeah, I agree, writing the engine part of it is much easier than writing classical chess engine. Eval is slow, so no need for optimizations, counting CPU clock, etc, basically the very first attempt will probably be good enough. E.g. I wrote initial version of Lc0, including backend (tensorflow) during one long Easter weekend. There was no any cleverness in it at all, e.g. I wrote simplest possible movegen, but Lc0 doesn't need fast movegen.

But then doesn't it make an engine which only replaces the simplest part of simple-to-implement engine even less unique?

As for code not being "particularly innovative": if you take a look into any chess engine, I'm pretty sure you won't find much "particularly innovative" there, and mostly either common knowledge or covered in chessprogrammingwiki. Still you cannot just take Stockfish eval and copy it into your engine, telling that it's all standard, I'm lazy to implement all that.

[dkappe]

If the speed bothers you, I’m working on a Julia version that should resolve that issue.

Will it be even cleaner and faster than your engine in Golang that you've been writing since May 2018, and seemingly actively developed at least until April 2019 (and was "finishing" in February 2019)?

Touché. Unfortunately, gpu/nn support in golang was less than optimal and hasn’t really improved. Definitely the wrong technology choice. The engine does run with a 1 ply material and mobility backend, but that’s less than exciting. Julia’s support, for both nn training and inference, is excellent by contrast.

So I’ve got one engine implemented in python, one half of one implemented in golang, and am working on one in Julia. Hopefully the trend is upward rather than downward.

[dkappe]

As for code not being "particularly innovative": if you take a look into any chess engine, I'm pretty sure you won't find much "particularly innovative" there, and mostly either common knowledge or covered in chessprogrammingwiki. Still you cannot just take Stockfish eval and copy it into your engine, telling that it's all standard, I'm lazy to implement all that.

I meant by that that it’s not really chess specific code but rather general neural network evaluation code. If you can use something like tensorrt to abstract and simplify the development, you should do so.

[Ovyron]

Haha, I just had an idea inspired by events that happened in other engine competitions (unrelated to chess), which didn't disallow engines by same author or clone engines. What you could do is entering to TCEC some hundred clones of Allie. Except, they're able to recognize each other by the moves that they play, so there's some Master Allie (the one you want to win the tournament), and lower hierarchies of Allies.

They send to each other "Passwords" (as chess moves), except for Master Allie, so that when they recognize the opponent as Master Allie, they throw the game so she wins. If they recognize that an opponent is an Allie, the "Password" encodes the hierarchy of the Allie clone (this could be split in a binary way, so half the Allies play 1.e4 and the ones in a lower hierarchy plays 1.d4, so they can recognize the hierarchy of 128 Allies in 7 half-moves*), so the one in a lower hierarchy throws away the game against the other one.

If this is done right, it'll ensure that the qualifying engines of the last phase are all Allie clones, and all of them would be throwing the games so Master Allie wins easily, though I doubt the Superfinal will be very exciting by then

*) opening books mess this up, but you get the idea.

[crem]

One of the ideas in the document that I wrote to TCEC was not to compare source code at all, and only look at some kind of “style” or “move” or “eval” similarity (I know all the arguments that those metrics are possible to game, but let’s imagine for a moment we found a way to make it robust).

I.e. if someone changed 1 line in Stockfish, and it made it’s play completely differently (but still strong), let it into TCEC as not clone.

On the other hand, if there are 10 fully independent AlphaZero clones which make the same moves (i.e. if it happens that all they will converge to the same thing), or, hypothetically, 10 different implementations of perfect chess (“32-men tablebase”), there’s no point to have them all.

I don’t think popular bestmove-based similarity test is robust enough. Instead we could for example make engines eval 100 positions of different kinds/imbalances (e.g. picked from games archives when engines don’t agree in eval), sort them by eval value, and then look at permutation similarity between different engines.
Unless there’s explicit scramble of eval, it’s pretty hard to game. And when there’s scramble, it will be noticeable in eval graph.


Back in March I thought that that idea would resolve Allie case by declaring it as unique despite using Lc0 code, but now I think it would actually not pass such test:

In TCEC during all games there are kibitz engines, “blue leela” and “red stockfish” (named after the colors of the lines in graphs). That makes it possible to see eval of Leela for all games of the DivP.

The shape of Allie’s eval graph is very similar to Lc0, that why I think now it wouldn’t pass that difference test. There is no such similarity between other engines, including ones which use NN (scorpioNN, Stoofvlees) or MCTS (KomodoMCTS). You can go to https://www.tcec-chess.com/archive.html, blindly click on random games on the right, and then play “guess the Allie” game by looking at the graph (for me, out of 30 attempts, I failed twice, misclassified Stoofvlees as Allie here, and didn’t recognize Allie here).


Examples (some cropped because when scaled up to +/-10, most of the details are collapsed to zero, also note that Allie still uses old Lc0 win%-to-centipawn conversion formula, so its evals are inflated from zero, but the shape is very similar)):

Allie is white (blue is Leela the kibitzer):



This game is Allie vs LCZero, here Allie is white, blue is Leela the kibitzer, and black is Leela the player:



Allie is black (blue is Leela the kibitzer):