AndrewGrant wrote: ↑Tue Sep 29, 2020 9:33 am
Two years and some months ago, Alpha Zero dropped their initial paper, claiming to have thrashed Stockfish by a damning margin. People were quick to take a side. After the dust settled, I think most agreed that Alpha Zero's conditions were inane, and the the result was not indicative of a shift in the times. In the same time, Leela entered the scene. Over the last two years Leela has stayed close to Stockfish in strength, but has increasingly reduced the hardware required to do so.
I was never a fan of the Leela projects appearance on the scene. I thought, and still think, that GPUs vs CPUs is an unfair comparison. Of course, time passed and the Leela team managed to reach similar strength, but without the use of GPUs -- a total repudiation of my original stance. Anyway, more so than Leela, I was concerned about a new wave of engines, all built upon the work of the Leela project. I expected to see a dozen engines, all with only slight variance and nuance added. Blessed, this never came to be. Leela and Allie are, to me, twins. The other NNs out there don't bare the same relation to the Leela projects. They have different trainers, or different datasets, or different structures, or different back-ends. I think, and I hope, there is something special about each of them.
Now, in recent months, a similar series of developments are happening. NNUEs, or rather, I want to be specific, the King-Piece structures used in Stockfish, are flooding the scene. With a few hours work, an author can copy paste some code from CFish or Stockfish, add the incremental update code, and with some debugging they can begin playing games with Stockfish's Networks. In the last month a half dozen authors have posted their results from using the Stockfish structure/networks -- almost all +200 elo or more.
For the most part in Computer Chess, specific ideas and implementations are not transferable. Sure, we all have the same NMP, LMR, FMP, Probcut, Singular Extensions techniques, but in search if you try a patch pushed to Stockfish this week, it will likely fail. An even better example is evaluation. Evaluation functions are so specific to engines, so fine tuned to the already existing ideas, that virtually all attempts to take an idea from another engine, without significant changes or reworking, is futile.
So its odd, or to me it is, that someone can plug Stockfish's Network files into their own engine, and share in the same success. In fact, its not odd, its concerning. One can treat the Network file as (I could be wrong on the math, I don't know the structure) nearly, if not over, a million tiny evaluation terms. So I ask myself this: I cannot copy idea X from Stockfish's eval into Ethereal. But if I copy a million weights from Stockfish, then my evaluation is so similar to Stockfish, that I then become able to take X.
Well so, obviously Stockfish does not have global claim to the idea of NNs using King-Piece inputs. Its essentially a giant PSQT input, with King-Piece crossing. No one here has the rights to PSQTs -- everyone has one, everyone uses one, no one bats an eye.
----
I worry about the future of Computer Chess. I see a timeline where a dozen engines use something very similar to Stockfish's methods. They all shoot up in elo. New developers, people working on their own innovations, are disheartened. They ask themselves why they toil away on their new ideas and tweaks, when one can just embrace the NNUE and be on an equal playing field with the top tiers of engines. So they leave. I leave. Alayan leaves. Many others leave. The result is that there are two engines left; Stockfish, and Leela. I don't find that interesting. Maybe others do I suppose.
At the same time, this is me jumping up and down, waving my hands, saying "Hey, I've been working on Ethereal for 60hrs a week for the last 6 years. I've done all this work, spent all this time and energy. But now if you want to play at Stockfish's level, you just need to download the training code, feed in your evals, wait a few billion clock cycles, and presto, its done. Why should I bother?"
I released a tuning paper a few months ago. It was the culmination of a years effort on various implementations, as well as likely over a hundred different methodologies for building datasets. I shared that paper, and I shared pieces of the Ethereal data, about 10 million positions at a time, to all those who asked. I think, two years ago, this would have made a splash. In fact, I still believe that someone could perform the same exercise as I outline, and gain +15 elo or more to Stockfish's static evaluation (pre-NNUE). But now my work is futile?
I built an open source framework that mimics Fishtest, but works for many engines at a time. We support engines of all types. C, C++, Rust, Java, and virtually anything you can compile on two platforms. With the help of Noobpwnftw, we hooked up machines and built a framework for authors to work on their own projects -- but share with others at the same time. I run my tests, and others can see them and tinker. Others run their tests, and I can see them in tinker. It was a venture in facilitating a greater exchange of ideas in Computer Chess. A venture in promoting stronger, but nuanced and diverse engines. But now my work is futile?
I feel that soon it will become clear that I've spent six years to do nothing. Ethereal, unless I too copy paste the NNUEs, will be tossed out and placed on the dustbin of history. I never expected to be at the top -- but I got close. At a time, the 5 strongest engines in the world were {Stockfish, Komodo, Houdini, Fire, Ethereal}. We learned that Houdini and Fire were stolen goods. So at a time Ethereal was #3 -- but still far far far from #2. I could make progress towards tackling Komodo, while gaining ground on the rest of the field. Prior to two months ago, if you asked me, I would tell you that Ethereal would surpass Komodo in two years. Now I can't say that. I won't use Stockfish's methods. I'm adding my own NN ideas to Ethereal, but it bores me. I'm praying that one can beat NNUE King-Piece networks with some brilliant architecture. But my hopes fall each day.
I could be all wrong. I could be out of touch. But if I'm not, then the future of computer chess, the future of unique and diverse engines, depends upon all of us, as individuals, to encourage and promote new ideas while discouraging those who take from Stockfish without trying their hand at the problem. I'm already concerned when I see engines with Stockfish nets being placed onto rating lists.
