A Crossroad in Computer Chess; Or Desperate Flailing for Relevance

[mvanthoor]

SUMMARY:
I believe you should continue your work as if Leela on the one hand and NNUE on the other never existed.
IGNORE modern fancy trends because they are just modern fancy trends.

This is the advice that Kodak got when the digital camera arrived. And Nokia and RIM when touchscreens started to take off.

It's ironic, how Kodak was unable to keep up with the revolution, knowing the fact that they invented the digital camera themselves. They invented their own doom.

It would be as if someone discovered a new technique such as NNUE, decides that it's not worth using because it is "not good enough" or "doesn't work as desired", and then gets crushed by others who do use refined versions the idea.

What Maksim probably means is: go your own way with your engine. Even if the first 5 engines on the list are all NNUE or Leela-like engines, you can STILL be the strongest pure alpha/beta engine. (I even think that there should be separate lists for pure A/B-engines, NNUE engines, and Leela-like engines that run on GPU's, because they're so different that it's almost impossible to really compare them fairly.)

[syzygy]

I think the largest problem is team size.
Stockfish and LC0 have giant teams.
Teams with highly skilled people who work for nothing.
Teams with sky-high piles of gleaming hardware worth millions of dollars, donating both hardware, time, electricity and effort in testing.

I guess that Andrew is as clever as the most clever person on either team.
But when you have 100 workers, it is hard to compete against that.
And when you have a million dollars of hardware on your team, how will a lone developer compete with that?

It is, in fact, the mega-teams that are squashing the smaller teams.
And the people taking things are taking them from the mega teams because that is the only way they can compete.

I guess that the only small entity who can beat a mega team is one who has a supremely clever idea (like the NNUE thing) and then hides what they did so that nobody can copy it.

If you even say what your approach was, someone else will try it.

The only reason why there are two mega-teams in computer chess is because AlphaZero spurred the development of Leela. Otherwise, there would only have been Stockfish. The die was cast when Marco Costalba and Joona Kiiski retired from Stockfish development in 2011, and the community chose to continue development on Stockfish as a distributed open-source project.

Marco remained involved for much longer (and he has never left the project as far as I am aware).

The die was cast when Gary Linscott created the Fishtest framework. This is what made it possible for so many people to contribute to Stockfish.

Interestingly, Gary Linscott is also the person who started LC0.

Obviously this does not take away anything from Marco and Joona and Tord and Gian-Carlo Pascutto and Deep Mind and Alan Turing, etc. etc.

In the end, this thread is about finding joy in making progress vs doing it just to compete. Andrew wrote "I was never a fan of the Leela projects appearance on the scene". Does anyone who thinks he agrees with Andrew actually agree with that statement?

[syzygy]

(I even think that there should be separate lists for pure A/B-engines, NNUE engines, and Leela-like engines that run on GPU's, because they're so different that it's almost impossible to really compare them fairly.)

How about a separate competition for chess engines without a functioning transposition table?

Well, I agree that "fair" competition between programmers is basically impossible nowadays. Teams are of vastly different sizes, available hardware resources are incomparable, "originality standards" vary greatly, etc. etc. There is really no point in this "sport" anymore if you just do it for the "competitive" element. However, it is still fun to create your own engine or to contribute to SF or LC0. Or if you have a "big idea" like NNUE, just start from SF or LC0 and see if it works (no need to reinvent the wheel if you want to spend your time on other things).

[syzygy]

Once these 3600 Elo over-night geniuses are discovered and hired by Elon Musk. We can finally start populating universum. Instead of playing some 8x8 board game.
Passing Van Allen Radiation belts, travelling astronomical distances to the nearest stars, surviving on hostile planets without oxygen/water/food/etc are soon to be solved.

Ehm, the Van Allen radiation belts aren't exactly what prevents mankind from travelling to, say, the moon...

[Madeleine Birchfield]

Well, I agree that "fair" competition between programmers is basically impossible nowadays. Teams are of vastly different sizes, available hardware resources are incomparable, "originality standards" vary greatly, etc. etc. There is really no point in this "sport" anymore if you just do it for the "competitive" element. However, it is still fun to create your own engine or to contribute to SF or LC0. Or if you have a "big idea" like NNUE, just start from SF or LC0 and see if it works (no need to reinvent the wheel if you want to spend your time on other things).

Tomasz Michniewski advocated for using a PESTO-like evaluation for all engines for testing and participation in tournaments, so that the only thing that matters is an engine's search.

https://www.chessprogramming.org/Simpli ... n_Function

I think he would be very proud of NNUE, as it basically does the same as his proposal, but with a much better evaluation function than PESTO. Just choose a standard net (say a sergio vieri net) and test away.

[mvanthoor]

(I even think that there should be separate lists for pure A/B-engines, NNUE engines, and Leela-like engines that run on GPU's, because they're so different that it's almost impossible to really compare them fairly.)

How about a separate competition for chess engines without a functioning transposition table?

My engine in version 1 doesn't even HAVE a transposition table, so that's not fair!

We need a competition for engines with no transposition table, and another for engines where the TT doesn't function or doesn't work right due to bugs

However, it is still fun to create your own engine...

Yes; I want to create my own engine (with levels function) to play against. The other reason is that I want to beat Fritz 11... even if I have to write another program to do it

[dkappe]

Tomasz Michniewski advocated for using a PESTO-like evaluation for all engines participating in a tournament, so that the only thing that matters is an engine's search.

https://www.chessprogramming.org/Simpli ... n_Function

I think he would be very proud of NNUE, as it basically does the same as his proposal, but with a much better evaluation function than PESTO.

Brilliant!

A different network every round.

Toga, Frosty, Night Nurse, Dark Horse, most recent SF, specific engine nets. We have enough variety in nets and engines to make for an interesting event.

[mvanthoor]

Well, I agree that "fair" competition between programmers is basically impossible nowadays. Teams are of vastly different sizes, available hardware resources are incomparable, "originality standards" vary greatly, etc. etc. There is really no point in this "sport" anymore if you just do it for the "competitive" element. However, it is still fun to create your own engine or to contribute to SF or LC0. Or if you have a "big idea" like NNUE, just start from SF or LC0 and see if it works (no need to reinvent the wheel if you want to spend your time on other things).

Tomasz Michniewski advocated for using a PESTO-like evaluation for all engines for testing and participation in tournaments, so that the only thing that matters is an engine's search.

https://www.chessprogramming.org/Simpli ... n_Function

I think he would be very proud of NNUE, as it basically does the same as his proposal, but with a much better evaluation function than PESTO. Just choose a standard net (say a sergio vieri net) and test away.

In that case, you can go the other way around as well: impose a search depth, let the engines think as long as they need to reach it. Then, the only thing that matters would be the evaluation function.

Then see if the engine with the best search function benefits by a retro-fit of the best evaluation...

[Wilson]

Andrew, let me know if I got it straight: after years of plundering Stockfish search and evaluation functions and gaining hundreds of ELO points, you are now complaining because there is a new breakthrough (NNUE) and many authors are implementing it. Some of them, and I guess more and more in the future, are training the nets using their own evaluation function and even modifying the net architecture (Nemorino) and who knows how this NN thing will evolve in the feature. I'm looking forward to the developments.

So, what is your problem exactly? What's the difference between taking the search function from Stockfish (from where most of the strength comes: just look at how many ELO points every algorithm is worth in SF comments...or in Ethereal also, you have the same type of comments too ) tweaking some parameters here and there in the process, and implementing a new NN architecture, modifying it and tuning it using one's own evaluation function?

It's not like you invested years in R&D inventing new search algorithms that brought Ethereal where it is today and now suddenly everyone is surpassing you because of this bad bad NNUE. Ethereal, like many others closed and open source engines, does everything like Stockfish except for some minor details. So why you can't do the same again? Perhaps because you have always bad-mouthed NN and if you now replaced your whole eval with one NN, you would seem a hypocrite?
(thus you can only at most use a small NN for pawns+kings?)

You have called Scorpio a Leela derivative in the past, you are questioning the status of NNUE engines even before the try to enter TCEC and one of your groupies is pointing the finger at Allie.
I think you should calm down and stop trying to take any engine stronger than Ethereal out of TCEC, otherwise some one could get annoyed and open a thread with actual code comparisons between Ethereal and Stockfish's search and evaluation functions, asking if Ethereal is a derivative of Stockfish and if so, why Baby Stockfish keeps playing in TCEC.

After everything I saw, I find a bit awkward to see "Stockfish" mentioned only one time in Ethereal sources, in the bestGroup function. You don't even mention Stockfish in Ethereal's Github page, just a vague "Ethereal is inspired by a number of open source projects". Quite telling.

[AndrewGrant]

Andrew, let me know if I got it straight: after years of plundering Stockfish search and evaluation functions and gaining hundreds of ELO points, you are now complaining because there

I stopped reading after that. Go take a look at Ethereal. I'm not fighting randoms over origins.

https://github.com/AndyGrant/Ethereal/wiki Look harder.