Alphazero news

[yanquis1972]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

That's true, but the most striking in my examples was letting Lc0 play whatever it wants from the Initial Board position against SF + Book, so Lc0 was put to play Chess by its definition, but it sometimes doesn't like Chess too much, it likes Chess much more in the paper.

How deep is the book? I don’t think there’s any claim that NNs are superior to top opening theory. Just that, at times, they can play them with relatively surprising accuracy. I do think it’s an interesting result, but imo any elite book (if that’s what you used) should be limited to 10 ply or less, otherwise you’re mostly demonstrating what’s already clear, that Lc0 hasn’t surpassed current theory.

[Laskos]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

What makes you believe that?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings.

Yes, I agree with Kai. Chess is not like that. No one or thing can "always play into closed openings no matter what the opponent does". That claim is not true.

And again, it wasn't a "claim".

"If I could always pick the best move, I would be the best chess player in the world." Clearly a true statement, but I'm not claiming to be the best chess player in the world in that statement.

My English is bad, the context was:

The TCEC openings are all open and tactical openings, favouring SF. Why do you say they are more reliable?

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It seems to me at least a justification to use favorable to A0 openings, because it's quite possible that A0 steers the openings into favorable ones. Am I missing something? I used a book for SF10, showing that it's far from the truth that A0 can steer every opening the way it likes. So, there is no justification to use favorable to A0 openings. And there is no "supreme" argument to not disturb A0 from playing by its own from Initial Board position, as "it knows better what to do". I let Lc0 play as it wants from the start, and against SF10 + book it performed very poorly compared to the picked openings from the paper. Also, I did have diversity with the book, which they did not have in all their matches aside the TCEC match. I really cannot understand how one can take very seriously a match of 1000 games from 1 position.

Yes, it was maybe not a complete claim, but it was used as an argument to justify the chosen methodology.
Anyway, all this is maybe not that important, A0 is here, Lc0 is here, they are great, the paper is great, so I don't want to be too confrontational.

It's a general claim that using forced openings introduces bias depending on opening selection, which I think you agree with. My point is that, if there are openings that AZ or SF will never play itself into, including them will affect the result in a way that does not reflect the strengths of the engines at playing the game of chess (from start position). It's the cost of using any opening suite for diversity.

For example, if SF plays 1. e4 openings very poorly as white, but never plays it, should any 1. e4 openings be used to estimate SF's strength as white?

That's true, but the most striking in my examples was letting Lc0 play whatever it wants from the Initial Board position against SF + Book, so Lc0 was put to play Chess by its definition, but it sometimes doesn't like Chess too much, it likes Chess much more in the paper.

How deep is the book? I don’t think there’s any claim that NNs are superior to top opening theory. Just that, at times, they can play them with relatively surprising accuracy. I do think it’s an interesting result, but imo any elite book (if that’s what you used) should be limited to 10 ply or less, otherwise you’re mostly demonstrating what’s already clear, that Lc0 hasn’t surpassed current theory.

Well, for variety --- books - no good, set of opening positions, like TCEC ones - no good, only 1 initial board position and some picked "12 human openings" are good, which introduce huge distortions, surprise, hugely favoring A0 (Lc0). I am bored.

[jp]

How deep is the book? I don’t think there’s any claim that NNs are superior to top opening theory. Just that, at times, they can play them with relatively surprising accuracy. I do think it’s an interesting result, but imo any elite book (if that’s what you used) should be limited to 10 ply or less, otherwise you’re mostly demonstrating what’s already clear, that Lc0 hasn’t surpassed current theory.

Well, for variety --- books - no good, set of opening positions, like TCEC ones - no good, only 1 initial board position and some picked "12 human openings" are good, which introduce huge distortions, surprise, hugely favoring A0 (Lc0). I am bored.

Books are good. yanquis, a computer program doesn't need to surpass current theory to win a game.

[yanquis1972]

If the book is top notch and always selects the optimal move, no current program without any book (or armed with an inferior one) will win a match against SF10. I doubt SF10 would defeat SF8 under such conditions. That said sf plays excellently from any number of very low ply openings),

I do think the deepmind set is mostly useful for benchmarking — I’m not advocating no book or suggesting they’re optimal for testing (one question that should be answered before any test is what exactly it’s purpose is, and this debate suggests we assume the motive is to confirm the others prejudices).

I simply advocate using a shallow, broad book for NN testing against strong AB engines; if you don’t like deepminds 12 most popular set, change them. Eg force the 12 most popular super gm openings, or the favored openings of the last few world championships (or your favorites from any point in history), ideally truncating by move 5 or so.

[Uri Blass]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings (closed, less tactics). With "Leela Ratio" of about 2-3, Lc0 is comparable to A0 playing against SF (10 or 8).
rom the beginning by itself). If these things happen (I will post the next results in say 2 hours), A0 CANNOT steer every game into convenient opening. In fact, I knew that from my experience with Lc0, so for me the only reliable result was that from TCEC "outrageous" openings. But let's see the next results.

I think you misunderstood the sentence, it starts with an "If" and was certainly meant as conjunctive construction. Common consensus seems to be that the "fairest" match condition is starting from the initial position and let SF use whatever book and with whatever diversification it wants to use. Same for AZ. I start considering different starting positions as different disciplines. Because we have no clue if AZ would be able to avoid running into some/many of that positions, it's would be unfair testing AZ on a possibly very untrained battlefield.

See the results:

Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


Lc0 vs SF10 from "12 human openings":
Score of lc0_v19_11261 vs SF10: 8 - 5 - 27 [0.537] 40
Elo difference: 26.11 +/- 61.76

In the paper only the latter 2 are presented, but the first one, which is close to 120-150 Elo points lower for Lc0, without any hindering to its playing (it plays all by itself form the start position), is absent. And the almost deterministic "best Cerebellum book moves" with A0 diversification is again a practice playing into A0 strength. In fact the first result is the most relevant, as Lc0 "shouldn't be hindered by openings", and if the diversification comes from SF10, SF will be punished by Lc0 all by itself, right?

Aside from lack of diversification or a bit of diversification coming mostly from A0, all this relates too to the fact that A0 (Lc0) is a more specialized task solver than a regular engine, a thread I posted two or three months ago. So, take A0 away from what it specializes in, say with an opening book, and it will buckle its strength.

I think that for comparing with the paper you need to use the same time control as the paper if you have the same hardware(or equivalent time control that means adjusting the time control based on your hardware).
It is logical that BookX.bin can give advantage for Stockfish at short time control that is going to disappear at long time control.

Simple reason is that I expect the opening moves of LC0 to be improved at long time control when
I do not expect moves of stockfish in the opening to be improved at long time control because stockfish is going to read the same moves directly from the book.

[noobpwnftw]

If the book is top notch and always selects the optimal move, no current program without any book (or armed with an inferior one) will win a match against SF10. I doubt SF10 would defeat SF8 under such conditions. That said sf plays excellently from any number of very low ply openings),

I do think the deepmind set is mostly useful for benchmarking — I’m not advocating no book or suggesting they’re optimal for testing (one question that should be answered before any test is what exactly it’s purpose is, and this debate suggests we assume the motive is to confirm the others prejudices).

I simply advocate using a shallow, broad book for NN testing against strong AB engines; if you don’t like deepminds 12 most popular set, change them. Eg force the 12 most popular super gm openings, or the favored openings of the last few world championships (or your favorites from any point in history), ideally truncating by move 5 or so.

It is known that most engines do not perform or diversify openings very well, and that is why people were using books in the first place.
Maybe there just wasn't enough effort to beef up things like BF, and if people are down to the level that using a database to assist the engine is fine, then I see plenty of ways to "teach" the engines about openings, even if arbitrary openings are chosen, like I said, chances.

I have very good confidence in building such an opening book over the course of a few years on my garage hardware, and once we have a big enough statistically fitted database, I do not see how odds are favoring otherwise.

[Uri Blass]

If the book is top notch and always selects the optimal move, no current program without any book (or armed with an inferior one) will win a match against SF10. I doubt SF10 would defeat SF8 under such conditions. That said sf plays excellently from any number of very low ply openings),

I do think the deepmind set is mostly useful for benchmarking — I’m not advocating no book or suggesting they’re optimal for testing (one question that should be answered before any test is what exactly it’s purpose is, and this debate suggests we assume the motive is to confirm the others prejudices).

I simply advocate using a shallow, broad book for NN testing against strong AB engines; if you don’t like deepminds 12 most popular set, change them. Eg force the 12 most popular super gm openings, or the favored openings of the last few world championships (or your favorites from any point in history), ideally truncating by move 5 or so.

It is known that most engines do not perform or diversify openings very well, and that is why people were using books in the first place.
Maybe there just wasn't enough effort to beef up things like BF, and if people are down to the level that using a database to assist the engine is fine, then I see plenty of ways to "teach" the engines about openings, even if arbitrary openings are chosen, like I said, chances.

I have very good confidence in building such an opening book over the course of a few years on my garage hardware, and once we have a big enough statistically fitted database, I do not see how odds are favoring otherwise.

People are using books in the first place in order to save time on the clock and because maybe you can get better moves with more time.

The advantage of using book in every thinking game are
1)The expected quality of the book moves may be higher because the engine used more time to calculate the book moves before the game(the engine could not use more time during the game because it means losing on time)
2)Even if the quality of the book moves is the same as the quality of the calculated moves you still saved time on the clock by playing the book moves immediately.

[noobpwnftw]

People are using books in the first place in order to save time on the clock and because maybe you can get better moves with more time.

That we can adapt, provided that we have a book move, we can still let the engine search for whatever time it may take and only change the move during actual play, clock should remain unaffected.

[Laskos]

If AZ can always play into closed openings from start position no matter what the opponent does, why should its performance on open openings be reflected in its Elo rating?

It is a question of philosophy. As 100% of the practical use of chess engines consists of analysis one can argue that a chess engine should be able to play good chess in any (reasonable) position...

That statement of Matthew is not true. A0 cannot play always as it likes the openings (closed, less tactics). With "Leela Ratio" of about 2-3, Lc0 is comparable to A0 playing against SF (10 or 8).
rom the beginning by itself). If these things happen (I will post the next results in say 2 hours), A0 CANNOT steer every game into convenient opening. In fact, I knew that from my experience with Lc0, so for me the only reliable result was that from TCEC "outrageous" openings. But let's see the next results.

I think you misunderstood the sentence, it starts with an "If" and was certainly meant as conjunctive construction. Common consensus seems to be that the "fairest" match condition is starting from the initial position and let SF use whatever book and with whatever diversification it wants to use. Same for AZ. I start considering different starting positions as different disciplines. Because we have no clue if AZ would be able to avoid running into some/many of that positions, it's would be unfair testing AZ on a possibly very untrained battlefield.

See the results:

Lc0 No Book vs SF10 BookX.bin:
Score of lc0_v191_11261 vs SF10: 5 - 16 - 19 [0.362] 40
Elo difference: -98.07 +/- 79.65


Lc0 No Book vs SF10 No Book (Initial Board position):
Score of lc0_v19_11261 vs SF10: 12 - 6 - 22 [0.575] 40
Elo difference: 52.51 +/- 73.05


Lc0 vs SF10 from "12 human openings":
Score of lc0_v19_11261 vs SF10: 8 - 5 - 27 [0.537] 40
Elo difference: 26.11 +/- 61.76

In the paper only the latter 2 are presented, but the first one, which is close to 120-150 Elo points lower for Lc0, without any hindering to its playing (it plays all by itself form the start position), is absent. And the almost deterministic "best Cerebellum book moves" with A0 diversification is again a practice playing into A0 strength. In fact the first result is the most relevant, as Lc0 "shouldn't be hindered by openings", and if the diversification comes from SF10, SF will be punished by Lc0 all by itself, right?

Aside from lack of diversification or a bit of diversification coming mostly from A0, all this relates too to the fact that A0 (Lc0) is a more specialized task solver than a regular engine, a thread I posted two or three months ago. So, take A0 away from what it specializes in, say with an opening book, and it will buckle its strength.

I think that for comparing with the paper you need to use the same time control as the paper if you have the same hardware(or equivalent time control that means adjusting the time control based on your hardware).
It is logical that BookX.bin can give advantage for Stockfish at short time control that is going to disappear at long time control.

Simple reason is that I expect the opening moves of LC0 to be improved at long time control when
I do not expect moves of stockfish in the opening to be improved at long time control because stockfish is going to read the same moves directly from the book.

I explained this earlier. One role of the book is indeed replying instantly to Lc0 with reasonable moves (even possibly not the best) at which Lc0 has to spend some time. But the main role of a diversified book against Lc0 is destabilizing it with unfamiliar or tactical positions, where Lc0 (A0) buckles. And this destabilizing of Lc0 (A0) will not vanish even at infinite time control. Again, A0 (Lc0) are more specialized on the specific task they are trained to do, that is given initial chess board, to win the game. Regular engines are more general chess positions solvers, I am stating it for the third or fifth time on this forum.
And generally, most of your comments are "we cannot know this, we cannot know that". No, you, and only you cannot know many, many things, and I feel sorry for you.

[Milos]

Simple reason is that I expect the opening moves of LC0 to be improved at long time control

You can expect whatever you want, but there is zero factual evidence that supports your expectation in this particular case.