Chess World to Google Deep Mind..Prove You beat Stockfish 8!

[pilgrimdan]

It should have to perceive that the match circumstances during games between A0 and Stockfish is an unimportant thing. The essence is the team of DeepMind have proved they can make a powerful system based on neural networks which can solve such complicated tasks like winning go and chess games against very strong opponents.

and chess games against very strong <crippled> opponents...

that statement is more accurate...

people are focussed on the wrong metric - it's not the settings, it's not the Sf version used, it's not the time control, -- just look at the games and the moves played by A0 - some of the move played appeared to come from another planet.., consider what they also did with AGo and it's impressive and its believable, it was clearly unexpected and apparently for many people , not believable - something most have been drastically wrong to the point where SF 8 was crippled - why use the word crippled , because the results were over overwhelmingly in favor AO such that SF had to be crippled... it was not done the way many of us would have to like seen it be done , but it is what it is ..I also understand why people feel the way ...it's called emotion. SF is open source , everybody gets it for free, hundreds of people have volunteered either time and money ( time, computers and electricity are not free) and it was and still is the strongest free open source engine available, SF brings a lot of positives to the chess community and people appreciate that it so much such that it has become the favorite son of many chess enthusiasts and rightfully so and then big bad google comes with big bad money, hires a bunch of really big bad smart people, spends a few million on big bad hardware and then blows away the white knight SF , and people get upset ...

for me personally, I want to see some AO games played the way we would want them to see them play - but not because I think it would then be even and SF would have its revenge - but purely from standpoint that it would think it would be interesting and fun to watch...anyway I feel your pain too as the keeper of SF-McB - lets just hope there are some more games we can get to watch and analyze..

I basically agree with what you are saying...

' ... just look at the games and the moves played by A0 - some of the move played appeared to come from another planet.. '

what I <still> don't understand...

if play looked like it came from another planet...

then why couldn't the 'play' be done against a fully functional Stockfish...

if it still would have produced play from another planet ... then fine ...

that would have been much more satisfying...

what am I missing...

[M ANSARI]

Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.

Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.

Hmmm ... good point and a valid possibility. But let's wait and see. It is very strange why not all the 100 games were not released. I think we can gain more information regarding real strength with the drawn games. There doesn't seem to be a valid reason why the games have not all been released. I will say one thing though ... chess wise the moves played by A0 are really a huge achievement ... I really don't think any other chess engine today could have reproduced such moves no matter how many cores. But I would like to see all the games to get through all the hype.

[Leto]

Reviewing the drawn games would be useful, and presumably there is no obstacle to theAlphaZero team making these available on-line.

Ofc that there is an obstacle. A0 has zero variability in search or evaluation. The only variability there comes on 8 threads waiting in the queue of each TPU for evaluation. Since A0 uses UTC and not alpha-beta this has minuscule impact on variability.
All other randomness comes from SMP search of SF. Considering TC was fixed time per move, it is very probable that many games are actually similar if not even identical. That would be the only logical explanation why they didn't publish even all the winning games. Many wins basically repeat and there are many more almost identical boring draws.
If they published all the games ppl would see the scam.

Hmmm ... good point and a valid possibility. But let's wait and see. It is very strange why not all the 100 games were not released. I think we can gain more information regarding real strength with the drawn games. There doesn't seem to be a valid reason why the games have not all been released. I will say one thing though ... chess wise the moves played by A0 are really a huge achievement ... I really don't think any other chess engine today could have reproduced such moves no matter how many cores. But I would like to see all the games to get through all the hype.

I would think they'd release all the games when their paper gets published, that will be a while. What we've seen so far is just a preview of their paper.

[Dann Corbit]

It should have to perceive that the match circumstances during games between A0 and Stockfish is an unimportant thing. The essence is the team of DeepMind have proved they can make a powerful system based on neural networks which can solve such complicated tasks like winning go and chess games against very strong opponents.

and chess games against very strong <crippled> opponents...

that statement is more accurate...

people are focussed on the wrong metric - it's not the settings, it's not the Sf version used, it's not the time control, -- just look at the games and the moves played by A0 - some of the move played appeared to come from another planet.., consider what they also did with AGo and it's impressive and its believable, it was clearly unexpected and apparently for many people , not believable - something most have been drastically wrong to the point where SF 8 was crippled - why use the word crippled , because the results were over overwhelmingly in favor AO such that SF had to be crippled... it was not done the way many of us would have to like seen it be done , but it is what it is ..I also understand why people feel the way ...it's called emotion. SF is open source , everybody gets it for free, hundreds of people have volunteered either time and money ( time, computers and electricity are not free) and it was and still is the strongest free open source engine available, SF brings a lot of positives to the chess community and people appreciate that it so much such that it has become the favorite son of many chess enthusiasts and rightfully so and then big bad google comes with big bad money, hires a bunch of really big bad smart people, spends a few million on big bad hardware and then blows away the white knight SF , and people get upset ...

for me personally, I want to see some AO games played the way we would want them to see them play - but not because I think it would then be even and SF would have its revenge - but purely from standpoint that it would think it would be interesting and fun to watch...anyway I feel your pain too as the keeper of SF-McB - lets just hope there are some more games we can get to watch and analyze..

I basically agree with what you are saying...

' ... just look at the games and the moves played by A0 - some of the move played appeared to come from another planet.. '

what I <still> don't understand...

if play looked like it came from another planet...

then why couldn't the 'play' be done against a fully functional Stockfish...

if it still would have produced play from another planet ... then fine ...

that would have been much more satisfying...

what am I missing...

There was nothing wrong with the SF it played against.
That's what you are missing.

[Cardoso]

I completely agree with Dann and Mike.
I don't think there is something seriously wrong with the SF setup. If someone says SF's setup could be a bit better I might agree, but that wouldn't change the final outcome of the match. And people's insistence that SF should have an human made book isn't it just an admition SF is not so good at openings?
A0 learned the entire game by itself including the openings without human intervention, this alone is an incredible achievement.
Even if the results were even DeepMind would achieved an astonishing feat.
With a radically different aproach they surprised everybody with a generic learning set of algorithms.
Remember DeepMind / Google has the resources to make a system similar to Deep Blue, only much stronger. If they can make TPUs I'm sure they could make a chess chip similar to what DB had, but instead of searching only 4 plies + QS they could go way beyond that and instead of 20Mhz they could have 2GHz chess chips with a transposition table wich DB's chess chips didn't have. They would still blow SF out of the water with such a system if all they wanted was to conquer chess.
But this way would cause even more complaints and criticism of an unfair match because of the hardware differences.
I must admit I was completely surprised. I didn't believe this could be done so soon (although I allways thought in terms of domestic hardware) because of the game differences between chess and Go (Go having only 1 piece for white and 1 piece for black I allways thought it was easier to spot patterns, and not so easy in chess).
So, nay sayers, get over it, Google have done something really impressive and a stronger SF woudn't change that. BTW if a 100 ELO stronger SF would have been used do you guys think that would translate to 100 ELO against A0? SF testing framework is allways done against a previous version of SF with all it's strengths and weaknesses.
How about counting the hours/equipment/electricity devoted to improve SF?
I suspect the SF team has burned more watts than DeepMind for a chess project.
This is not to downplay SF and their developers wich I respect a lot.
Just please stop the whining and get over it.
Better yet: improve SF so it doesn't loose again against A0 (if another match will ever be played).

[pilgrimdan]

I completely agree with Dann and Mike.
I don't think there is something seriously wrong with the SF setup. If someone says SF's setup could be a bit better I might agree, but that wouldn't change the final outcome of the match. And people's insistence that SF should have an human made book isn't it just an admition SF is not so good at openings?
A0 learned the entire game by itself including the openings without human intervention, this alone is an incredible achievement.
Even if the results were even DeepMind would achieved an astonishing feat.
With a radically different aproach they surprised everybody with a generic learning set of algorithms.
Remember DeepMind / Google has the resources to make a system similar to Deep Blue, only much stronger. If they can make TPUs I'm sure they could make a chess chip similar to what DB had, but instead of searching only 4 plies + QS they could go way beyond that and instead of 20Mhz they could have 2GHz chess chips with a transposition table wich DB's chess chips didn't have. They would still blow SF out of the water with such a system if all they wanted was to conquer chess.
But this way would cause even more complaints and criticism of an unfair match because of the hardware differences.
I must admit I was completely surprised. I didn't believe this could be done so soon (although I allways thought in terms of domestic hardware) because of the game differences between chess and Go (Go having only 1 piece for white and 1 piece for black I allways thought it was easier to spot patterns, and not so easy in chess).
So, nay sayers, get over it, Google have done something really impressive and a stronger SF woudn't change that. BTW if a 100 ELO stronger SF would have been used do you guys think that would translate to 100 ELO against A0? SF testing framework is allways done against a previous version of SF with all it's strengths and weaknesses.
How about counting the hours/equipment/electricity devoted to improve SF?
I suspect the SF team has burned more watts than DeepMind for a chess project.
This is not to downplay SF and their developers wich I respect a lot.
Just please stop the whining and get over it.
Better yet: improve SF so it doesn't loose again against A0 (if another match will ever be played).

okay then ... you guys have much more knowledge than me...

so ... I will go with what you say...

the moves are amazing ... that is true ...

[Ovyron]

And people's insistence that SF should have an human made book isn't it just an admition SF is not so good at openings?

Yeah, SF is not good at openings.

There's still dispute about whether SF was crippled by the number of CPUs, the hash, endgame tablebases or fixed time control per move, perhaps it wasn't crippled by all that.

But it's clear SF is so bad at openings that it'd have played a much stronger match with an opening book like Cerebellum.

And I know, because I have tested Stockfish vs Stockfish, the former with Cerebellum, and the latter with a big variety of Bin books, including Jeroen Noomen's best effort and a huge one by Dann Corbit, and Cerebellum destroyed them all.

And this is with books, I wouldn't be surprised if Stockfish with Cerebellum defeated Stockfish without book by a bigger margin than Alpha Zero, it's that bad.

However, one can't really expect the scientists to use such a book, it's not even officially part of Stockfish.

And this goes for all the engines, in general, all of them are bad in the opening, the only decent ones are those that are able to read book contents and modify their evaluation (then Shredder is best at opening) but it's still using an internal book.

Perhaps A0 is not different, and it'd have destroyed Stockfish if it used an opening book. Remember A0 could have destroyed SF if instead of training against itself, it trained against it, so that instead of maximizing wins in general, it'd mazimize wins against Stockfish. Then it's clear A0 would play stronger chess by playing those openings from a book.

Is this really the best white has for winning chances?

[d]rn1q1rk1/pbppbppp/1p2pn2/3P4/2P5/5NP1/PP2PPBP/RNBQ1RK1 b - -

[syzygy]

And people's insistence that SF should have an human made book isn't it just an admition SF is not so good at openings?

Yeah, SF is not good at openings.

There's still dispute about whether SF was crippled by the number of CPUs, the hash, endgame tablebases or fixed time control per move, perhaps it wasn't crippled by all that.

But it's clear SF is so bad at openings that it'd have played a much stronger match with an opening book like Cerebellum.

If you want to test the engine, then using the Cerebellum book seems a bit out of place. And there would no reason not to let AlphaZero use the same book. What it would really mean is that you'd let the engines play out the ending positions of the book. You wouldn't even need to fire up the engines, you could just let some GUI select the moves from the Cerebellum book.

It would have made sense to let the engines start from 50 2-move openings, each engine playing each position once with white and once with black. That would have ensured the games would all be unique and would have ruled out some sort of repeated opening luck.

But as it is there is no real indication that it was just opening luck. AlphaZero also won the 12 matches played from other opening positions.

[Evert]

But he has trouble believing that it was possible to rival in 4 hours of computation time what mankind took 50 years to do (but it's actually closer to 70 years).

It helps to consider that "4 hours" is wall time, not CPU time. The CPU time spent on training is much longer (can't be bothered to look it up at this hour though). It's still impressive, of course.

[Ovyron]

You wouldn't even need to fire up the engines, you could just let some GUI select the moves from the Cerebellum book.

Cerebellum is like A0, it always chooses the same moves, from both sides, and always ends up in this position:

[d]r5k1/1p4p1/1bp3Rp/p6r/3p1p2/2P5/PP3PKN/R1B5 b - -

So SF+Cerebellum Vs. A0+Cerebellum would have been 100 games played from there... I agree, not a very interesting proposition...

But as it is there is no real indication that it was just opening luck. AlphaZero also won the 12 matches played from other opening positions.

What we want to know is if, say, Alpha Zero was tested under CCRL conditions, if it would be at the top of the list, and if so, by how much would its advantage be to the second engine. Are there tests we can make to approximate this data?