If you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.Dann Corbit wrote: ↑Fri Aug 30, 2019 3:08 am It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo here:
https://deepmind.com/blog/article/alpha ... ogi-and-go
Komodo vs. Larry K on chess.com
Moderators: hgm, Rebel, chrisw
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Komodo vs. Larry K on chess.com
Komodo rules!
-
- Posts: 4556
- Joined: Tue Jul 03, 2007 4:30 am
Re: Komodo vs. Larry K on chess.com
That's very interesting. I wonder how the chess rating lists would look like with draws removed (some kind of chess variant where if the game ends in draw it doesn't count.)
Your beliefs create your reality, so be careful what you wish for.
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Komodo vs. Larry K on chess.com
There were also complaints from the Shogi/Elmo community about how they used Elmo there, so the comparison would need to judge whether the Elmo crippling in that match was the same disadvantage as the SF8 crippling.lkaufman wrote: ↑Fri Aug 30, 2019 6:35 amIf you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.Dann Corbit wrote: ↑Fri Aug 30, 2019 3:08 am It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo
-
- Posts: 4556
- Joined: Tue Jul 03, 2007 4:30 am
Re: Komodo vs. Larry K on chess.com
(except SF wasn't crippled because Alpha played under the same conditions)
Your beliefs create your reality, so be careful what you wish for.
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Komodo vs. Larry K on chess.com
Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
Komodo rules!
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Komodo vs. Larry K on chess.com
Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.lkaufman wrote: ↑Sat Aug 31, 2019 7:29 am Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.
When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Komodo vs. Larry K on chess.com
The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.jp wrote: ↑Sat Aug 31, 2019 2:05 pmDo you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.lkaufman wrote: ↑Sat Aug 31, 2019 7:29 am Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.
When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.
Komodo rules!
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Komodo vs. Larry K on chess.com
Does the Komodo team have a recommended "best" hash size for a given number of nodes per move?
For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Komodo vs. Larry K on chess.com
Not exactly, but we do have guidelines in our readme for how to set Hash size. 1 GB hash for a monster machine is so obviously silly that I thought at first it was a typo, maybe they meant 1 GB per thread or core, but apparently not. I can only guess that no one knew much about this or thought it through.jp wrote: ↑Sat Aug 31, 2019 5:56 pmDoes the Komodo team have a recommended "best" hash size for a given number of nodes per move?
For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).
Komodo rules!
-
- Posts: 10316
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: Komodo vs. Larry K on chess.com
The standard position is playable between top engines as long as there are wins.lkaufman wrote: ↑Sat Aug 31, 2019 5:51 pmThe tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.jp wrote: ↑Sat Aug 31, 2019 2:05 pmDo you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.lkaufman wrote: ↑Sat Aug 31, 2019 7:29 am Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.
When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.
<5% wins is still not 0% wins and it is possible to give white more time to reduce the percentage of draws.
It may be interesting to know the percentage of draws in a match with no opening book from the opening position between stockfish and lc0 when white get 10 minutes per move against 1 minutes per move by black(of course no pondering).