For how long will Stockfish be competitive versus the best NN ?

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: For how long will Stockfish be competitive versus the best NN ?

Post by zullil »

zullil wrote: Thu Apr 16, 2020 4:29 pm
Milos wrote: Thu Apr 16, 2020 2:56 pm
zullil wrote: Thu Apr 16, 2020 2:10 pm
Alayan wrote: Thu Apr 16, 2020 1:58 pm This is Stockfish-Komodo with black and white reversed, isn't it ?

Then Bg5 is guaranteed to lose, and Ke2 likely holds (or at least, we don't know how to beat it).
Yes, it is. The original position was discussed at this forum, so I intentionally obfuscated the position so that searching the FEN would not be immediately useful. Here's what I currently see after Bg5:

-25.30 51... Ba5 52. Ke4 Bb6 53. Bc1 Bc5 54. Kf4 Be7 55. Bb2 Bf6 56. Ba1 Kd6 57. Kg3 Ke5 58. Bb2 Be7 59. Bc1 Ke6 60. Bb2 Bd6+ 61. Kf3 Kf5 62. Bc1 Be7 63. Kg3 Bc5 64. Kh3 Bd6 65. Kg2 Kg4 66. Bb2 Bc5 67. Kf2 Be7 68. Kf1 Bxh4 69. Ke2 Kf5 70. Ba1 Ke5 71. Kd1 Bg5 72. Ke2 Bc1 73. Ke1 Bxa3 74. Kf2 Bb4 75. Bb2 a3 76. Ba1 Bc3 77. Bxc3 dxc3 78. Ke3 c2 79. d4+ Ke6 80. d5+ Kd6 (depth 51, 1:01:15)

That's after a deep Stockfish search of the original position with MultiPV = 2, followed by forward/backward analysis of the Bg5 line.

20 threads, 64 GB hash table.
I really don't know what are you taking about. SFdev, 6-men, MultiPV=2, depth 80, Bg5 and Kd2 are a transposition so lines are identical, score -1.07.
Line is 51. Bg5 Ba5 52. Ke2 Bc7 or 51.Ke2 Ba5 52. Bg5 Bc7
Please let us know if you discover that your initial analysis is incorrect.
Here's what Stockfish currently shows with MultiPV = 3 after Bg5??? Ba5:

-64.41 52. Bc1 Bd8 53. Kg3 Bc7+ 54. Kf3 Bd6 55. Kg2 Be7 56. Kg3 Kd7 57. Bb2 Kc6 58. Bc1 b6 59. Bb2 Kc5 60. Bc1 Kd6 61. Bg5 Bf8 62. Bc1 Ke6 63. Bb2 Be7 64. Kf4 Bxh4 65. Bxd4 Be7 66. Bb2 Kd6 67. d4 Kc6 68. Kf5 h4 69. Kg4 b5 70. c5 Kd5 71. Kh3 Kc4 72. Kh2 Bf6 73. Kh1 Bd8 74. Kg2 Kd5 75. Bc3 Bg5 76. Bb2 Bf6 77. Bc3 Bd8 78. Kg1 Kc4 79. Ba1 Be7 80. c6 Bd8 81. c7 Bxc7 (depth 60, 1:56:02)

-71.23 52. Ke4 Bb6 53. Kf3 Bc5 54. Bc1 Be7 55. Kg3 Kd7 56. Bb2 Kd6 57. Ba1 Kc6 58. Bb2 b6 59. Bc1 Kc7 60. Bb2 Kd6 61. Bc1 Kc6 62. Bb2 b5 63. cxb5+ Kxb5 64. Bc1 Kc6 65. Kh3 Kc5 66. Kg3 Kd5 67. Bb2 Ke5 68. Bc1 Kf5 69. Bb2 Bd6+ 70. Kf3 Bf4 71. Bxd4 Bd6 72. Bb2 Be7 73. Kg3 Ke6 74. Kh3 Kd7 75. Kg3 Kd6 76. Bd4 Ke6 77. Bb2 Kf5 78. Bc1 Bf6 79. Bd2 Bd4 80. Bc1 Be5+ 81. Kf3 Bf6 82. Kg3 Ke6 83. Kh3 (depth 60, 1:56:02)

-87.09 52. Kg2 Bb6 53. Kg3 Bc5 54. Bc1 Kf5 55. Bb2 Bd6+ 56. Kf3 Be7 57. Kg3 Ke6 58. Bc1 Ke5 59. Bb2 b6 60. Bc1 Ke6 61. Bb2 Kd6 62. Bc1 Kc6 63. Bb2 b5 64. cxb5+ Kxb5 65. Bc1 Kb6 66. Kh3 Kc5 67. Bg5 Bd6 68. Bc1 Kd5 69. Kg2 Bf8 70. Kh2 Kc6 71. Kg2 Be7 72. Kh3 Bc5 73. Kg2 Kd7 74. Kg3 Ke6 75. Kf4 Be7 76. Kg3 Kf5 77. Bb2 Bd6+ 78. Kf3 Bf4 79. Bxd4 Bd6 80. Bb2 Ke6 81. Kg2 Be7 82. Kh3 Kd6 (depth 60, 1:56:02)
Milos
Posts: 4190
Joined: Wed Nov 25, 2009 1:47 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by Milos »

zullil wrote: Thu Apr 16, 2020 9:24 pm
zullil wrote: Thu Apr 16, 2020 4:29 pm
Milos wrote: Thu Apr 16, 2020 2:56 pm
zullil wrote: Thu Apr 16, 2020 2:10 pm
Alayan wrote: Thu Apr 16, 2020 1:58 pm This is Stockfish-Komodo with black and white reversed, isn't it ?

Then Bg5 is guaranteed to lose, and Ke2 likely holds (or at least, we don't know how to beat it).
Yes, it is. The original position was discussed at this forum, so I intentionally obfuscated the position so that searching the FEN would not be immediately useful. Here's what I currently see after Bg5:

-25.30 51... Ba5 52. Ke4 Bb6 53. Bc1 Bc5 54. Kf4 Be7 55. Bb2 Bf6 56. Ba1 Kd6 57. Kg3 Ke5 58. Bb2 Be7 59. Bc1 Ke6 60. Bb2 Bd6+ 61. Kf3 Kf5 62. Bc1 Be7 63. Kg3 Bc5 64. Kh3 Bd6 65. Kg2 Kg4 66. Bb2 Bc5 67. Kf2 Be7 68. Kf1 Bxh4 69. Ke2 Kf5 70. Ba1 Ke5 71. Kd1 Bg5 72. Ke2 Bc1 73. Ke1 Bxa3 74. Kf2 Bb4 75. Bb2 a3 76. Ba1 Bc3 77. Bxc3 dxc3 78. Ke3 c2 79. d4+ Ke6 80. d5+ Kd6 (depth 51, 1:01:15)

That's after a deep Stockfish search of the original position with MultiPV = 2, followed by forward/backward analysis of the Bg5 line.

20 threads, 64 GB hash table.
I really don't know what are you taking about. SFdev, 6-men, MultiPV=2, depth 80, Bg5 and Kd2 are a transposition so lines are identical, score -1.07.
Line is 51. Bg5 Ba5 52. Ke2 Bc7 or 51.Ke2 Ba5 52. Bg5 Bc7
Please let us know if you discover that your initial analysis is incorrect.
Here's what Stockfish currently shows with MultiPV = 3 after Bg5??? Ba5:

-64.41 52. Bc1 Bd8 53. Kg3 Bc7+ 54. Kf3 Bd6 55. Kg2 Be7 56. Kg3 Kd7 57. Bb2 Kc6 58. Bc1 b6 59. Bb2 Kc5 60. Bc1 Kd6 61. Bg5 Bf8 62. Bc1 Ke6 63. Bb2 Be7 64. Kf4 Bxh4 65. Bxd4 Be7 66. Bb2 Kd6 67. d4 Kc6 68. Kf5 h4 69. Kg4 b5 70. c5 Kd5 71. Kh3 Kc4 72. Kh2 Bf6 73. Kh1 Bd8 74. Kg2 Kd5 75. Bc3 Bg5 76. Bb2 Bf6 77. Bc3 Bd8 78. Kg1 Kc4 79. Ba1 Be7 80. c6 Bd8 81. c7 Bxc7 (depth 60, 1:56:02)

-71.23 52. Ke4 Bb6 53. Kf3 Bc5 54. Bc1 Be7 55. Kg3 Kd7 56. Bb2 Kd6 57. Ba1 Kc6 58. Bb2 b6 59. Bc1 Kc7 60. Bb2 Kd6 61. Bc1 Kc6 62. Bb2 b5 63. cxb5+ Kxb5 64. Bc1 Kc6 65. Kh3 Kc5 66. Kg3 Kd5 67. Bb2 Ke5 68. Bc1 Kf5 69. Bb2 Bd6+ 70. Kf3 Bf4 71. Bxd4 Bd6 72. Bb2 Be7 73. Kg3 Ke6 74. Kh3 Kd7 75. Kg3 Kd6 76. Bd4 Ke6 77. Bb2 Kf5 78. Bc1 Bf6 79. Bd2 Bd4 80. Bc1 Be5+ 81. Kf3 Bf6 82. Kg3 Ke6 83. Kh3 (depth 60, 1:56:02)

-87.09 52. Kg2 Bb6 53. Kg3 Bc5 54. Bc1 Kf5 55. Bb2 Bd6+ 56. Kf3 Be7 57. Kg3 Ke6 58. Bc1 Ke5 59. Bb2 b6 60. Bc1 Ke6 61. Bb2 Kd6 62. Bc1 Kc6 63. Bb2 b5 64. cxb5+ Kxb5 65. Bc1 Kb6 66. Kh3 Kc5 67. Bg5 Bd6 68. Bc1 Kd5 69. Kg2 Bf8 70. Kh2 Kc6 71. Kg2 Be7 72. Kh3 Bc5 73. Kg2 Kd7 74. Kg3 Ke6 75. Kf4 Be7 76. Kg3 Kf5 77. Bb2 Bd6+ 78. Kf3 Bf4 79. Bxd4 Bd6 80. Bb2 Ke6 81. Kg2 Be7 82. Kh3 Kd6 (depth 60, 1:56:02)
Yes Bg5 is definitely loosing, Kd2 seems to hold. However, one can't see that from MultiPV from the root without going really deep. One needs to guide the search. But instead of spending 2h on 20 threads very fast machine, one can probably make the same conclusion with the same machine in 20 minutes by going back and forth and some intuition.
This is still not a very good example to show that just straight search from the root gives result faster than in case one manually explores the position.
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: For how long will Stockfish be competitive versus the best NN ?

Post by zullil »

Milos wrote: Thu Apr 16, 2020 10:40 pm
zullil wrote: Thu Apr 16, 2020 9:24 pm
zullil wrote: Thu Apr 16, 2020 4:29 pm
Milos wrote: Thu Apr 16, 2020 2:56 pm
zullil wrote: Thu Apr 16, 2020 2:10 pm
Alayan wrote: Thu Apr 16, 2020 1:58 pm This is Stockfish-Komodo with black and white reversed, isn't it ?

Then Bg5 is guaranteed to lose, and Ke2 likely holds (or at least, we don't know how to beat it).
Yes, it is. The original position was discussed at this forum, so I intentionally obfuscated the position so that searching the FEN would not be immediately useful. Here's what I currently see after Bg5:

-25.30 51... Ba5 52. Ke4 Bb6 53. Bc1 Bc5 54. Kf4 Be7 55. Bb2 Bf6 56. Ba1 Kd6 57. Kg3 Ke5 58. Bb2 Be7 59. Bc1 Ke6 60. Bb2 Bd6+ 61. Kf3 Kf5 62. Bc1 Be7 63. Kg3 Bc5 64. Kh3 Bd6 65. Kg2 Kg4 66. Bb2 Bc5 67. Kf2 Be7 68. Kf1 Bxh4 69. Ke2 Kf5 70. Ba1 Ke5 71. Kd1 Bg5 72. Ke2 Bc1 73. Ke1 Bxa3 74. Kf2 Bb4 75. Bb2 a3 76. Ba1 Bc3 77. Bxc3 dxc3 78. Ke3 c2 79. d4+ Ke6 80. d5+ Kd6 (depth 51, 1:01:15)

That's after a deep Stockfish search of the original position with MultiPV = 2, followed by forward/backward analysis of the Bg5 line.

20 threads, 64 GB hash table.
I really don't know what are you taking about. SFdev, 6-men, MultiPV=2, depth 80, Bg5 and Kd2 are a transposition so lines are identical, score -1.07.
Line is 51. Bg5 Ba5 52. Ke2 Bc7 or 51.Ke2 Ba5 52. Bg5 Bc7
Please let us know if you discover that your initial analysis is incorrect.
Here's what Stockfish currently shows with MultiPV = 3 after Bg5??? Ba5:

-64.41 52. Bc1 Bd8 53. Kg3 Bc7+ 54. Kf3 Bd6 55. Kg2 Be7 56. Kg3 Kd7 57. Bb2 Kc6 58. Bc1 b6 59. Bb2 Kc5 60. Bc1 Kd6 61. Bg5 Bf8 62. Bc1 Ke6 63. Bb2 Be7 64. Kf4 Bxh4 65. Bxd4 Be7 66. Bb2 Kd6 67. d4 Kc6 68. Kf5 h4 69. Kg4 b5 70. c5 Kd5 71. Kh3 Kc4 72. Kh2 Bf6 73. Kh1 Bd8 74. Kg2 Kd5 75. Bc3 Bg5 76. Bb2 Bf6 77. Bc3 Bd8 78. Kg1 Kc4 79. Ba1 Be7 80. c6 Bd8 81. c7 Bxc7 (depth 60, 1:56:02)

-71.23 52. Ke4 Bb6 53. Kf3 Bc5 54. Bc1 Be7 55. Kg3 Kd7 56. Bb2 Kd6 57. Ba1 Kc6 58. Bb2 b6 59. Bc1 Kc7 60. Bb2 Kd6 61. Bc1 Kc6 62. Bb2 b5 63. cxb5+ Kxb5 64. Bc1 Kc6 65. Kh3 Kc5 66. Kg3 Kd5 67. Bb2 Ke5 68. Bc1 Kf5 69. Bb2 Bd6+ 70. Kf3 Bf4 71. Bxd4 Bd6 72. Bb2 Be7 73. Kg3 Ke6 74. Kh3 Kd7 75. Kg3 Kd6 76. Bd4 Ke6 77. Bb2 Kf5 78. Bc1 Bf6 79. Bd2 Bd4 80. Bc1 Be5+ 81. Kf3 Bf6 82. Kg3 Ke6 83. Kh3 (depth 60, 1:56:02)

-87.09 52. Kg2 Bb6 53. Kg3 Bc5 54. Bc1 Kf5 55. Bb2 Bd6+ 56. Kf3 Be7 57. Kg3 Ke6 58. Bc1 Ke5 59. Bb2 b6 60. Bc1 Ke6 61. Bb2 Kd6 62. Bc1 Kc6 63. Bb2 b5 64. cxb5+ Kxb5 65. Bc1 Kb6 66. Kh3 Kc5 67. Bg5 Bd6 68. Bc1 Kd5 69. Kg2 Bf8 70. Kh2 Kc6 71. Kg2 Be7 72. Kh3 Bc5 73. Kg2 Kd7 74. Kg3 Ke6 75. Kf4 Be7 76. Kg3 Kf5 77. Bb2 Bd6+ 78. Kf3 Bf4 79. Bxd4 Bd6 80. Bb2 Ke6 81. Kg2 Be7 82. Kh3 Kd6 (depth 60, 1:56:02)
Yes Bg5 is definitely loosing, Kd2 seems to hold. However, one can't see that from MultiPV from the root without going really deep. One needs to guide the search. But instead of spending 2h on 20 threads very fast machine, one can probably make the same conclusion with the same machine in 20 minutes by going back and forth and some intuition.
This is still not a very good example to show that just straight search from the root gives result faster than in case one manually explores the position.
It seems to be enough to reveal that Ovyron needs better hardware. :wink:
jp
Posts: 1470
Joined: Mon Apr 23, 2018 7:54 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by jp »

zullil wrote: Thu Apr 16, 2020 11:41 pm It seems to be enough to reveal that Ovyron needs better hardware. :wink:
"Needs" is a strong word. But this was a better test than the mate-in-N test.
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: For how long will Stockfish be competitive versus the best NN ?

Post by zullil »

jp wrote: Thu Apr 16, 2020 11:43 pm
zullil wrote: Thu Apr 16, 2020 11:41 pm It seems to be enough to reveal that Ovyron needs better hardware. :wink:
"Needs" is a strong word. But this was a better test than the mate-in-N test.
Yes, no one really needs much of anything in this world.

I gave him a position to test his hardware/method. His initial attempt was a dramatic failure. I don't really care, one way or the other. He can try the test again, perhaps spending more time, and see if he ever succeeds. Maybe he will.
User avatar
Ovyron
Posts: 4556
Joined: Tue Jul 03, 2007 4:30 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by Ovyron »

zullil wrote: Thu Apr 16, 2020 11:41 pm It seems to be enough to reveal that Ovyron needs better hardware. :wink:
No, one of two things are true:

- I'd have spent more than 15 minutes on the position and found that Bg5 was losing and would have avoided it. "I don't need better hardware" stands because if with faster hardware I'd have avoided it, I'd also have avoided it if I let my current hardware analyze for that long (what Dann Corbit is saying, more time = faster hardware)

- I'd have done the exact same thing I did (just, much faster) and wrongly concluded that Bg5 was drawn EVEN WITH BIGGER HARDWARE. In this case, bigger hardware would be useless to me because I wouldn't have known how to use it, as Alayan was saying on the other thread.

Anyway, good work at indeed trapping me, I just want to know if you'd feel capable of driving a game into one of these positions and making me lose live in a game, so we can arrange something. Probably to make it interesting we set some time limits for me based on the honor system, like I promise to never analyze a move for more than 1 hour, and knowing about my hardware and limited time, you drive to a position that makes me lose the game. The kind of game Zenmastur left me hungry for.
jp
Posts: 1470
Joined: Mon Apr 23, 2018 7:54 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by jp »

Ovyron wrote: Fri Apr 17, 2020 6:20 am Anyway, good work at indeed trapping me, I just want to know if you'd feel capable of driving a game into one of these positions and making me lose live in a game, so we can arrange something.
Again, even if it were true that you could avoid "being driven into one of these positions", you don't know that your effort to avoid those positions did not make you draw lots of games you would have won if you had more options.
User avatar
Ovyron
Posts: 4556
Joined: Tue Jul 03, 2007 4:30 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by Ovyron »

I have yet to see anyone with big hardware demonstrating that they play into better positions because they have more options.

Please compare:

Ovyron - Has slow hardware. Playing the King's Gambit would be unsustainable because of big dangers that are there for white that'd require too much work.

Zenmastur - Has big hardware. Playing the King's Gambit is no problem because there's no danger whatsoever. The king's gambit is an option.

Outcome:

Zenmastur plays the King's Gambit.

He has the hardware to play it without problems, but I reckon he'd have given me a much better game if it wasn't an option.

Sometimes being restricted makes you play best. I'd have played 2.Nf3 instead of 2.f4, wasn't 2.Nf3 a better move?
jp
Posts: 1470
Joined: Mon Apr 23, 2018 7:54 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by jp »

KG was a very bad choice. Everyone knows that.
User avatar
Ovyron
Posts: 4556
Joined: Tue Jul 03, 2007 4:30 am

Re: For how long will Stockfish be competitive versus the best NN ?

Post by Ovyron »

A choice that was made BECAUSE of big hardware. Big hardware can make people do that, it's not a guarantee of better play, it just gives more options, but more often than not the extra playable options aren't better and the person playing them overestimates the complications they create.

I can see myself getting much faster hardware years ago and playing a lot more wild lines like the KG and because of the lower quality in openings having the same or a worse record than I have now.

For the question about Stockfish remaining competitive against the NNs, it'll depend on who uses them and how they're used, recent examples on this forum have included using NNs to save 1.g4 and using Stockfish to play the King's Gambit, so perhaps we're at the point where chess mainlines have gotten boring to discuss and they just lead to draws (regardless of hardware!)