Interesting article about strategy to defeat NN Go programs

[jdart]

https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.

[dkappe]

https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.

Very interesting. I hope the leela project gives this a try to uncover some of its weaknesses.

[lkaufman]

https://arstechnica.com/information-tec ... -amateurs/

Reportedly if a human uses a strategy not seen in the NN training games it is possible to defeat a strong NN program, even though that strategy is not really that good and does not work well against strong human players.

Well, it sounded interesting from the description, but when I read it I see that it basically amounts to the program not knowing the rules; it is ridiculous to "pass" in GO if there are useful moves remaining and the game is not yet won. The program just needs to be told to determine what the score would be if the game ended now, and not to pass if it's not 100% won. It's the same as the chess programs that would troll with queen and rook vs. bare king by waiting until move 49 of the 50 move rule to mate (as many NNs used to do), but without even knowing the 50 move rule. So nothing general to learn here, just they need to fix that bug in the program. In other words, an estimated 99.999% chance of winning is still not as good as an actual win, and obviously much worse than a loss, which is what simple counting would determine. Probably the problem is that the rules are a big vague for how to count when a game is terminated too early. But this is an issue for the rules of Go, not a programing issue.

[MonteCarlo]

It's also less interesting for another reason, as they only convincingly beat KataGo when it used no search at all.

They had a slight plus when it was allowed to search small numbers of nodes, but past 100 nodes they lost virtually every game, it seems.

It's not so surprising that one can exploit either a pure policy network or so little search that the choice of move is dominated by the policy network.

Cheers!

[Eelco de Groot]

Sory if too off topic to the thread, but a bit excited about the latest KataGo neural network being tested at https://katagotraining.org/

It has only played 156 Rating games, as I write this 172, so the 95% confidence interval is huge, but it started out really well;



Network is kata1-b18c384nbt-s8738519040-d3984211304

After 172 games the rating had dropped again though, to 13607.4 +/- 58.0 but that is still 7 points better than the best Net so far. I can't really judge how it plays as I only started to learn the rules a bit, I can see it is good at creating 'eyes' but that's about it
(I watched this game: https://katagotraining.org/sgfplayer/ra ... s/1172965/ and after 17 moves or so White had already three of those eyes, as far as I could see, Black still trying to surround the territory that was unconquerable at top left of the board, with two eyes in it. But I don't know how winning it was at that point, it just looked very good there.)

[Eelco de Groot]

That network (...211304) in the end did not turn out so well, it seems they always start higher, but come down later because of the way they choose opponents maybe... But at least now, almost a month later, I think there was some small progress, KatGo may cross the 13,620 Elo level!

Okay I stop posting about KataGo progress for now, just this one update:


Network is kata1-b18c384nbt-s9040915712-d4064991045

[Eelco de Groot]

There is a new Network for Kata! With new "dimensions" as some use the term in here (b28c512). Only 35 games played as I write this but I am very excited

[Uri Blass]

It's also less interesting for another reason, as they only convincingly beat KataGo when it used no search at all.

They had a slight plus when it was allowed to search small numbers of nodes, but past 100 nodes they lost virtually every game, it seems.

It's not so surprising that one can exploit either a pure policy network or so little search that the choice of move is dominated by the policy network.

Cheers!

It is interesting if there is a strategy that weak chess players can use to beat lc0 in chess when lc0 is allowed to search one node per move or very small number of nodes and not repeating existing game.

[Eelco de Groot]

The new "2003" net is really making mincemeat of the competition in this 'equal visits' test. Out of the last 100 games I count just five losses and five draws. Rest wins. Haven't checked any games yet. But that is like running equal number of nodes test in chess, with a very slow but big Lc0 like net.

2024-05-02 - First "b28" neural net uploaded! This new larger neural net runs more slowly, similar to the old 60-block nets, but should be much stronger. From tests on discord, under normal usage this series of nets should be stronger than the b18 nets given equal total time, despite the slower speed, and be by far the strongest nets so far on a per-visit basis. We've also upgraded contributing to require KataGo v1.14.0 or later to support this net.

It does not run in my KataGo v1.14.0 GUI but that is maybe because I can't run a GPU on this computer, I have to use the EigenAVX2 version in it that is just 1.13.0 .