Why are super top professional Go players so stupid?

[Laskos]

Maybe I will now analyze the Meijin Retirement Game and the games of Go Seigen, to see how stupid these legends were.

Not so fast.

Now I see that Leela Go Zero is stupid too. Top Go pros are stupid, Leela Go is stupid, only I am smart.
I thought that setting komi (KM) in the SGF file is enough to make Leela play with desired komi. Wrong! Leela engine doesn't support komi setting, it is designed with fixed komi 7.5 and no handicap. Not only that, all the networks are trained to play well only with this komi of 7.5 and no handicap. Again, like in Leela Chess, the engine and the weights are very specific in what they do well. Traditional MCTS Go engines had no problems dealing with various komi values and various handicaps. This specialization of NN-approach (DCNN) is really annoying, as it surely won't bring us closer to human-like abstraction and adaptation.

I took the pre komi era "The Game of the Century" between Honinbo Shusai (white) and Go Seigen (black), which lasted for about 100 days (!!!) in the years 1933-34. Shusai was the master of the Go world, as he was the head of Honinbo Go school, the most prestigious of the schools founded at the behest of Shōgun Tokugawa Ieyasu at the start of the 17th century. Go Seigen was famed as a prodigy, first among a generation of young new brilliant players, and would go on to become the most celebrated player of the 20th century. So, no komi, and the favorite (master) Shushai takes white as a favor to the candidate Seigen (black).

The SGF of the game is here:

Code: Select all

(;
PB[Go Seigen]
BR[5d]
PW[Honinbo Shusai]
WR[Meijin]
RE[W+2]
DT[1933-10-16..1934-01-29]KM[0]
SZ[19];
B[qc];W[cd];B[dp];W[pq];B[jj];W[pd];B[qd];W[pe];B[ob];W[qn];B[jp];W[lq];B[dj];W[ed];B[lo];W[nq];B[kq];W[lp];B[ko];W[ch];B[gj];W[fq];B[fo];W[iq];B[gp];W[cq];B[cp];W[dq];B[nd];W[ph];B[pj];W[ne];B[me];W[mf];B[nf];W[ng];B[oe];W[of];B[ne];W[nj];B[om];W[mg];B[ke];W[hc];B[qm];W[rm];B[rn];W[rl];B[ro];W[pm];B[ql];W[qk];B[pl];W[pn];B[pk];W[qj];B[nl];W[pi];B[qp];W[on];B[nn];W[no];B[mo];W[op];B[qr];W[qq];B[rq];W[pr];B[rr];W[mn];B[nm];W[lm];B[ll];W[kp];B[mp];W[mq];B[ip];W[jq];B[kk];W[cn];B[ep];W[cj];B[ck];W[dk];B[cl];W[dl];B[ci];W[bj];B[bi];W[di];B[ej];W[bh];B[dm];W[ei];B[fj];W[cm];B[ak];W[ai];B[el];W[bp];B[rf];W[co];B[qg];W[rh];B[jc];W[gq];B[eq];W[en];B[fn];W[er];B[fr];W[dr];B[hq];W[hr];B[hp];W[gr];B[fc];W[ic];B[dc];W[cc];B[gd];W[he];B[ge];W[je];B[hd];W[id];B[hf];W[kd];B[le];W[kc];B[og];W[oh];B[pg];W[em];B[ek];W[fm];B[gm];W[mb];B[cb];W[ec];B[eb];W[dd];B[db];W[gb];B[bc];W[bd];B[bb];W[nb];B[oa];W[fb];B[ea];W[gh];B[jf];W[ie];B[ff];W[hi];B[hg];W[eo];B[fp];W[hl];B[il];W[jh];B[fh];W[fi];B[hj];W[if];B[ih];W[ii];B[ig];W[jg];B[ji];W[fg];B[kf];W[ij];B[ik];W[hh];B[gf];W[mk];B[df];W[ml];B[sm];W[sk];B[mm];W[dg];B[li];W[ol];B[ln];W[nk];B[km];W[fl];B[gl];W[oj];B[qh];W[ri];B[cf];W[ef];B[gi];W[eh];B[ee];W[eg];B[de];W[bf];B[bq];W[br];B[ps];W[os];B[qs];W[oq];B[bo];W[aq];B[an];W[bn];B[am];W[ao];B[rg];W[mi];B[qi];W[lh];B[kh];W[lj];B[rj];W[rk];B[sh];W[sj];B[bg];W[ae];B[cg];W[ag];B[fe];W[gc];B[fd];W[ce];B[kg];W[mc];B[ki];W[fk];B[ld];W[lc];B[gk];W[bk];B[bl];W[na];B[nc];W[gg];B[fa];W[ga];B[ac];W[ad];B[lg];W[mh];B[qo];W[lk])

Trying to analyze the game, it immediately became apparent that Leela doesn't play no komi games, not reasonably (it plays legally, but is making stupid moves). I went along trying to find out about this issue, and after a search, found a version of Leela (10 months old) which is modified to play with desired komi (set the option "--target-komi 0" for komi 0). But there is still a similarly serious issue about the weights files. The training is strict using komi 7.5, and even if the engine is modified, there are no weights for no komi games. I was not sure how severe is the issue, but it seems it is severe enough to miss most of the decisive moves of the game even at 1 million playouts per move. I have an analysis graph of the game, but it is useless. None of the pre komi era legendary games can be seriously analyzed by this modified Leela engine using the usual weights.

Do they plan in some future to train on komi 0 self-games?
Do they plan in some future to train using not the winning probability as the goal but the number of stones at the end of the game? This will automatically make the nets universal to various komi (but not to various handicaps). Yes, the resulting nets would be a bit weaker as winning probability goes, but for the sake of some generality, this effort seems worth taking.

I will post two crucial moves of the game the modified Leela with the latest weights misses in >1 million playouts, but after I make the moves, the eval slowly, steadily soars.

[Laskos]

The legendary W160. It was rumored that it was not Shusai but one of his students, Maeda Nobuaki, who was the author of this legendary move. Leela after >1 million playouts wants to play M10 which gives an equal play (blue mark). The correct H8 (hollow white circle) is winning, and after actually playing it, Leela slowly sees that this move is winning.

Legendary_05.jpg

[Laskos]

The tough W184 D13 completely winning move is missed by Leela after >1 million playouts (it sticks to the blue M9, missing the win). After the correct D13 is played, Leela slowly realizes it is completely winning.

So, pre komi era legendary games have to wait to be analyzed by Leela, as it (and its weights) is not designed to handle them.

Legendary_03.jpg

[Hai]

Good to see that Leela changed from 20x256 to 40x256 net, because at the end bigger is better .

Depends on hardware balance. They don't seem to use CUDA (cuDNN) anywhere, only OpenCL, my speeds are not impressive on my RTX 2070. Might be due to batch sizes used not helping much with cuDNN, I don't know. But Leela's 40x256 latest net beats conclusively FaceBook latest 20x256 net at fixed nodes. Not sure at fixed time, especially LTC (I expect 40b to scale better than 20b).

If they use only OpenCL and not CUDA (cuDNN) that's really... unclever.

[S.Taylor]

I learned in one-two hours of analysis more than they have learnt in decades of study and practice. I guess they are natively not that gifted. Lack of vitamins.

I would honestly hope that chess players are the most intelligent humans (apart of course from top talmudic scholars).

Ok, to put it another way, i thought that chess was the king of mind games. But it doesn't really matter if it is or isn't and that go is the ultimate. So I'm open to that opinion if anyone wishes to express it.

[Gian-Carlo Pascutto]

If they use only OpenCL and not CUDA (cuDNN) that's really... unclever.

I believe Ttl/Henrik Forsten has a CUDA+cuDNN version in one of his github branches. AFAIK it is slower or at least the performance difference was completely uninteresting. On RTX cards the OpenCL code emits the new Tensor Core instructions directly.

Go boards are bigger so there's less incentive to use larger batches, and the policy is more reliable so searching very wide with larger batches becomes less attractive. And without large batches cuDNN is nothing special. Comparing to chess or lc0 will be rather misleading, although I remember seeing lc0 patches to stop using cuDNN when the batches are small, so they must have made the same observation.

Nice comment BTW, reminds me why I read this forum at most once every 2 months.

[Gian-Carlo Pascutto]

I am pretty amazed having this thing on my PC as free open source software, that's why I posted here. These human players I am writing here about are Gods, and my stupid irreverencies please me .

The most amazing thing IMHO is that in chess, we had ~15 years or so to get used to computers gradually increasing strength from strong amateur to post-world-champion. Chess players and top players gradually adapted and used more and more computer analysis and learned how to interpret it.

In Go, computers made the same evolution in about 1.5 years. I am sure there was a mad scramble to understand how to make use of the computers at the top level.

[Gian-Carlo Pascutto]

I thought that setting komi (KM) in the SGF file is enough to make Leela play with desired komi. Wrong! Leela engine doesn't support komi setting, it is designed with fixed komi 7.5 and no handicap. Not only that, all the networks are trained to play well only with this komi of 7.5 and no handicap. Again, like in Leela Chess, the engine and the weights are very specific in what they do well. Traditional MCTS Go engines had no problems dealing with various komi values and various handicaps. This specialization of NN-approach (DCNN) is really annoying, as it surely won't bring us closer to human-like abstraction and adaptation.

https://arxiv.org/abs/1809.03928

This approach has been scaled up successfully so far. I hope the SAI team joins in organizing the "Leela Zero 2.0" effort.

[Laskos]

I thought that setting komi (KM) in the SGF file is enough to make Leela play with desired komi. Wrong! Leela engine doesn't support komi setting, it is designed with fixed komi 7.5 and no handicap. Not only that, all the networks are trained to play well only with this komi of 7.5 and no handicap. Again, like in Leela Chess, the engine and the weights are very specific in what they do well. Traditional MCTS Go engines had no problems dealing with various komi values and various handicaps. This specialization of NN-approach (DCNN) is really annoying, as it surely won't bring us closer to human-like abstraction and adaptation.

https://arxiv.org/abs/1809.03928

This approach has been scaled up successfully so far. I hope the SAI team joins in organizing the "Leela Zero 2.0" effort.

Thanks, this two-parameter sigmoid which is built from two predicted parameters is a nice way to extend the one predicted parameter of present day Leela. How much it loses in strength with the same amount of training for the same komi (say current 7.5 for 19x19) compared to current fixed-komi approach? With the new approach, training is performed having the same goal, maximizing the win rate, and not the score, right? Then you build also "agents" training to maximize the score? Then again, this will diminish somewhat the strength, but since AlphaGo matches and the preprint, I proposed a side "agent" maximizing the score, as some games were annoying towards the end. And these "agents" somehow come as a natural by-product of varied komi two-parameter approach.

On another note, I played 10 games in Sabaki (first time I am trying it), pretty long ones (about 90 minutes per game) using my RTX 2070 GPU, between latest Leela 40b net and latest Facebook 20b net. Komi was 7.5. The score was equal 5:5, but the odd thing was that there were no black wins, only 10 white wins. Does a komi of 7.5 favor so much white? Is "no variety" of the starting position an issue? The games looked quite different from the beginning, but I am not sure they are unrelated. Maybe one would better have the first stones played using some non-zero temperature? The eval of the staring position is about 56% for white with both nets, not that decisive to give so skewed white/black an outcome. Also, 4/5 of 40b net wins were right in the opening or just after it. Only one of the 20b net wins was right after the opening. I can attach the SGF files of you need them. The nets give quite different evals for a lot of positions, sometimes one favors white and the other black (usually the "optimist" for yourself way).