LC0 on the WAC test suite, bad results

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Gian-Carlo Pascutto
Posts: 1238
Joined: Sat Dec 13, 2008 6:00 pm
Contact:

Re: LC0 on the WAC test suite, bad results

Post by Gian-Carlo Pascutto » Fri Apr 27, 2018 7:27 pm

Daniel Shawul wrote: About the 3x3 filters, why does LCzero have code for inference anyway ? My first take with NN chess engine would probably use the c++ tensorflow backend for inference, which would use cuDNN as its backend which probably has better optimized algorithms than hand-written ones. Is this done for to avoid dependency on tensorflow or am I missing something ?
Many reasons.

Leela (Zero) was written before TensorFlow had a C++ API that didn't suck. I originally used Caffe even, because TensorFlow itself also didn't exist outside Google. Meanwhile, someone already did a re-implementation using TF as the backend, it's in the same github, so you're not "missing something" in that sense.

Using OpenCL avoided a dependency on Caffe (or TensorFlow), cuDNN (don't see the point of arguing with Mr. Stanisavljevic here, as anyone can just go out and find the actual license, and AFAIK Linscott/Huizinga are negotiating with NVIDIA about it so it may not even matter) and consequently NVIDIA (I did not anticipate AMD basically self destructing). This made distributing the original Leela (a windows GUI program with a simple installer) manageable even with GPU support. Losing those dependencies makes the program usable by a lot more people. (Even the commercial Go engines don't bother to include GPU support so far.)

Re-implementing the filters manually was very interesting and fun. I like hardware, didn't do GPU programming before, it was a good learning experience. I got to learn OpenCL and CUDA (quiz: how do you optimize OpenCL code for NVIDIA? answer: you port it to CUDA, optimize, and port back the changes, because NVIDIA got rid of their OpenCL profilers) and more about modern GPU hardware than I wanted to know.

There is a port of Leela Zero to cuDNN, and it's slower than the OpenCL code. This seems to be because cuDNN is not very well optimized for small batch sizes (though TensorRT is!) which Leela uses by default (there have been some forks to address this). I was skeptical of large batches because of how they make the tree search less efficient, but current evidence seems to indicate the gain on the DCNN side overwhelms the search loss efficiency. There may be other things at play, IIRC Huizinga compared cuDNN with TensorRT and got basically batshit insane performance differences.

Also, I judged that just a simple exe that anyone can click would bring more users than requiring a Python/TensorFlow/cuDNN install. Sure, you can make an installer that deals with all of those, in theory. You go ahead and do that, because I won't.

As for CPU-only operation, you can just link with MKL, it's fully supported. The closed source version ships with it. I think nowadays you'd probably use MKL-DNN though, but no-one has bothered porting the backend to it. I even have a version that doesn't need a BLAS lib at all, and uses my own code and autovectorization. But it's very easy to suddenly lose a factor of 2 somewhere if the cache layout of the target system doesn't exactly match what you were expecting, so this is one of the few parts where I gave up trying to do things manually.

tl;dr Yes you can replace the whole thing by 2000 lines of python (https://github.com/tensorflow/minigo) and 500 dependencies but I actually like programming.

Milos
Posts: 4064
Joined: Wed Nov 25, 2009 12:47 am

Re: LC0 on the WAC test suite, bad results

Post by Milos » Fri Apr 27, 2018 9:55 pm

Gian-Carlo Pascutto wrote:Using OpenCL avoided a dependency on Caffe (or TensorFlow), cuDNN (don't see the point of arguing with Mr. Stanisavljevic here, as anyone can just go out and find the actual license, and AFAIK Linscott/Huizinga are negotiating with NVIDIA about it so it may not even matter) and consequently NVIDIA (I did not anticipate AMD basically self destructing).
Well, are you a licensing lawyer? AFAIK you are not.
I read the whole thread where you discussed the licensing and frankly speaking it's a load of bollocks.
You simply misunderstood the license, totally.
In the license Enterprise is defined as "you or any company or legal entity for which you accepted the terms of this SLA, and their subsidiaries of which your company or legal entity owns more than fifty percent (50%) of the issued and outstanding equity". However, part related to redistribution in the supplement is binding only to legal persons such as private (i.e., business entity or non-governmental organization) or public (i.e., government) organizations, but not to natural persons.
AFAIK, there are no legal persons included in LC0 project so that part of the licence you are so much afraid of is simply non-binding and by redistributing cuDNN libraries within LC0 you would have no obligations related to the Audit clause of the Supplement of SLA.
And the part where you interpret basically putting a licence clause in LC0 sources as snitching is just hilarious.
Please next time go consult a licensing lawyer, because that thread was not only paranoid but outright embarrassing.
Last edited by Milos on Fri Apr 27, 2018 9:56 pm, edited 1 time in total.

carldaman
Posts: 2094
Joined: Sat Jun 02, 2012 12:13 am

Re: LC0 on the WAC test suite, bad results

Post by carldaman » Fri Apr 27, 2018 9:56 pm

Albert Silver wrote:
Damir wrote:Leela's King Safety and Tactical ability are very weak. :( :( . This must be improved. :o :o
I find Leela's understanding of king safety very advanced actually. This is seen not only in her impressive attacking ability, but the evaluations she displays when one king is exposed or stuck in the center.
+1

And the tactics are "weak" only relative to other strong engines. Human masters have a tough time outplaying Leela tactically. This means its tactics are really not that weak.

Milos
Posts: 4064
Joined: Wed Nov 25, 2009 12:47 am

Re: LC0 on the WAC test suite, bad results

Post by Milos » Fri Apr 27, 2018 10:28 pm

Milos wrote:
Gian-Carlo Pascutto wrote:Using OpenCL avoided a dependency on Caffe (or TensorFlow), cuDNN (don't see the point of arguing with Mr. Stanisavljevic here, as anyone can just go out and find the actual license, and AFAIK Linscott/Huizinga are negotiating with NVIDIA about it so it may not even matter) and consequently NVIDIA (I did not anticipate AMD basically self destructing).
Well, are you a licensing lawyer? AFAIK you are not.
I read the whole thread where you discussed the licensing and frankly speaking it's a load of bollocks.
You simply misunderstood the license, totally.
Please next time go consult a licensing lawyer, because that thread was not only paranoid but outright embarrassing.
And the part where you claim that you can't include cuDNN headers in LC0 code because you'd breach GPL, really, wtf???? It's like saying you can't include MKL header because you break GPL.
cuDNN library is clearly under System library exception of GPL v3 and is totally fine to be included in any GPLed project.
You really seems to have a general problem with license understanding.

IQ
Posts: 162
Joined: Thu Dec 17, 2009 9:46 am

Re: LC0 on the WAC test suite, bad results

Post by IQ » Sat Apr 28, 2018 1:41 pm

jkiliani wrote:
JohnS wrote:
jkiliani wrote:OK, and what's your conclusion from all that?

It's no secret that Leela's weakest point is tactics, especially if you just give her FENs instead of positions with move history. If you want a tactics solver, you're simply using the wrong engine here..
Would the performance improve if you make some random moves starting from the test position to fill in the planes, say king shuffles that eventually lead back to the test position?
Rather unlikely I think, the neural net expects history planes that actually make sense (i.e. the position could potentially happen this way in a real game). You could try it anyway, maybe it helps...
Many of these positions come from real games. Shouldn't be to hard finding those in a database and then just quickly play through that game to get to the key move. Would be interesting to see if leela really improves tactically if the history plane input is present......

Uri Blass
Posts: 8972
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: LC0 on the WAC test suite, bad results

Post by Uri Blass » Sat Apr 28, 2018 5:33 pm

carldaman wrote:
Albert Silver wrote:
Damir wrote:Leela's King Safety and Tactical ability are very weak. :( :( . This must be improved. :o :o
I find Leela's understanding of king safety very advanced actually. This is seen not only in her impressive attacking ability, but the evaluations she displays when one king is exposed or stuck in the center.
+1

And the tactics are "weak" only relative to other strong engines. Human masters have a tough time outplaying Leela tactically. This means its tactics are really not that weak.
No
It does not mean it.

The way to test if it weak tactically relative to humans is not by games against humans.

The way to test it is by competition with humans in solving chess tactical problems.

Of course you need to define what is tactics and you can say that WAC is not the right test to test tactics because tactics is not only sacrifices.

Uri

syzygy
Posts: 5000
Joined: Tue Feb 28, 2012 10:56 pm

Re: LC0 on the WAC test suite, bad results

Post by syzygy » Sat Apr 28, 2018 6:27 pm

Uri Blass wrote:
carldaman wrote:
Albert Silver wrote:
Damir wrote:Leela's King Safety and Tactical ability are very weak. :( :( . This must be improved. :o :o
I find Leela's understanding of king safety very advanced actually. This is seen not only in her impressive attacking ability, but the evaluations she displays when one king is exposed or stuck in the center.
+1

And the tactics are "weak" only relative to other strong engines. Human masters have a tough time outplaying Leela tactically. This means its tactics are really not that weak.
No
It does not mean it.

The way to test if it weak tactically relative to humans is not by games against humans.

The way to test it is by competition with humans in solving chess tactical problems.

Of course you need to define what is tactics and you can say that WAC is not the right test to test tactics because tactics is not only sacrifices.
If you define "tactically weak relative to humans" as "human masters can easily outplay it tactically", then Carl's observation that "human masters have a tough time outplaying Leela tactically" shows that it is not tactically weak relative to humans.

And I think that definition makes a lot of sense.

As to tactical problems, if you give me a position and tell me that there is some tactic to be found, I am more likely to find it than when I would encounter the same position in a game.

Werewolf
Posts: 1351
Joined: Thu Sep 18, 2008 8:24 pm

Re: LC0 on the WAC test suite, bad results

Post by Werewolf » Sat Apr 28, 2018 6:36 pm

Also there are signs LCZero is slowly getting a bit better at tactics, just a little

Robert Flesher
Posts: 1246
Joined: Tue Aug 18, 2009 1:06 am

Re: LC0 on the WAC test suite, bad results

Post by Robert Flesher » Sat Apr 28, 2018 11:15 pm

nabildanial wrote:
Guenther wrote:
nabildanial wrote:This test should prove that the policy head does improve tactically and you could say that Leela is learning tactics via pattern recognition.

https://docs.google.com/spreadsheets/d/ ... yHLVnffLag
This sheet contains only 1 position.
It needs at least a few hundred to have some significance.
It is a pretty big result considering that older nets than id199 couldn't find the correct move by using only the policy head. Most of us over Discord didn't even believe it would ever happen. I mean the move is absurdly hard to find without search, even for humans. I guess now people can now look for more similar positions for these kind of tests to be more statistically significant.

This looks promising. I can confirm that LCZERO cannot win KQ vs KR, so maybe this could be a position to check as the network progresses.

Robert Flesher
Posts: 1246
Joined: Tue Aug 18, 2009 1:06 am

Re: LC0 on the WAC test suite, bad results

Post by Robert Flesher » Sat Apr 28, 2018 11:28 pm

carldaman wrote:
Albert Silver wrote:
Damir wrote:Leela's King Safety and Tactical ability are very weak. :( :( . This must be improved. :o :o
I find Leela's understanding of king safety very advanced actually. This is seen not only in her impressive attacking ability, but the evaluations she displays when one king is exposed or stuck in the center.
+1

And the tactics are "weak" only relative to other strong engines. Human masters have a tough time outplaying Leela tactically. This means its tactics are really not that weak.
+2

Post Reply