Help with Scorpio 3.0.8

[Dann Corbit]

I do not have the EGBB files.
I do not know where to find them.
I tried to build the tool that makes the files but it segfaults instantly.
How do I fix this?

[Joerg Oster]

I do not have the EGBB files.
I do not know where to find them.
I tried to build the tool that makes the files but it segfaults instantly.
How do I fix this?

https://github.com/dshawul/Scorpio/rele ... 0/egbb.zip

[Daniel Shawul]

The egbb files (egbbdll64.dll and libnnprobe.so) will be in nnprobe-windows-gpu/. The client.bat calls the installer script if it can not find
the Scorpio/ folder under the nn-dist/ directory. So to reinstall Scorpio, all you have to do is remove the Scorpio folder and call client.bat
again. The three steps I mentioned above will fix this.

For a fresh install, the steps are easy,

Code: Select all

git clone https://github.com/dshawul/nn-dist.git
cd nn-dist
client.bat

client.bat will install Scorpio and connect to the server.

@Joerg Dann do not need egbb files ( the actual endgame tbs ) because we disabled those for selfplay.

[Joerg Oster]

The egbb files (egbbdll64.dll and libnnprobe.so) will be in nnprobe-windows-gpu/. The client.bat calls the installer script if it can not find
the Scorpio/ folder under the nn-dist/ directory. So to reinstall Scorpio, all you have to do is remove the Scorpio folder and call client.bat
again. The three steps I mentioned above will fix this.

For a fresh install, the steps are easy,

Code: Select all

git clone https://github.com/dshawul/nn-dist.git
cd nn-dist
client.bat

client.bat will install Scorpio and connect to the server.

@Joerg Dann do not need egbb files ( the actual endgame tbs ) because we disabled those for selfplay.

I see.
Then this

I do not have the EGBB files.
I do not know where to find them.
...

was quite misleading.

[Daniel Shawul]

No harm

I feel like only the most patient people such as Dann get through the complexity of Scorpio.

One problem is I don't have or can't even rent a machine with GPU and windows so problems pop up there
now and then.

Daniel

[Dann Corbit]

Well, the good news is that the new version is using the GPUs very well:

Code: Select all

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>C:\Windows\System32\DriverStore\FileRepository\nv_dispsi.inf_amd64_86e7a5db5f94798e\nvidia-smi
Wed Jun 24 01:27:59 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:01:00.0  On |                  N/A |
| 90%   87C    P2   132W / 250W |   1913MiB /  8192MiB |     71%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208... WDDM  | 00000000:4E:00.0 Off |                  N/A |
| 89%   86C    P2   136W / 250W |    784MiB /  8192MiB |     69%      Default |
+-------------------------------+----------------------+----------------------+

In addition, I am now generating 140 games per minute:

Code: Select all

0: Downloading new network from: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: Finished downloading netowrk!
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[14380] Games 516: + 242 - 252 = 22
[14380] generated 516 games in 3.68 min : Rate 140.41 games/min
0: Finished executing job!
0: Sending 516 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[10948] Games 514: + 256 - 242 = 16
[10948] generated 514 games in 3.65 min : Rate 141.01 games/min
0: Finished executing job!
0: Sending 514 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[12700] Games 515: + 251 - 249 = 15
[12700] generated 515 games in 3.55 min : Rate 144.89 games/min
0: Finished executing job!
0: Sending 515 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[7080] Games 78: + 39 - 39 = 0

Unfortunately, it is also using all of my CPU so I cannot do anything else on the machine:

[Dann Corbit]

Now, before you go and "fix" a thing like this, I think it should be given some careful thought.
It should be made configurable so that you can get the old behavior back.
The reason that I say that is that you have achieved something incredible that LC0 cannot do:
Utilize the full resources of the machine..

According to my understanding, LC0 goes straight into the crapper if you give it too may CPU threads.
Here, it is using more than 60 threads and it is chugging along like a steamship at full throttle.
It is seven times faster than it was at the slowest point, so you are clearly doing something right.

[Dann Corbit]

Suggestion:
compress the games before sending

Suggestion:
set the priority of scorpio to high.
It will give you 5% more games

[Daniel Shawul]

Well, the good news is that the new version is using the GPUs very well:

Code: Select all

C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>C:\Windows\System32\DriverStore\FileRepository\nv_dispsi.inf_amd64_86e7a5db5f94798e\nvidia-smi
Wed Jun 24 01:27:59 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208... WDDM  | 00000000:01:00.0  On |                  N/A |
| 90%   87C    P2   132W / 250W |   1913MiB /  8192MiB |     71%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208... WDDM  | 00000000:4E:00.0 Off |                  N/A |
| 89%   86C    P2   136W / 250W |    784MiB /  8192MiB |     69%      Default |
+-------------------------------+----------------------+----------------------+

In addition, I am now generating 140 games per minute:

Code: Select all

0: Downloading new network from: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: Finished downloading netowrk!
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[14380] Games 516: + 242 - 252 = 22
[14380] generated 516 games in 3.68 min : Rate 140.41 games/min
0: Finished executing job!
0: Sending 516 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[10948] Games 514: + 256 - 242 = 16
[10948] generated 514 games in 3.65 min : Rate 141.01 games/min
0: Finished executing job!
0: Sending 514 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[12700] Games 515: + 251 - 249 = 15
[12700] generated 515 games in 3.55 min : Rate 144.89 games/min
0: Finished executing job!
0: Sending 515 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[7080] Games 78: + 39 - 39 = 0

Unfortunately, it is also using all of my CPU so I cannot do anything else on the machine:

Indeed that is fantastic news!
That is the kind of game generation rate I expected from your machine with a powerful CPU and GPU.

There are two solutions for reducing CPU usage
a) set delay=1. If this gives acceptable behavior for you, it is the best solution as it will affect the game generation rate the minimum.
b) The other is to specify the cores that the client process use directly.
On Linux I do this, to restrict it to the first 12 cores

Code: Select all

taskset -c 0-11 ./client.sh

On windows

Code: Select all

START /A 8F client.bat

8F is hexadecimal representation of the first 12 cores. I believe you can also set this via the task manager.

Btw at the rate you are generating games, you should probably set the number of games to generate before sending to a higher value like 2048.

According to my understanding, LC0 goes straight into the crapper if you give it too may CPU threads.
Here, it is using more than 60 threads and it is chugging along like a steamship at full throttle.
It is seven times faster than it was at the slowest point, so you are clearly doing something right.

I am actually happy that the "too many threads approach" worked at all. The main reason I did it was that it makes parallelizing any search algorithm
including alpha-beta in standard or rollouts from, mcts etc for use of NNs without rewriting the algorithm. Eg. Lc0 code is messy for the sake of enabling "batching". Using linbnnprobe.so, I am able to make it use NN with ABDADA, LazySMP and even YBW splitting just fine.

Aloril from TCEC tested Scorpio with a record 1280 threads on 8 voltas yesterday! It worked and scaled better than using 640 threads on it.

Suggestion:
compress the games before sending

I have put this in the todo list.

Suggestion:
set the priority of scorpio to high.
It will give you 5% more games

I am not sure I can do this programatically just like setting processor affinity, but I will put it atleast in the readme
that setting priority higher gives a boost in performance.

Thanks a lot
Daniel

[Daniel Shawul]

Another good news is that we may have reached 3000 ccrl elo! I was actually surpized to find this out yesterday
because I was thinking maybe it is 2500 elo. Selfplay elo on the webiste is as we all know inflated.
The net is starting to beat 2-thread Scorpio alpha-beta.
If that result persists we have met our goal in the first learning-rate phase.

It has some problems with openings. Up until a few nets ago its best opening move is b4!
That move is now finally out of the top 4 but I was about to accept the result as is, and preparing excuses that it is a good opening
https://en.wikipedia.org/wiki/Sokolsky_Opening

We will let the training go with this LR until it reshuffles the opening enough to make a decent opening even if there will not be a significant
ELO increase. Lowering LR prematurely I learned the hard way how bad it is.