I do not have the EGBB files.
I do not know where to find them.
I tried to build the tool that makes the files but it segfaults instantly.
How do I fix this?
Help with Scorpio 3.0.8
Moderators: hgm, Rebel, chrisw
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Help with Scorpio 3.0.8
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Help with Scorpio 3.0.8
https://github.com/dshawul/Scorpio/rele ... 0/egbb.zipDann Corbit wrote: ↑Tue Jun 23, 2020 9:58 am I do not have the EGBB files.
I do not know where to find them.
I tried to build the tool that makes the files but it segfaults instantly.
How do I fix this?
Jörg Oster
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Help with Scorpio 3.0.8
The egbb files (egbbdll64.dll and libnnprobe.so) will be in nnprobe-windows-gpu/. The client.bat calls the installer script if it can not find
the Scorpio/ folder under the nn-dist/ directory. So to reinstall Scorpio, all you have to do is remove the Scorpio folder and call client.bat
again. The three steps I mentioned above will fix this.
For a fresh install, the steps are easy,
client.bat will install Scorpio and connect to the server.
@Joerg Dann do not need egbb files ( the actual endgame tbs ) because we disabled those for selfplay.
the Scorpio/ folder under the nn-dist/ directory. So to reinstall Scorpio, all you have to do is remove the Scorpio folder and call client.bat
again. The three steps I mentioned above will fix this.
For a fresh install, the steps are easy,
Code: Select all
git clone https://github.com/dshawul/nn-dist.git
cd nn-dist
client.bat
@Joerg Dann do not need egbb files ( the actual endgame tbs ) because we disabled those for selfplay.
-
- Posts: 937
- Joined: Fri Mar 10, 2006 4:29 pm
- Location: Germany
Re: Help with Scorpio 3.0.8
I see.Daniel Shawul wrote: ↑Tue Jun 23, 2020 1:44 pm The egbb files (egbbdll64.dll and libnnprobe.so) will be in nnprobe-windows-gpu/. The client.bat calls the installer script if it can not find
the Scorpio/ folder under the nn-dist/ directory. So to reinstall Scorpio, all you have to do is remove the Scorpio folder and call client.bat
again. The three steps I mentioned above will fix this.
For a fresh install, the steps are easy,client.bat will install Scorpio and connect to the server.Code: Select all
git clone https://github.com/dshawul/nn-dist.git cd nn-dist client.bat
@Joerg Dann do not need egbb files ( the actual endgame tbs ) because we disabled those for selfplay.
Then this
was quite misleading.Dann Corbit wrote: ↑Tue Jun 23, 2020 9:58 am I do not have the EGBB files.
I do not know where to find them.
...
Jörg Oster
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Help with Scorpio 3.0.8
No harm
I feel like only the most patient people such as Dann get through the complexity of Scorpio.
One problem is I don't have or can't even rent a machine with GPU and windows so problems pop up there
now and then.
Daniel
I feel like only the most patient people such as Dann get through the complexity of Scorpio.
One problem is I don't have or can't even rent a machine with GPU and windows so problems pop up there
now and then.
Daniel
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Help with Scorpio 3.0.8
Well, the good news is that the new version is using the GPUs very well:
In addition, I am now generating 140 games per minute:
Unfortunately, it is also using all of my CPU so I cannot do anything else on the machine:
Code: Select all
C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>C:\Windows\System32\DriverStore\FileRepository\nv_dispsi.inf_amd64_86e7a5db5f94798e\nvidia-smi
Wed Jun 24 01:27:59 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19 Driver Version: 442.19 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... WDDM | 00000000:01:00.0 On | N/A |
| 90% 87C P2 132W / 250W | 1913MiB / 8192MiB | 71% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... WDDM | 00000000:4E:00.0 Off | N/A |
| 89% 86C P2 136W / 250W | 784MiB / 8192MiB | 69% Default |
+-------------------------------+----------------------+----------------------+
Code: Select all
0: Downloading new network from: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff
0: Finished downloading netowrk!
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[14380] Games 516: + 242 - 252 = 22
[14380] generated 516 games in 3.68 min : Rate 140.41 games/min
0: Finished executing job!
0: Sending 516 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[10948] Games 514: + 256 - 242 = 16
[10948] generated 514 games in 3.65 min : Rate 141.01 games/min
0: Finished executing job!
0: Sending 514 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[12700] Games 515: + 251 - 249 = 15
[12700] generated 515 games in 3.55 min : Rate 144.89 games/min
0: Finished executing job!
0: Sending 515 games to server
0: pong
0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat
[7080] Games 78: + 39 - 39 = 0
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Help with Scorpio 3.0.8
Now, before you go and "fix" a thing like this, I think it should be given some careful thought.
It should be made configurable so that you can get the old behavior back.
The reason that I say that is that you have achieved something incredible that LC0 cannot do:
Utilize the full resources of the machine..
According to my understanding, LC0 goes straight into the crapper if you give it too may CPU threads.
Here, it is using more than 60 threads and it is chugging along like a steamship at full throttle.
It is seven times faster than it was at the slowest point, so you are clearly doing something right.
It should be made configurable so that you can get the old behavior back.
The reason that I say that is that you have achieved something incredible that LC0 cannot do:
Utilize the full resources of the machine..
According to my understanding, LC0 goes straight into the crapper if you give it too may CPU threads.
Here, it is using more than 60 threads and it is chugging along like a steamship at full throttle.
It is seven times faster than it was at the slowest point, so you are clearly doing something right.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Help with Scorpio 3.0.8
Suggestion:
compress the games before sending
Suggestion:
set the priority of scorpio to high.
It will give you 5% more games
compress the games before sending
Suggestion:
set the priority of scorpio to high.
It will give you 5% more games
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Help with Scorpio 3.0.8
Indeed that is fantastic news!Dann Corbit wrote: ↑Wed Jun 24, 2020 10:36 am Well, the good news is that the new version is using the GPUs very well:In addition, I am now generating 140 games per minute:Code: Select all
C:\Users\dannc\chess\nn-dist-master\Scorpio\bin\Windows>C:\Windows\System32\DriverStore\FileRepository\nv_dispsi.inf_amd64_86e7a5db5f94798e\nvidia-smi Wed Jun 24 01:27:59 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 442.19 Driver Version: 442.19 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce RTX 208... WDDM | 00000000:01:00.0 On | N/A | | 90% 87C P2 132W / 250W | 1913MiB / 8192MiB | 71% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce RTX 208... WDDM | 00000000:4E:00.0 Off | N/A | | 89% 86C P2 136W / 250W | 784MiB / 8192MiB | 69% Default | +-------------------------------+----------------------+----------------------+
Unfortunately, it is also using all of my CPU so I cannot do anything else on the machine:Code: Select all
0: Downloading new network from: http://scorpiozero.ddns.net/scorpiozero/nets-11/ID-0-model-3.uff 0: Finished downloading netowrk! 0: pong 0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat [14380] Games 516: + 242 - 252 = 22 [14380] generated 516 games in 3.68 min : Rate 140.41 games/min 0: Finished executing job! 0: Sending 516 games to server 0: pong 0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat [10948] Games 514: + 256 - 242 = 16 [10948] generated 514 games in 3.65 min : Rate 141.01 games/min 0: Finished executing job! 0: Sending 514 games to server 0: pong 0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat [12700] Games 515: + 251 - 249 = 15 [12700] generated 515 games in 3.55 min : Rate 144.89 games/min 0: Finished executing job! 0: Sending 515 games to server 0: pong 0: Executing job : C:\Users\dannc\chess\nn-dist-master\scripts\job-client.bat [7080] Games 78: + 39 - 39 = 0
That is the kind of game generation rate I expected from your machine with a powerful CPU and GPU.
There are two solutions for reducing CPU usage
a) set delay=1. If this gives acceptable behavior for you, it is the best solution as it will affect the game generation rate the minimum.
b) The other is to specify the cores that the client process use directly.
On Linux I do this, to restrict it to the first 12 cores
Code: Select all
taskset -c 0-11 ./client.sh
Code: Select all
START /A 8F client.bat
Btw at the rate you are generating games, you should probably set the number of games to generate before sending to a higher value like 2048.
I am actually happy that the "too many threads approach" worked at all. The main reason I did it was that it makes parallelizing any search algorithmAccording to my understanding, LC0 goes straight into the crapper if you give it too may CPU threads.
Here, it is using more than 60 threads and it is chugging along like a steamship at full throttle.
It is seven times faster than it was at the slowest point, so you are clearly doing something right.
including alpha-beta in standard or rollouts from, mcts etc for use of NNs without rewriting the algorithm. Eg. Lc0 code is messy for the sake of enabling "batching". Using linbnnprobe.so, I am able to make it use NN with ABDADA, LazySMP and even YBW splitting just fine.
Aloril from TCEC tested Scorpio with a record 1280 threads on 8 voltas yesterday! It worked and scaled better than using 640 threads on it.
I have put this in the todo list.Suggestion:
compress the games before sending
I am not sure I can do this programatically just like setting processor affinity, but I will put it atleast in the readmeSuggestion:
set the priority of scorpio to high.
It will give you 5% more games
that setting priority higher gives a boost in performance.
Thanks a lot
Daniel
-
- Posts: 4185
- Joined: Tue Mar 14, 2006 11:34 am
- Location: Ethiopia
Re: Help with Scorpio 3.0.8
Another good news is that we may have reached 3000 ccrl elo! I was actually surpized to find this out yesterday
because I was thinking maybe it is 2500 elo. Selfplay elo on the webiste is as we all know inflated.
The net is starting to beat 2-thread Scorpio alpha-beta.
If that result persists we have met our goal in the first learning-rate phase.
It has some problems with openings. Up until a few nets ago its best opening move is b4!
That move is now finally out of the top 4 but I was about to accept the result as is, and preparing excuses that it is a good opening
https://en.wikipedia.org/wiki/Sokolsky_Opening
We will let the training go with this LR until it reshuffles the opening enough to make a decent opening even if there will not be a significant
ELO increase. Lowering LR prematurely I learned the hard way how bad it is.
because I was thinking maybe it is 2500 elo. Selfplay elo on the webiste is as we all know inflated.
The net is starting to beat 2-thread Scorpio alpha-beta.
If that result persists we have met our goal in the first learning-rate phase.
It has some problems with openings. Up until a few nets ago its best opening move is b4!
That move is now finally out of the top 4 but I was about to accept the result as is, and preparing excuses that it is a good opening
https://en.wikipedia.org/wiki/Sokolsky_Opening
We will let the training go with this LR until it reshuffles the opening enough to make a decent opening even if there will not be a significant
ELO increase. Lowering LR prematurely I learned the hard way how bad it is.