Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Discussion of computer chess matches and engine tournaments.

Moderators: hgm, Rebel, chrisw

tpoppins
Posts: 919
Joined: Tue Nov 24, 2015 9:11 pm
Location: upstate

Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by tpoppins »

Image

Games: PGN
Test setup: details

Code: Select all

CCRL 40/40 Rating List - Custom engine selection
1004662 games played by 2384 programs, run by 22 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz), 
about 15 minutes on a modern Intel CPU.
Computed on March 16, 2019 with Bayeselo based on 1'004'662 games
Tested by CCRL team, 2005-2019, http://computerchess.org.uk/ccrl/4040/

        Engine                        Elo   +    -   Score  AvOp  Games
  Scorpio 2.8 64-bit                 2885  +15  -15  49.7%   +0.8  1530
  Scorpio 2.8.5 MCTS 64-bit          2704  +17  -17  46.7%  +23.2  1166
  Scorpio 2.9 MCTS+NN 64-bit         2621  +21  -21  49.6%   +0.2   808
Tirsa Poppins
CCRL
Daniel Shawul
Posts: 4185
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by Daniel Shawul »

Hi tpoppins,

Thank you very much for the test. Did you figure out what the issue was with cpu scorpio version ?
It maybe that i continually update the versions and there was a mixup in your download.

I have added an install.sh script that will download and configure scorpio and test it for you.
https://github.com/dshawul/Scorpio/rele ... install.sh
It works as is for linux but for windows you need something to run bash scripts with, cygwin or alike.
In the script first lines you can choose OS and whether to install CPU/GPU versions.

regards,
Daniel
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by Dann Corbit »

Daniel Shawul wrote: Thu Mar 21, 2019 4:57 pm Hi tpoppins,

Thank you very much for the test. Did you figure out what the issue was with cpu scorpio version ?
It maybe that i continually update the versions and there was a mixup in your download.

I have added an install.sh script that will download and configure scorpio and test it for you.
https://github.com/dshawul/Scorpio/rele ... install.sh
It works as is for linux but for windows you need something to run bash scripts with, cygwin or alike.
In the script first lines you can choose OS and whether to install CPU/GPU versions.

regards,
Daniel
For running bash on Windows, I prefer msys2:
https://www.msys2.org/

The one caveat with msys2 is that there is no true fork(). On the other hand, the fork for cygwin is not like the linux fork, but it does work.
Msys2 is the neatest thing since sliced bread.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Chessqueen
Posts: 5580
Joined: Wed Sep 05, 2018 2:16 am
Location: Moving
Full name: Jorge Picado

Re: Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by Chessqueen »

tpoppins wrote: Thu Mar 21, 2019 7:41 am Image

Games: PGN
Test setup: details

Code: Select all

CCRL 40/40 Rating List - Custom engine selection
1004662 games played by 2384 programs, run by 22 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 40 minutes on Athlon 64 X2 4600+ (2.4 GHz), 
about 15 minutes on a modern Intel CPU.
Computed on March 16, 2019 with Bayeselo based on 1'004'662 games
Tested by CCRL team, 2005-2019, http://computerchess.org.uk/ccrl/4040/

        Engine                        Elo   +    -   Score  AvOp  Games
  Scorpio 2.8 64-bit                 2885  +15  -15  49.7%   +0.8  1530
  Scorpio 2.8.5 MCTS 64-bit          2704  +17  -17  46.7%  +23.2  1166
  Scorpio 2.9 MCTS+NN 64-bit         2621  +21  -21  49.6%   +0.2   808
I do Not believe that MTCS + NN perform better than AB + NN. Allie+Stein MTCS+NN performed worse than AB+NN.
Do NOT worry and be happy, we all live a short life :roll:
tpoppins
Posts: 919
Joined: Tue Nov 24, 2015 9:11 pm
Location: upstate

Re: Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by tpoppins »

Daniel, I did figure out the immediate cause of Scorpio failing to load NNs and crashing under Cute Chess GUI and brought it to your attention in the issues I opened on your Github site. The datestamps of the exec, NN and DLL are also provided there, along with the CC debug logs, PGNs and WER reports.

Your questions here and the suggestion of a mixup shows that you didn't bother to look at any of it, and your continual ignoring of my posts in those issues creates the impression that you are not interested in receiving bug reports and working together on solving the problems. Things were quite different a year ago when I was helping you test your first MCTS version.
Tirsa Poppins
CCRL
Daniel Shawul
Posts: 4185
Joined: Tue Mar 14, 2006 11:34 am
Location: Ethiopia

Re: Scorpio 2.9.0 64-bit MCTS+NN Gauntlet for CCRL 40/40

Post by Daniel Shawul »

Ofcourse I am interested in reports of issues of installation. Sorry you felt that way, but i tried my best to address the issues you raised.
I just wasn't able to reproduce the problem, but we both suspected that my frequent updating could have been a problem.
Your questions here and the suggestion of a mixup shows that you didn't bother to look at any of it, and your continual ignoring of my posts in those issues creates the impression that you are not interested in receiving bug reports and working together on solving the problems. Things were quite different a year ago when I was helping you test your first MCTS version.
I think you are a bit unfair to me. I do appreciate anyone who do tests and report issues -- you might have noticed you are the first one to report issues to me on github so i have no reason to ignore you. Anyway lets leave that behind us. The install.sh script, as I have mentioned, downloads and configures scorpio for you and there will not be inconsistencies between scorpio/egbbdlll/networks.

best
Daniel