STS test suite and engine analysis interface

[elcabesa]

hi, i tried to run it but i can't get yur beautiful output, where I'm doing mistake?

Code: Select all

C:\Users\elcab_000\Downloads\App_STS_Rating_v3>STS_Rating_v3 -f "STS1-STS15_LAN.
epd" -e "vajolet2 1.45x64 pop.exe" --proto uci -h 256 --getrating
STS Rating v3.0

Intel(R) Core(TM) i5-4210U CPU @ 1.70GHz
Physical cores : 2
System         : Windows


Engine: vajolet2 1.45x64 pop.exe
Hash: 256, Threads: 1, MoveTime: 1.0s
Number of positions in STS1-STS15_LAN.epd: 1500

Your bench : 2.546908s
My bench   : 2.553400s
Analysis Time to get CCRL 40/4 rating estimate : 199ms
Starting engine vajolet2 1.45x64 pop.exe ...
Vajolet2 1.45


Id: STS(v15.0), Position: 100

Press enter key to exit

C:\Users\elcab_000\Downloads\App_STS_Rating_v3>

[elcabesa]

I'm sorry , and I'm also stupid
I just noticed th STS_Rating.txt file just now........

[Canoike]

What are the specifications for running STS_EEAI_v1.exe or version 3 ? Which OS and what hardware ?
I always get several error messages and the last one is : "AttibuteError <unknown>.NumberOfcores".

Thanks

[Ferdy]

What are the specifications for running STS_EEAI_v1.exe or version 3 ? Which OS and what hardware ?
I always get several error messages and the last one is : "AttibuteError <unknown>.NumberOfcores".

Thanks

This is developed under windows os. I use wmi to detect couple of hardware specs. I can probably revise this to use python cpu_info module to get hardware info.
What os did you use when you get that error message? If windows what hardware?

Display of CPU info is not that important. I can just bypass this if this is the cause of the trouble.

[Canoike]

I use Windows XP Professional 64 bits edition. The hardware is a PC with a Intel Core I5.

[Ferdy]

Replaced wmi with psutil and cpuinfo to get cpu brand name and number of cores info. Done some error handling that even if reading of cpu info fails the program will continue to run.
This version also uses new sts epd now with san move for winboard engines that outputs move in san notation. Can handle winboard st command with decimal value like st 0.2.
Not done yet with refactoring and error handling improvement, but it is better if you try this one first.

Download:
http://www.mediafire.com/download/k4q6u ... ing_v9.rar

There is a file called RunSTSRating.bat. Inside there is a command line,

Code: Select all

STS_Rating_v9 -f "STS1-STS15_LAN_v2.epd" -e "CDrill 1599.exe" --proto uci -h 128 --getrating

Replace that CDrill 1599.exe with sf6.exe (stockfish 6) for example and make sure sf6.exe is in same directory with this sts_rating_v9.exe file.
Something like this.

Code: Select all

STS_Rating_v9 -f "STS1-STS15_LAN_v2.epd" -e "sf6.exe" --proto uci -h 128 --getrating

[Canoike]

Thank you.
It worked on my computer with this new version.
The CPU has 4 physical cores and not 4 logical cores.
Windows XP Professional 64 bits edition.
It would be nice if your utility works under Linux because the same engine runs faster under this OS. From the fastest to the slowest OS : 1) Linux 2) MacOs 3) guess...

Stockfish latest dev, my compil from Github with mingW :

Code: Select all

STS_Rating_v9 -f "STS1-STS15_LAN_v2.epd" -e "stockfish20150910.exe" --proto uci -h 128 --getrating

Intel(R) Core(TM) i5-3450S CPU @ 2.80GHz
Physical Cores: 1, Logical Cores: 4
Engine: Stockfish 120915 64 POPCNT
Hash: 128, Threads: 1, time/pos: 0.166s
Test duration: 00:04:41
Expected time to finish: 00:04:54
STS rating: 2978

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     87     64     72     74     71     75     53     68     66     78     65     52     67     62     42    996
   Score    891    683    796    787    752    794    573    783    703    820    696    566    747    706    553  10850
Score(%)   89.1   68.3   79.6   78.7   75.2   79.4   57.3   78.3   70.3   82.0   69.6   56.6   74.7   70.6   55.3   72.3
  Rating   3724   2798   3301   3261   3105   3292   2308   3243   2887   3408   2856   2277   3083   2900   2219   2978

Komodo 9.2 x64

Code: Select all

STS_Rating_v9 -f "STS1-STS15_LAN_v2.epd" -e "ko
modo-9.2-64bit.exe" --proto uci -h 128 --getrating
Intel(R) Core(TM) i5-3450S CPU @ 2.80GHz
Physical Cores: 1, Logical Cores: 4
Engine: Komodo 9.2 64-bit
Hash: 128, Threads: 1, time/pos: 0.165s
Test duration: 00:04:24
Expected time to finish: 00:04:52
STS rating: 2990

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     85     67     71     74     74     75     49     66     66     71     66     52     70     63     47    996
   Score    874    741    789    770    768    800    561    736    706    778    725    573    764    703    602  10890
Score(%)   87.4   74.1   78.9   77.0   76.8   80.0   56.1   73.6   70.6   77.8   72.5   57.3   76.4   70.3   60.2   72.6
  Rating   3648   3056   3270   3185   3177   3319   2255   3034   2900   3221   2985   2308   3159   2887   2437   2990

[Ferdy]

Intel(R) Core(TM) i5-3450S CPU @ 2.80GHz
Physical Cores: 1, Logical Cores: 4

When you run the batch file, did you notice the following warning printed in console?

Code: Select all

Warning, psutil failed to determine the number of physical cores

[Canoike]

When you run the batch file, did you notice the following warning printed in console?

Code: Select all

Warning, psutil failed to determine the number of physical cores

[/quote]

No warning of psutil. Just this :

STS_Rating_v9 -f "STS1-STS15_LAN_v2.epd" -e "st
ockfish20150910.exe" --proto uci -h 128 --getrating
STS Rating v9.0

Intel(R) Core(TM) i5-3450S CPU @ 2.80GHz
Cores, physical: 1, logical: 4

Engine: stockfish20150910.exe
Hash: 128, Threads: 1, MoveTime: 1.0s
Number of positions in STS1-STS15_LAN_v2.epd: 1500

Your bench : 2.118753s
My bench : 2.553400s
Analysis Time to get CCRL 40/4 rating estimate : 166ms
Starting engine stockfish20150910.exe ...
id name: Stockfish 120915 64 POPCNT

[MikeB]

Now v3 has basic support for winboard (WB) engines. It does not approximate STS rating for WB engines but only for UCI engines.
For WB engines analysis time can be sent by level command, say you want the engine to analyze at around 200ms/pos, you may use
--mps 40 --tc 0:8 command line options. Only those WB engines that output similar to "move e2e4" - LAN notation are considered, and only those engines that hve support for setboard and level commands. Also the tool has no control on number of threads and hash size of the engines. But for uci engines you can control the hash and the threads by --hash <value> and --threads <value> respectively.

Sample command lines.

Code: Select all

STS_Rating_v3 -f "STS1-STS15_LAN.epd" -e "LambChop_1099.exe -hash 128" --proto wb --mps 300 --tc 1 --log

or

Code: Select all

STS_Rating_v3 -f "STS1-STS15_LAN.epd" -e "C:\Chess\engines\nobook\RomiChess_P3L\RomiChessp3L64.exe" --proto wb --mps 40 --tc 0:8 --log

For uci engines.

Code: Select all

STS_Rating_v3 -f "STS1-STS15_LAN.epd" -e "Stockfish 6.exe" --proto uci -h 128 --getrating

In --getrating the tool will determine the time and the threads.

Usage:

Code: Select all

program -f <epdfile> -e <engname> -t <numthreads> --movetime <timeinms>

Options&#58;
-f or --file <value>, for epd file input
-e or --engine <value>, name of engine
-h or --hash <value>, hash size in MB, default is 32 MB
-t or --threads <value>, for threads, Cores and Max CPUs setting
--movetime <value>, time in ms, default is 1000ms
--log, save engine log
--getrating, calculate CCRL 40/4 rating estimate for uci engines only
--mps <integer value>, moves per session for winboard engines
--tc <integer value in minutes or mm&#58;ss>, timce control for winboard engines
--proto <uci or wb>, protocol the engine supports

Example&#58;
Ex1. Analyze test.epd with 2 threads and 128 MB hash, at 3s/pos using sf 6.exe
STS_Rating -f test.epd -e "sf 6.exe" -h 128 -t 2 --movetime 3000

Tested WB engines.

Code: Select all

Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: LambChop_1099.exe -hash
Estimated time/pos: 0.200s
Test duration: 00:10:50
Expected time to finish: 00:05:45
Command: level 300 1 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     36     40     44     32     56     46     25     18     23     65     21     34     37     55     18    550
   Score    439    478    560    419    625    704    398    344    405    717    351    429    524    659    376   7428
Score(%)   43.9   47.8   56.0   41.9   62.5   70.4   39.8   34.4   40.5   71.7   35.1   42.9   52.4   65.9   37.6   49.5


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Averno081
Estimated time/pos: 0.200s
Test duration: 00:10:28
Expected time to finish: 00:05:45
Command: level 300 1 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     60     41     50     54     67     47     43     30     38     60     38     48     47     50     39    712
   Score    668    499    624    638    740    720    543    460    495    682    508    598    571    667    563   8976
Score(%)   66.8   49.9   62.4   63.8   74.0   72.0   54.3   46.0   49.5   68.2   50.8   59.8   57.1   66.7   56.3   59.8


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Myrddin_0.87-64
Estimated time/pos: 0.200s
Test duration: 00:05:00
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     52     47     43     44     65     54     40     22     37     58     34     46     64     53     30    689
   Score    606    581    551    588    696    747    559    391    508    673    476    590    723    651    480   8820
Score(%)   60.6   58.1   55.1   58.8   69.6   74.7   55.9   39.1   50.8   67.3   47.6   59.0   72.3   65.1   48.0   58.8


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Natwarlal_v0.14
Estimated time/pos: 0.200s
Test duration: 00:10:18
Expected time to finish: 00:05:45
Command: level 300 1 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     54     52     41     44     64     51     42     33     29     54     39     59     66     64     28    720
   Score    634    634    536    587    715    736    534    451    446    654    513    681    739    749    477   9086
Score(%)   63.4   63.4   53.6   58.7   71.5   73.6   53.4   45.1   44.6   65.4   51.3   68.1   73.9   74.9   47.7   60.6


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Gerbil_02_x64_ja
Estimated time/pos: 0.200s
Test duration: 00:06:14
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     37     30     31     35     58     53     30     26     24     55     31     37     45     49     29    570
   Score    482    410    491    457    691    751    412    364    403    633    440    485    565    626    483   7693
Score(%)   48.2   41.0   49.1   45.7   69.1   75.1   41.2   36.4   40.3   63.3   44.0   48.5   56.5   62.6   48.3   51.3


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: scorpio-276-64-ja
Estimated time/pos: 0.200s
Test duration: 00:07:54
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     73     58     61     67     69     68     54     32     46     67     45     67     59     65     35    866
   Score    784    687    739    761    751    849    617    474    592    740    554    764    697    755    608  10372
Score(%)   78.4   68.7   73.9   76.1   75.1   84.9   61.7   47.4   59.2   74.0   55.4   76.4   69.7   75.5   60.8   69.1


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\RomiChess_P3L\RomiChessp3L64
Estimated time/pos: 0.200s
Test duration: 00:05:47
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     51     49     51     45     63     56     46     30     42     67     41     46     51     63     22    723
   Score    605    575    612    572    718    763    585    457    517    742    532    583    641    724    407   9033
Score(%)   60.5   57.5   61.2   57.2   71.8   76.3   58.5   45.7   51.7   74.2   53.2   58.3   64.1   72.4   40.7   60.2


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\Thinker54D\ThinkerInert 5.4D x64 SP
Estimated time/pos: 0.200s
Test duration: 00:05:55
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     69     67     70     59     67     59     53     51     55     72     60     60     63     67     43    915
   Score    762    751    775    692    763    808    681    662    667    799    705    687    725    766    621  10864
Score(%)   76.2   75.1   77.5   69.2   76.3   80.8   68.1   66.2   66.7   79.9   70.5   68.7   72.5   76.6   62.1   72.4


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\Satana.2.0.7\Satana.2.0.7.w64bit
Estimated time/pos: 0.200s
Test duration: 00:06:51
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     18     28     22     25     34     34     27     18     18     61     14     15     19     34      5    372
   Score    334    385    339    322    430    597    355    314    275    676    279    247    287    458    176   5474
Score(%)   33.4   38.5   33.9   32.2   43.0   59.7   35.5   31.4   27.5   67.6   27.9   24.7   28.7   45.8   17.6   36.5


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\shallow-rev688-win-ja\shallow-rev688-win-ja\Windows\shallow-rev688-64-ja
Estimated time/pos: 0.200s
Test duration: 00:06:52
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     50     43     38     55     58     52     41     28     37     54     36     46     59     67     32    696
   Score    607    535    525    639    663    723    542    447    504    642    520    574    707    747    517   8892
Score(%)   60.7   53.5   52.5   63.9   66.3   72.3   54.2   44.7   50.4   64.2   52.0   57.4   70.7   74.7   51.7   59.3


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\Nemeton\Nemeton\Nemeton_1
Estimated time/pos: 0.200s
Test duration: 00:06:17
Expected time to finish: 00:05:45
Command: level 40 0:8 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     39     40     45     42     57     55     33     20     25     63     38     45     55     45     25    627
   Score    528    507    576    544    635    763    480    395    380    704    503    551    672    612    458   8308
Score(%)   52.8   50.7   57.6   54.4   63.5   76.3   48.0   39.5   38.0   70.4   50.3   55.1   67.2   61.2   45.8   55.4


Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: C:\Chess\engines\nobook\Knightx1.92\Knightx192
Estimated time/pos: 0.400s
Test duration: 00:08:52
Expected time to finish: 00:10:45
Command: level 40 0:16 0

  STS ID   STS1   STS2   STS3   STS4   STS5   STS6   STS7   STS8   STS9  STS10  STS11  STS12  STS13  STS14  STS15    ALL
  NumPos    100    100    100    100    100    100    100    100    100    100    100    100    100    100    100   1500
 BestCnt     47     49     47     46     57     46     40     27     35     60     38     49     55     64     27    687
   Score    567    555    600    589    642    709    518    397    475    689    506    580    654    756    452   8689
Score(%)   56.7   55.5   60.0   58.9   64.2   70.9   51.8   39.7   47.5   68.9   50.6   58.0   65.4   75.6   45.2   57.9

Download:
http://www.mediafire.com/download/3xyap ... ing_v3.rar

Very nice Ferdy!

Could you post/send the source so I can compile a mac version?

Best, Mike