MikeB wrote:Ferdy wrote:MikeB wrote:Latest developmental version of crafty.
At 10 seconds per move. iMac Retina 5K , 4 CORE i7 4790 4.0Ghz
Code: Select all
test results summary:
total positions searched.......... 1300
number right...................... 1056
number wrong...................... 244
percentage right.................. 81
percentage wrong.................. 18
total nodes searched..............355918365972
average search depth.............. 24.6
nodes per second.................. 27160114
total time........................ 218:24
not sure how this stacks against other programs...
STS has 15 themes now equivalent to 100x15=1500positions.
Download the complete epd here.
https://sites.google.com/site/strategictestsuite/
or if you process the alternative moves and point system, download the complete 15 themes in the following, at the bottom of the first post. There are couple of changes in the alternative moves, and also uniform formatting in the epd with additional opcodes converting SAN to LAN.
http://www.talkchess.com/forum/viewtopi ... 06&t=56653
BTW is there an option for Crafty to output move in LAN format like
e2e4 instead of e4?
Appreciate the reply. Crafty does output in LAN - option "output long".
I just knew it, unfortunately it also outputs the Piece letter as in Ng1f3. I thought it is g1f3, as in uci output. So in this case the tool is of no use to calculate points. Later I will implement SAN, but I need to create a new format on the epd. Also on my list is to support st (decimal and integer values in seconds) command for WB engines.
Looks like your program is pc. If you are interested, I can try to compile a mac version.
I am using windows, the source is a python script and converted to exe file. I am using python 2.7.6. Does python run on your system?
st 200ms- single core:
Code: Select all
test results summary:
total positions searched.......... 1500
number right...................... 862
number wrong...................... 638
percentage right.................. 57
percentage wrong.................. 42
total nodes searched.............. 2055634974
average search depth.............. 14.0
nodes per second.................. 6852116
total time........................ 5:00
White(1):
I run Crafty on my system also at 200ms/pos.
Code: Select all
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Physical Cores: 4, HT: ON
Crafty v24.1 (1 cpus)
White(1): bench
Running benchmark. . .
.......
Total nodes: 194234149
Raw nodes per second: 5242487
Total elapsed time: 37.05
STS 1-15, 0.2s/pos
test results summary:
total positions searched.......... 1500
number right...................... 826
number wrong...................... 674
percentage right.................. 55
percentage wrong.................. 44
total nodes searched.............. 1055659387
average search depth.............. 12.9
nodes per second.................. 3468114
total time........................ 5:04
Your machine is faster than mine. Here are some results from different engines, also at 200ms. We can only compare the BestCnt line.
Code: Select all
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Rodent 1.7 build 1
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:03:33
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 65 61 50 62 65 64 49 33 48 60 52 54 59 69 33 824
Score 746 680 646 728 726 819 621 509 598 670 633 620 707 783 558 10044
Score(%) 74.6 68.0 64.6 72.8 72.6 81.9 62.1 50.9 59.8 67.0 63.3 62.0 70.7 78.3 55.8 67.0
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Arasan 17.5
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:23
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 61 60 56 55 59 51 47 29 39 65 47 47 52 65 37 770
Score 688 674 678 643 690 762 600 486 527 741 617 574 636 760 575 9651
Score(%) 68.8 67.4 67.8 64.3 69.0 76.2 60.0 48.6 52.7 74.1 61.7 57.4 63.6 76.0 57.5 64.3
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Deuterium v14.3.34.130
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:25
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 67 61 63 57 67 67 59 43 38 69 45 60 69 63 31 859
Score 762 699 743 687 736 842 699 573 508 775 579 687 759 730 547 10326
Score(%) 76.2 69.9 74.3 68.7 73.6 84.2 69.9 57.3 50.8 77.5 57.9 68.7 75.9 73.0 54.7 68.8
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Gaviota v1.0
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:04
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 66 56 62 58 65 65 63 41 57 66 52 63 62 63 45 884
Score 743 680 714 724 717 817 717 559 642 750 602 712 729 724 695 10525
Score(%) 74.3 68.0 71.4 72.4 71.7 81.7 71.7 55.9 64.2 75.0 60.2 71.2 72.9 72.4 69.5 70.2
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Senpai 1.0
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:44
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 67 68 62 64 75 69 63 38 51 78 63 69 64 68 42 941
Score 756 746 741 749 794 829 730 575 643 852 759 768 753 801 627 11123
Score(%) 75.6 74.6 74.1 74.9 79.4 82.9 73.0 57.5 64.3 85.2 75.9 76.8 75.3 80.1 62.7 74.2
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Nemo SP64o 1.0.1 Beta
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:22
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 69 57 60 56 65 69 55 37 53 73 54 66 60 55 36 865
Score 769 697 699 684 723 843 676 527 630 803 655 743 699 673 577 10398
Score(%) 76.9 69.7 69.9 68.4 72.3 84.3 67.6 52.7 63.0 80.3 65.5 74.3 69.9 67.3 57.7 69.3
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Rhetoric 1.4.1 x64
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:44
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 61 62 63 55 66 59 56 41 53 68 45 63 63 63 39 857
Score 697 682 721 679 711 765 683 585 632 767 591 719 744 746 639 10361
Score(%) 69.7 68.2 72.1 67.9 71.1 76.5 68.3 58.5 63.2 76.7 59.1 71.9 74.4 74.6 63.9 69.1
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
Engine: Stockfish 6 64 POPCNT
Hash: 128, Threads: 1, time/pos: 0.200s
Test duration: 00:05:20
Expected time to finish: 00:05:45
STS ID STS1 STS2 STS3 STS4 STS5 STS6 STS7 STS8 STS9 STS10 STS11 STS12 STS13 STS14 STS15 ALL
NumPos 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 1500
BestCnt 84 76 73 70 77 76 69 68 69 81 73 72 78 75 48 1089
Score 874 847 818 807 821 893 787 794 765 864 795 786 842 849 708 12250
Score(%) 87.4 84.7 81.8 80.7 82.1 89.3 78.7 79.4 76.5 86.4 79.5 78.6 84.2 84.9 70.8 81.7
Comparison based on my machine. Rating in parenthesis is from CCRL 40/4.
Code: Select all
Arasan 17.5 770/1500 (2849)
Rodent 1.7 build 1 824/1500 (2833)
Crafty v24.1 (1 cpus) 826/1500 (No record, but v24.0 is 2801)
Rhetoric 1.4.1 x64 857/1500 (2794)
Deuterium v14.3.34.130 859/1500 (2889)
Nemo SP64o 1.0.1 Beta 865/1500 (2864)
Gaviota v1.0 884/1500 (2895)
Senpai 1.0 941/1500 (3016)
Stockfish 6 64 POPCNT 1089/1500 (3317)
Not really a good fit, just a rough formula to get a rating estimate for CCRL 40/4 based on bestmove counts percentage from 1500 pos, at 200ms/pos.
I think Crafty does fine here as with other engines though the data is not that big.
Code: Select all
sts_rating = 25.212 x bestmove_percentage + 1439.4