First Naum 3.1 rating !?

[IWB]

Hello

I have finished the first 1000 games of Naum 3.1, played vs 10 Engines from the top to the bottom of the list. Have a look at rank 8 and 9.

Just 1 Thread !
Ponder ON !!
Time: 6min + 3s/move (average of 16.5min per game)
50 opening positions, changing colors = 100 games vs each opponent
256 MB Hash
Only 4 pc Table-/Bit-/Shredderbases (64 MB Cache)
No books and no learning, pure engine!
Equal Hardware of course

The bases for the whole Rating list is 67200 games now.


Program Elo + - Games Score Av.Op. Draws

1 Rybka 2.3.2a mp 1T : 2722 9 9 4000 70.6 % 2568 36.4 %
2 Rybka 1.2f : 2673 7 7 6600 66.0 % 2556 33.1 %
3 Zappa Mexico II x64 1T : 2623 13 13 1700 56.1 % 2578 40.9 %
4 Deep Shredder 11 x64 1T : 2600 11 11 2300 55.7 % 2558 35.0 %
5 Strelka 2.0 B x64 : 2599 16 16 1000 52.8 % 2577 43.5 %
6 Rybka 1.0 Beta x64 : 2595 11 11 2300 53.5 % 2568 36.7 %
7 Zappa Mexico X64 Final : 2593 9 9 3600 53.2 % 2568 40.4 %
8 Naum 3.1 x64 1T : 2592 17 17 1000 52.5 % 2573 39.9 %
9 Naum 3 x64 1T : 2591 13 13 1800 51.2 % 2580 38.5 %
10 Toga II 1.4 beta5c BB : 2578 14 14 1300 50.8 % 2571 42.5 %
11 HIARCS 12 MP 1T : 2577 15 15 1300 47.9 % 2590 39.8 %
12 Naum 2.2 : 2544 10 10 2700 43.5 % 2588 42.9 %
13 Rybka 1.0 Beta 32-bit : 2543 8 8 4400 50.1 % 2540 33.2 %
14 HIARCS 11.2 : 2537 7 7 6600 45.5 % 2566 36.3 %
15 Fruit 2.3 : 2532 22 22 600 51.9 % 2517 39.2 %
16 Fruit 05/11/03 : 2532 8 8 3900 43.1 % 2578 39.8 %
17 DS 10 Balmung : 2531 10 10 2600 49.9 % 2530 42.5 %
18 Loop 13.5 : 2525 8 8 4400 44.2 % 2564 37.6 %
19 LoopMP 12.32 : 2525 9 9 3800 49.3 % 2528 34.8 %
20 Loop M1-P : 2523 24 24 500 49.2 % 2527 39.6 %
21 Toga II 1.2.1a : 2520 7 7 6600 46.6 % 2542 35.2 %
22 ListMP 11.64b : 2518 12 12 2200 44.0 % 2558 35.9 %
23 Deep Shredder 10 x64 1T : 2512 7 7 5400 44.7 % 2547 37.8 %
24 HIARCS 11 MP 1T : 2507 19 19 800 44.9 % 2541 36.4 %
25 Naum 2.1 NoLearn : 2505 9 9 3700 44.6 % 2541 36.4 %
26 Toga II 1.3x4 : 2499 19 19 800 44.6 % 2535 38.9 %
27 Hiarcs X54 64bit UCI NORM : 2497 22 22 600 43.9 % 2537 36.5 %
28 Spike 1.2 Turin : 2477 7 7 6300 39.3 % 2551 36.0 %
29 DS 9.02 1T : 2467 16 16 1400 37.9 % 2551 29.1 %
30 Deep Sjeng 2.7 : 2456 13 13 1900 31.5 % 2589 33.7 %
31 Glaurung 2-epsilon/5 : 2446 16 16 1300 34.8 % 2554 30.9 %
32 Deep Sjeng 2.5 : 2386 20 20 900 29.9 % 2532 30.6 %

Bye
Ingo

PS: Unfortunately I can not make a "nice" table here.

[Le Fou numerique]

Bye
Ingo

PS: Unfortunately I can not make a "nice" table here.

Hi Ingo,

Just used Code

Code: Select all

    Program                          Elo    +   -  Games   Score   Av.Op.  Draws

1  Rybka 2.3.2a mp 1T             : 2722    9   9  4000    70.6 %   2568   36.4 %
2  Rybka 1.2f                     : 2673    7   7  6600    66.0 %   2556   33.1 %
3  Zappa Mexico II x64 1T         : 2623   13  13  1700    56.1 %   2578   40.9 %
4  Deep Shredder 11 x64 1T        : 2600   11  11  2300    55.7 %   2558   35.0 %
5  Strelka 2.0 B x64              : 2599   16  16  1000    52.8 %   2577   43.5 %
6  Rybka 1.0 Beta x64             : 2595   11  11  2300    53.5 %   2568   36.7 %
7  Zappa Mexico X64 Final         : 2593    9   9  3600    53.2 %   2568   40.4 %
8  Naum 3.1 x64 1T                : 2592   17  17  1000    52.5 %   2573   39.9 %
9  Naum 3 x64 1T                  : 2591   13  13  1800    51.2 %   2580   38.5 %
10 Toga II 1.4 beta5c BB          : 2578   14  14  1300    50.8 %   2571   42.5 %
11 HIARCS 12 MP 1T                : 2577   15  15  1300    47.9 %   2590   39.8 %
12 Naum 2.2                       : 2544   10  10  2700    43.5 %   2588   42.9 %
13 Rybka 1.0 Beta 32-bit          : 2543    8   8  4400    50.1 %   2540   33.2 %
14 HIARCS 11.2                    : 2537    7   7  6600    45.5 %   2566   36.3 %
15 Fruit 2.3                      : 2532   22  22   600    51.9 %   2517   39.2 %
16 Fruit 05/11/03                 : 2532    8   8  3900    43.1 %   2578   39.8 %
17 DS 10 Balmung                  : 2531   10  10  2600    49.9 %   2530   42.5 %
18 Loop 13.5                      : 2525    8   8  4400    44.2 %   2564   37.6 %
19 LoopMP 12.32                   : 2525    9   9  3800    49.3 %   2528   34.8 %
20 Loop M1-P                      : 2523   24  24   500    49.2 %   2527   39.6 %
21 Toga II 1.2.1a                 : 2520    7   7  6600    46.6 %   2542   35.2 %
22 ListMP 11.64b                  : 2518   12  12  2200    44.0 %   2558   35.9 %
23 Deep Shredder 10 x64 1T        : 2512    7   7  5400    44.7 %   2547   37.8 %
24 HIARCS 11 MP 1T                : 2507   19  19   800    44.9 %   2541   36.4 %
25 Naum 2.1 NoLearn               : 2505    9   9  3700    44.6 %   2541   36.4 %
26 Toga II 1.3x4                  : 2499   19  19   800    44.6 %   2535   38.9 %
27 Hiarcs X54 64bit UCI NORM      : 2497   22  22   600    43.9 %   2537   36.5 %
28 Spike 1.2 Turin                : 2477    7   7  6300    39.3 %   2551   36.0 %
29 DS 9.02 1T                     : 2467   16  16  1400    37.9 %   2551   29.1 %
30 Deep Sjeng 2.7                 : 2456   13  13  1900    31.5 %   2589   33.7 %
31 Glaurung 2-epsilon/5           : 2446   16  16  1300    34.8 %   2554   30.9 %
32 Deep Sjeng 2.5                 : 2386   20  20   900    29.9 %   2532   30.6 %

Regards,
Patrick

[IWB]

Ahhh, thx!

For some reason I played around with "List" - and failed of course.

Thx again - next tables will be better!

Bye
Ingo

Bye
Ingo

PS: Unfortunately I can not make a "nice" table here.

Hi Ingo,

Just used Code

Code: Select all

    Program                          Elo    +   -  Games   Score   Av.Op.  Draws

1  Rybka 2.3.2a mp 1T             : 2722    9   9  4000    70.6 %   2568   36.4 %
2  Rybka 1.2f                     : 2673    7   7  6600    66.0 %   2556   33.1 %
3  Zappa Mexico II x64 1T         : 2623   13  13  1700    56.1 %   2578   40.9 %
4  Deep Shredder 11 x64 1T        : 2600   11  11  2300    55.7 %   2558   35.0 %
5  Strelka 2.0 B x64              : 2599   16  16  1000    52.8 %   2577   43.5 %
6  Rybka 1.0 Beta x64             : 2595   11  11  2300    53.5 %   2568   36.7 %
7  Zappa Mexico X64 Final         : 2593    9   9  3600    53.2 %   2568   40.4 %
8  Naum 3.1 x64 1T                : 2592   17  17  1000    52.5 %   2573   39.9 %
9  Naum 3 x64 1T                  : 2591   13  13  1800    51.2 %   2580   38.5 %
10 Toga II 1.4 beta5c BB          : 2578   14  14  1300    50.8 %   2571   42.5 %
11 HIARCS 12 MP 1T                : 2577   15  15  1300    47.9 %   2590   39.8 %
12 Naum 2.2                       : 2544   10  10  2700    43.5 %   2588   42.9 %
13 Rybka 1.0 Beta 32-bit          : 2543    8   8  4400    50.1 %   2540   33.2 %
14 HIARCS 11.2                    : 2537    7   7  6600    45.5 %   2566   36.3 %
15 Fruit 2.3                      : 2532   22  22   600    51.9 %   2517   39.2 %
16 Fruit 05/11/03                 : 2532    8   8  3900    43.1 %   2578   39.8 %
17 DS 10 Balmung                  : 2531   10  10  2600    49.9 %   2530   42.5 %
18 Loop 13.5                      : 2525    8   8  4400    44.2 %   2564   37.6 %
19 LoopMP 12.32                   : 2525    9   9  3800    49.3 %   2528   34.8 %
20 Loop M1-P                      : 2523   24  24   500    49.2 %   2527   39.6 %
21 Toga II 1.2.1a                 : 2520    7   7  6600    46.6 %   2542   35.2 %
22 ListMP 11.64b                  : 2518   12  12  2200    44.0 %   2558   35.9 %
23 Deep Shredder 10 x64 1T        : 2512    7   7  5400    44.7 %   2547   37.8 %
24 HIARCS 11 MP 1T                : 2507   19  19   800    44.9 %   2541   36.4 %
25 Naum 2.1 NoLearn               : 2505    9   9  3700    44.6 %   2541   36.4 %
26 Toga II 1.3x4                  : 2499   19  19   800    44.6 %   2535   38.9 %
27 Hiarcs X54 64bit UCI NORM      : 2497   22  22   600    43.9 %   2537   36.5 %
28 Spike 1.2 Turin                : 2477    7   7  6300    39.3 %   2551   36.0 %
29 DS 9.02 1T                     : 2467   16  16  1400    37.9 %   2551   29.1 %
30 Deep Sjeng 2.7                 : 2456   13  13  1900    31.5 %   2589   33.7 %
31 Glaurung 2-epsilon/5           : 2446   16  16  1300    34.8 %   2554   30.9 %
32 Deep Sjeng 2.5                 : 2386   20  20   900    29.9 %   2532   30.6 %

Regards,
Patrick

[Tony Thomas]

I stopped my testing after 100 games when I found out that Naum 3.1 couldnt surpass the score of the previous version even with a perfect score.

Code: Select all

2 Naum 3   : 2725 51 49 172 78.5 % 2400 25.6 % 
5 Naum 3.1 : 2640 59 58 109 67.4 % 2414 26.6 %

[Nid Hogge]

I was hoping for a tad more than that. What opening positions wer'e used?

Naum shines on MP machines.. single CPU not so great, but plays so much better on 2CPU and Quad (Only behind Rybka and Zappa according to CEGT http://www.husvankempen.de/nunn/40_40%2 ... liste.html).

[IWB]

Hi Tony,

Opening positions or opening book?

Even if mathematicaly correct I doubt the number of games is sufficiant. I have seen more than the 1000 games in the meantime and basicly I doubt any significant difference between the two, especially a difference of 85 Elo !

In short: You should check you method or Naum 3.1 had just very bad luck!

Bye and keep on testing
Ingo

I stopped my testing after 100 games when I found out that Naum 3.1 couldnt surpass the score of the previous version even with a perfect score.

Code: Select all

2 Naum 3   : 2725 51 49 172 78.5 % 2400 25.6 % 
5 Naum 3.1 : 2640 59 58 109 67.4 % 2414 26.6 %

[IWB]

I have to agree,

but I do not test "parallel capabilities", I am interested in evaluation and search.

My opening positions are not "official" but as I said, I switch colors so independently of the quality of the opening they should be equal for both engines.

Bye
Ingo

I was hoping for a tad more than that. What opening positions wer'e used?

Naum shines on MP machines.. single CPU not so great, but plays so much better on 2CPU and Quad (Only behind Rybka and Zappa according to CEGT http://www.husvankempen.de/nunn/40_40%2 ... liste.html).

[Nid Hogge]

I have to agree,

but I do not test "parallel capabilities", I am interested in evaluation and search.

My opening positions are not "official" but as I said, I switch colors so independently of the quality of the opening they should be equal for both engines.

Bye
Ingo

I was hoping for a tad more than that. What opening positions wer'e used?

Naum shines on MP machines.. single CPU not so great, but plays so much better on 2CPU and Quad (Only behind Rybka and Zappa according to CEGT http://www.husvankempen.de/nunn/40_40%2 ... liste.html).

Fair enough.

Looking forward for newer versions to pop up soon.

Thanks for tests,

[Tony Thomas]

Hi Tony,

Opening positions or opening book?

Even if mathematicaly correct I doubt the number of games is sufficiant. I have seen more than the 1000 games in the meantime and basicly I doubt any significant difference between the two, especially a difference of 85 Elo !

In short: You should check you method or Naum 3.1 had just very bad luck!

Bye and keep on testing
Ingo

I stopped my testing after 100 games when I found out that Naum 3.1 couldnt surpass the score of the previous version even with a perfect score.

Code: Select all

2 Naum 3   : 2725 51 49 172 78.5 % 2400 25.6 % 
5 Naum 3.1 : 2640 59 58 109 67.4 % 2414 26.6 %

Yes, I used own book (tiny), I doubt that my results are accurate, but I can say with confidence that it is no whopping 50 point improvement. They are more or less the same strength, just had few extra draws, and lost few games it could have drawn.

[Tomcass]

Hardware: Quad 6700
Game in 10 minutes
All engines with Hiarcs 12 book.


2008Naum3.1-1 2008

Naum 3.1(x4) - Deep Hiarcs 12 11.0 - 9.0 +8/-6/=6 55.00%
Naum 3.1(x4) - Deep Hiarcs Paderborn 2007 12.5 - 7.5 +7/-2/=11 62.50%
Naum 3.1(x4) - Deep Shredder 11 UCI 11.0 - 9.0 +7/-5/=8 55.00%
Naum 3.1(x4) - Loop 13.6(x4) 14.5 - 5.5 +11/-2/=7 72.50%
Naum 3.1(x4) - Rybka 2.3.2a mp 32-bit 6.0 - 14.0 +1/-9/=10 30.00%
Naum 3.1(x4) - Toga II 1.4.2JD 10.5 - 9.5 +5/-4/=11 52.50%
Naum 3.1(x4) - Zappa Mexico II(x4) 9.5 - 10.5 +6/-7/=7 47.50%
Naum 3.1(x4) - Deep Fritz 10 13.0 - 7.0 +11/-5/=4 65.00%

My 'feelings'.

- A very strong program. Letting Rybka apart, only Zappa Mexico II was able to win Naum 3.1 in a very narrow fight. This result is consistent with the rating posted above.

- Comparing Naum 3.1 with Naum 3.0 in similar tests I have done, it seems to be a bit stronger. It is difficult for me be more precise here, but I think around 30/35 Elo points.

- Again a great performance of Toga II 1.4.2 JD. 47,50% against this very strong Naum 3.1 confirms that this version of Toga deserves close attention. I would like this version of Toga to be more tested by Talkchess members, please. Thanks!.

Regards from Barcelona.

Tom