SF6 moves comparison: Event Vugar Gashimov Mem 2015

Ferdy · Post by **Ferdy** » Mon Apr 27, 2015 11:49 am

Guenther wrote:Hi Ferdinand,

You should also exclude one time repetition moves, because Humans just play them to gain time to reach the time control and have more time to calculate the winning line.
If the weaker side really has nothing better then to repeat too, there is nothing lost for the better side!

Guenther

I will look into that, this will also save analysis time.

Vinvin · Post by **Vinvin** » Mon Apr 27, 2015 12:25 pm

Ferdy wrote: This is just a quick sample using sf6 at 10 ms / position. As you can see the EECnt (engine error count) is high and the ave depth is low.
I will give more analysis time later.

Yes, for error 0.2 to 0.49, I'd give at least 30 sec/position on a single core or 10 sec/position on a quad-core. This, because we need more accurate analysis.
I'll give a shot with my 6cores@4GHz this evening.

A new idea to avoid that the human move gives a higher score than the engine : First, analyze the position after the human move, then backup one move (to preserve hash table) and analyze the position. But I'm not sure it's feasible with a batch.

Ferdy · Post by **Ferdy** » Mon Apr 27, 2015 1:41 pm

Vinvin wrote: A new idea to avoid that the human move gives a higher score than the engine : First, analyse the position after the human move, then backup one move (to preserve hash table) and analyse the position. But I'm not sure it's feasible with a batch.

This is possible, but this is unfair to the human. I am not going to change my current script.
Here we also see not only the human weakness but also probably the computer weakness (mainly horizon effect and low analysis time, I guess as human move is done on the board, before the engine will do the analysis). Right I will save such position when human moves is better than computer according to computer.

Vinvin · Post by **Vinvin** » Mon Apr 27, 2015 3:29 pm

Ferdy wrote:Here it is, also with new info on dangerous error counts, DECnt. This is the count of positions where players' position goes below zero after a move. This feature will give us info how many times a player is in trouble.

Nice !
Please, could I get your latest executable and the EPD file for the whole Shamkir-2015 ?
One more question : if I set 10 sec per position, will it use 20 sec per EPD lines (10 for the engine's move and 10 for the player's move) ?

Thanks,
Vincent

Ferdy · Post by **Ferdy** » Mon Apr 27, 2015 4:29 pm

Vinvin wrote:
Ferdy wrote:Here it is, also with new info on dangerous error counts, DECnt. This is the count of positions where players' position goes below zero after a move. This feature will give us info how many times a player is in trouble.
Nice !
Please, could I get your latest executable and the EPD file for the whole Shamkir-2015 ?
One more question : if I set 10 sec per position, will it use 20 sec per EPD lines (10 for the engine's move and 10 for the player's move) ?

Thanks,
Vincent

This is not ready yet. I make changes to EPD also remove repeat positions, also modify the tool to output positions where player move is better.

Not exactly 10 sec for computer and 10 sec for player. What I do is analyze given position by computer for given time say 10 sec and save score and move. If the best move matches the player move then I do not search the player move anymore. If the best move found by computer is not the same to that of player, I will let the engine (new instance) analyze the position plus the move of the player for same given time say 10 sec. Save the move and score then make some calculations, compare moves, get score diff and others.
I will upload the tool once it is ready.

Ferdy · Post by **Ferdy** » Mon Apr 27, 2015 7:21 pm

New version see notes for some features. This will also save positions where player move is better than engine suggested move.
The epd file is no longer displaying the correct full move number as well as half move clock, as position duplicates are removed in the attached epd file.

Output files:
1) So, Wesley_blunder_20cp_49cp.txt - blunder files per player per error range are in append mode, make sure to delete or make a backup copy if you need one, before running new analysis session.
2) So, Wesley_better_than_engine.txt - also append mode, same as in (1).
3) move_error_summary.txt - append mode so previous results are saved. Important file.
4) move_error_log.txt - analysis log are here, this is overwrite mode, as this may get bigger for longer analysis.
5) pair.txt - overwrite mode, temp file for program
6) players.txt - overwrite mode, temp file for program

Sample, 0.20 sec/pos engine analysis.

Code: Select all

Move Error Evaluator v8.7

Moves Error calculation based from Stockfish 6 64 POPCNT and players from
Event Vugar Gashimov Mem 2015, after round 9.5

Intel&#40;R&#41; Core&#40;TM&#41; i7-2600K CPU @ 3.40GHz
Engine id name Stockfish 6 64 POPCNT
Hash 128 MB, Threads 1, MoveTime 0.20 sec/pos

 Rank                       Players  AveDepth  TotalPos  AveError  DECnt  E300CP  E200CP  E100CP  E50CP  E20CP  EECnt   EERate    W    L    D  Games  Score  SRate

    1               Carlsen, Magnus        15       280      9.68     30       0       0       1      5     29     29   10.36%    5    0    4      9    7.0   77.8%
    2            Anand, Viswanathan        14       274     12.72     46       0       0       3      8     24     30   10.95%    3    0    6      9    6.0   66.7%
    3                 Mamedov, Rauf        14       254     13.39     43       0       0       2      4     26     31   12.20%    0    2    7      9    3.5   38.9%
    4        Mamedyarov, Shakhriyar        14       300     14.70     44       1       1       1      6     30     33   11.00%    1    2    6      9    4.0   44.4%
    5              Caruana, Fabiano        15       321     15.68     57       0       0       1      6     33     38   11.84%    2    1    6      9    5.0   55.6%
    6                    So, Wesley        14       281     15.77     42       0       1       0      7     25     23    8.19%    3    2    4      9    5.0   55.6%
    7       Vachier-Lagrave, Maxime        15       303     16.49     50       1       0       2      5     26     35   11.55%    0    2    7      9    3.5   38.9%
    8                   Giri, Anish        14       296     19.92     51       0       1       3     11     26     23    7.77%    0    2    7      9    3.5   38.9%
    9             Kramnik, Vladimir        14       333     23.31     62       0       1       3      8     38     31    9.31%    2    3    4      9    4.0   44.4%
   10                Adams, Michael        14       251     28.43     68       2       1       1      8     36     19    7.57%    1    3    5      9    3.5   38.9%

Overall Average Error&#58; 17.01

Legend&#58;
Rank     - ranking based on low average error
AveDepth - average depth reached by the engine in analyzing those positions
AveError - average difference between the engine and player score, in centipawn &#40;1p = 100&#41;
DECnt    - count of positions where position score is below zero, as a result of move error.
         - This is called Dangerous Error Counts
E300CP   - count of positions where player has blundered by 3 or more pawns
E200CP   - count of positions where player has blundered by 2 to 2.99 pawns
E100CP   - count of positions where player has blundered by 1 to 1.99 pawns
E50CP    - count of positions where player has blundered by 0.5 to 0.99 pawns
E20CP    - count of positions where player has blundered by 0.2 to 0.49 pawns
EECnt    - count of positions where player move score is better than engine move score
EERate   - 100*EECnt/TotalPos or the percentage of the Engine Error Counts

Notes&#58;
1. Analysis of positions start at move 12.
2. If best score of engine is already above 3 pawns and the score of player move
   is also above 3 pawns then error for this position is set to zero. The more cases
   like this only proves that that player is able to maintain its advantage and this
   will minimize its average error.
   On the other hand if the best score of engine is already below minus 3 pawns, no further
   analysis will be done, and this position will not be included in error calculation.
3. When a player move is a blunder that brings down the score below minus 3 pawns,
   error for this position is set to 6 pawns in the calculation. The reason being once
   the opponent found the refutation, the succeeding positions will no longer be evaluated.
   It is just fair to penalize this heavily to also reflect on the average error which
   should be high for players that have more game loses or fatal blunders.
4. move_error = engine_score - player_score.
   engine_score = score returned by the engine after searching for a given amount of time.
   player_score = score returned by the engine after searching the player move for a given amount of time.
   AveError = sum of &#91;move_error&#93; / totalPos.
5. Error is increased if the score of position after player move has dropped to below zero.
   6x error if &#40;score <= -200 cp&#41;, 5x error if &#40;score <= -150 cp&#41;, 4x error if &#40;score <= -100 cp&#41;,
   3x error if &#40;score <= -50 cp&#41;, and 2x error if &#40;score < 0 cp&#41;.
   The idea is only those errors which will result to inferior position &#40;below zero score&#41; should be penalized more.
   If the increased error penalty is more than 3 pawns, error is set to 3 pawns.
6. Regarding EECnt or Engine error counts, this is a counter check on engine where the human move
   is sometimes better than the engine move, this is because the analysis time given to the engine is
   shorter and also perhaps the position is complicated that bigger depths are required to determine
   correct assessment of the position.
7. DECnt or Dangerous Error Counts info is added to see which player is always in trouble. Strong players
   usually have low count of this.

Sample epd where human move is better. Check c4 field for engine move.
From Anand, Viswanathan_better_than_engine.txt:

Code: Select all

r4rk1/3q1ppp/p5b1/1p2B3/1b6/2NP1Q2/PP3PPP/R3R1K1 w - - fmvn 1; sm "c3d5"; c3 "Result 1/2-1/2"; c2 "Anand, Viswanathan / Carlsen, Magnus"; c1 "Round 1.3"; c0 "Event Vugar Gashimov Mem 2015"; hmvc 0; c4 "engine bestmove a2a3";

Download:
http://www.mediafire.com/download/xl8ds ... E_v8.7.rar

Next version, save data to sqlite database to easily get information thru queries.

Ferdy · Post by **Ferdy** » Mon Apr 27, 2015 9:43 pm

Quick comparison with Linares 1994.

Code: Select all

Moves Error calculation based from Stockfish 6 64 POPCNT and players from
Event Vugar Gashimov Mem 2015, after round 9.5

Intel&#40;R&#41; Core&#40;TM&#41; i7-2600K CPU @ 3.40GHz
Engine id name Stockfish 6 64 POPCNT
Hash 128 MB, Threads 1, MoveTime 0.20 sec/pos

 Rank                       Players  AveDepth  TotalPos  AveError  DECnt  E300CP  E200CP  E100CP  E50CP  E20CP  EECnt   EERate    W    L    D  Games  Score  SRate

    1               Carlsen, Magnus        15       280      9.68     30       0       0       1      5     29     29   10.36%    5    0    4      9    7.0   77.8%
    2            Anand, Viswanathan        14       274     12.72     46       0       0       3      8     24     30   10.95%    3    0    6      9    6.0   66.7%
    3                 Mamedov, Rauf        14       254     13.39     43       0       0       2      4     26     31   12.20%    0    2    7      9    3.5   38.9%
    4        Mamedyarov, Shakhriyar        14       300     14.70     44       1       1       1      6     30     33   11.00%    1    2    6      9    4.0   44.4%
    5              Caruana, Fabiano        15       321     15.68     57       0       0       1      6     33     38   11.84%    2    1    6      9    5.0   55.6%
    6                    So, Wesley        14       281     15.77     42       0       1       0      7     25     23    8.19%    3    2    4      9    5.0   55.6%
    7       Vachier-Lagrave, Maxime        15       303     16.49     50       1       0       2      5     26     35   11.55%    0    2    7      9    3.5   38.9%
    8                   Giri, Anish        14       296     19.92     51       0       1       3     11     26     23    7.77%    0    2    7      9    3.5   38.9%
    9             Kramnik, Vladimir        14       333     23.31     62       0       1       3      8     38     31    9.31%    2    3    4      9    4.0   44.4%
   10                Adams, Michael        14       251     28.43     68       2       1       1      8     36     19    7.57%    1    3    5      9    3.5   38.9%

Overall Average Error&#58; 17.01

Code: Select all

Moves Error calculation based from Stockfish 6 64 POPCNT and players from
Event Linares 1994, after round 13.0

Intel&#40;R&#41; Core&#40;TM&#41; i7-2600K CPU @ 3.40GHz
Engine id name Stockfish 6 64 POPCNT
Hash 128 MB, Threads 1, MoveTime 0.20 sec/pos

 Rank                       Players  AveDepth  TotalPos  AveError  DECnt  E300CP  E200CP  E100CP  E50CP  E20CP  EECnt   EERate    W    L    D  Games  Score  SRate

    1               Karpov, Anatoly        14       418     10.45     49       0       0       2     14     37     45   10.77%    9    0    4     13   11.0   84.6%
    2                Shirov, Alexei        15       465     20.35     78       2       0       3     23     39     36    7.74%    6    2    5     13    8.5   65.4%
    3                Bareev, Evgeny        14       416     21.33     50       1       4       6     15     40     28    6.73%    5    3    5     13    7.5   57.7%
    4                Kasparov, Gary        14       355     22.57     56       1       4       4     19     34     34    9.58%    6    2    5     13    8.5   65.4%
    5             Kramnik, Vladimir        13       352     23.91     55       1       2       5     18     36     29    8.24%    4    3    6     13    7.0   53.8%
    6             Ivanchuk, Vassily        14       363     25.27     53       0       0       7     16     40     27    7.44%    4    5    4     13    6.0   46.2%
    7            Anand, Viswanathan        13       321     26.10     56       0       0       4     20     30     37   11.53%    4    4    5     13    6.5   50.0%
    8                  Kamsky, Gata        14       403     27.34     83       1       0       8     15     54     30    7.44%    3    3    7     13    6.5   50.0%
    9                 Lautier, Joel        14       447     27.47     71       3       3       7     23     45     23    5.15%    5    4    4     13    7.0   53.8%
   10              Topalov, Veselin        14       400     30.69     63       2       4       4     29     40     25    6.25%    5    5    3     13    6.5   50.0%
   11                Gelfand, Boris        13       382     36.29     71       1       2      10     33     44     24    6.28%    1    3    9     13    5.5   42.3%
   12      Illescas Cordoba, Miguel        13       351     40.17     88       1       2      10     19     33     26    7.41%    4    8    1     13    4.5   34.6%
   13        Beliavsky, Alexander G        14       354     49.84     99       1       2      14     23     55     27    7.63%    0    9    4     13    2.0   15.4%
   14                 Polgar, Judit        13       337     51.44    109       1       3       8     24     47     35   10.39%    2    7    4     13    4.0   30.8%

Overall Average Error&#58; 29.52

Better ave error in Gashimov. Shirov has good results even though he had 2 positions with 3 pawns or more error.
Here is one of the 2 positions saved in Shirov, Alexei_blunder_300cp_or_more.epd, he played b2b1 in this position.

Code: Select all

4R3/5p1k/6pp/8/1pQ5/2n4P/1q3PP1/6K1 b - - fmvn 1; sm "b2b1"; c3 "Result 1-0"; c2 "Bareev, Evgeny / Shirov, Alexei"; c1 "Round 3"; c0 "Event Linares 1994"; hmvc 0;

[d]4R3/5p1k/6pp/8/1pQ5/2n4P/1q3PP1/6K1 b - -
Kramnik has somewhat the same ave error in both tournaments. Carlsen leads in ave error in this analysis setup.

Vinvin · Post by **Vinvin** » Tue Apr 28, 2015 8:40 am

Ferdy wrote:...
Download:
http://www.mediafire.com/download/xl8ds ... E_v8.7.rar

I'm running 20 sec/pos on 6cores@4GHz with the latest SF. There still a lot where the human play _better_than_engine.
Results this evening !

Ferdy · Post by **Ferdy** » Tue Apr 28, 2015 9:30 am

Vinvin wrote:
Ferdy wrote:...
Download:
http://www.mediafire.com/download/xl8ds ... E_v8.7.rar
I'm running 20 sec/pos on 6cores@4GHz with the latest SF. There still a lot where the human play _better_than_engine.
Results this evening !

It is nice to see your result.

There is another method so we get a nice comparison, takes time but more fair.
1. Given pos n analyze it with engine at given time. Save the best score and best move.
2. If the best move in (1) is the same to human move then error is set to 0. No further analysis will be done for human move.
3. If the move is not the same then continue the engine analysis. This time
the position to be analyzed is pos n + best move saved in (1). Save best score and best move when analysis is done. Then for player analyze pos n + player move in actual game. Save also the best score and best move. Then make calculation based on those info.

If player move score is better then it is really better based from the analysis time given to engine in (1).

I will implement this in next version.

Vinvin · Post by **Vinvin** » Tue Apr 28, 2015 9:43 am

Ferdy wrote:
Vinvin wrote:
Ferdy wrote:...
Download:
http://www.mediafire.com/download/xl8ds ... E_v8.7.rar
I'm running 20 sec/pos on 6cores@4GHz with the latest SF. There still a lot where the human play _better_than_engine.
Results this evening !
It is nice to see your result.

There is another method so we get a nice comparison, takes time but more fair.
1. Given pos n analyze it with engine at given time. Save the best score and best move.
2. If the best move in (1) is the same to human move then error is set to 0. No further analysis will be done for human move.
3. If the move is not the same then continue the engine analysis. This time
the position to be analyzed is pos n + best move saved in (1). Save best score and best move when analysis is done. Then for player analyze pos n + player move in actual game. Save also the best score and best move. Then make calculation based on those info.

If player move score is better then it is really better based from the analysis time given to engine in (1).

I will implement this in next version.

I thought about 2 other things too :
1) I had a trouble : usually I set the hash table to the half of my memory (16 GB/2). But MEE start 2 engines at the same time (or start a new engine when the old one have not free memory yet). This cause a big trouble with all the memory was monopolized by this 2 engines

.

2) An idea from the app "ChessGameReport". It use several threads to analyze the positions (but only 1 thread by engine). This is more efficient because there's no lost caused by multi-threading in one engine and more, the search is deterministic.

SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015

Re: SF6 moves comparison: Event Vugar Gashimov Mem 2015