uci eval command

[Ferdy]

this post is just to report some result I got comparing eval of egines

standard deviation of eval of a selected pool of positions

stockfish6 vs stockfish -> 0.20
critter vs stockfish6 -> 0.62
vajolet vs stockfish6 -> 0.7
gaviota vs stockfish6 -> 1.23

this is incidentally the same order of elo strenght that we have in CCRL, can we draw any conclusion?? I don't think so

Which version is stockfish in stockfish6 vs stockfish -> 0.20?

What do you mean by "same order of elo strenght"?

1) stockfish 6 vs stockfish_14053109
2) I considered the elo strenght in CCRL 40 1 cpu, I considered stockfish evalutation to be the strongest and evalutated the strenght of the other looking at the std deviation from stockfish.
It seems that the higher the standard deviation, the weaker is the engine.

Thanks nice idea.

One idea I had is to use sf games for tuning. Example collect some games by sf with score comments. Convert it to epd with score, tune the engine minimizing its score difference to that of sf score in epd, similar to Texel tuning, except that it uses the eval of sf instead of results. I don't know if someone has already tried this. Initially I thought of downloading strong engine games from Frank, moves in these games are reliable, compared to tuning which uses qsearch values in my case.

[cdani]

One idea I had is to use sf games for tuning. Example collect some games by sf with score comments. Convert it to epd with score, tune the engine minimizing its score difference to that of sf score in epd, similar to Texel tuning, except that it uses the eval of sf instead of results. I don't know if someone has already tried this. Initially I thought of downloading strong engine games from Frank, moves in these games are reliable, compared to tuning which uses qsearch values in my case.

When Andscacs was may be like 2200 I used a similar system to improve its evaluation, as I commented somewhere, and I was able to improve it to may be 2450 in a week.

I'm sure if I had know something more of mathematics, I could be able to do it better, because I had just done some basic means by intuition, not knowing really which was the right method.

The database of positions was done by taking some pgn and passing each position to stockfish with the eval command.

[Ferdy]

One idea I had is to use sf games for tuning. Example collect some games by sf with score comments. Convert it to epd with score, tune the engine minimizing its score difference to that of sf score in epd, similar to Texel tuning, except that it uses the eval of sf instead of results. I don't know if someone has already tried this. Initially I thought of downloading strong engine games from Frank, moves in these games are reliable, compared to tuning which uses qsearch values in my case.

When Andscacs was may be like 2200 I used a similar system to improve its evaluation, as I commented somewhere, and I was able to improve it to may be 2450 in a week.

I'm sure if I had know something more of mathematics, I could be able to do it better, because I had just done some basic means by intuition, not knowing really which was the right method.

The database of positions was done by taking some pgn and passing each position to stockfish with the eval command.

Right I will try it but will use the existing score of the move in the pgn.

[cdani]

Right I will try it but will use the existing score of the move in the pgn.

In the moment I tried also with the evaluation at fixed depth of Stockfish, may be 10, I do not remember, but the result was clearly worst. I suppose somehow the search evaluation does not fit well with the eval function, probably related to not containing parameters than can predict the outcome of the search.

[Robert Pope]

Can someone explain this Stockfish output to me?

Code: Select all

Stockfish 6 by Tord Romstad, Marco Costalba and Joona Kiiski
position fen 1b6/8/8/8/8/4k2p/8/7K w - -
d

 +---+---+---+---+---+---+---+---+
 |   | b |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   | k |   |   | p |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   |   |
 +---+---+---+---+---+---+---+---+
 |   |   |   |   |   |   |   | K |
 +---+---+---+---+---+---+---+---+

Fen: 1b6/8/8/8/8/4k2p/8/7K w - - 0 1
Key: 6F98708514DB90FF
Checkers:
eval
      Eval term |    White    |    Black    |    Total
                |   MG    EG  |   MG    EG  |   MG    EG
----------------+-------------+-------------+-------------
       Material |   ---   --- |   ---   --- | -2.95 -4.83
      Imbalance |   ---   --- |   ---   --- |  0.05  0.05
          Pawns |   ---   --- |   ---   --- |  0.14  0.17
        Knights |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
        Bishops |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
          Rooks |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
         Queens |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
       Mobility |  0.00  0.00 |  0.29  0.33 | -0.29 -0.33
    King safety | -0.71  0.00 |  0.00 -0.19 | -0.71  0.19
        Threats |  0.00  0.00 |  0.00  0.00 |  0.00  0.00
   Passed pawns |  0.00  0.00 |  0.84  0.49 | -0.84 -0.49
          Space |  0.00  0.00 |  0.01  0.00 | -0.01  0.00
----------------+-------------+-------------+-------------
          Total |   ---   --- |   ---   --- | -4.62 -5.27

Total Evaluation: 0.07 (white side)

Why is Total Evaluation 0.07? Shouldn't it be between -4.62 and -5.27? And closer to -5.27 since this is an endgame position?

ETA: I just noticed the ---'s. Does that mean Stockfish is ignoring that component of the evaluation, i.e. material balance is irrelevant in this position?

[cdani]

This endgame is a theoretical draw, so is the final eval. The components of the eval are basically ignored.

[op12no2]

Here v2 with improved fmvn parsing.

Download:
http://www.mediafire.com/download/h62m3 ... v2Pack.rar

This is great Ferdinand. Just used v2 to generate an EPD that I intend to use to tune my eval in a simple least squares match way.

[op12no2]

This is a lot of fun. I downloaded a wodge of Stockfish games from CCRL and used Ferdinands script to generate ~65k EPDs, which reduced to ~62k unique positions. I compared the eval to my engine to find the biggest difference:-

[d]3rq3/p4r1k/bpnppnp1/1N2b1B1/1PP1PNP1/P6Q/6B1/3R1RK1 b - -

My eval is an order of magnitude less !! than SFs eval of 683 which is based largely on the white attack of the black king position. Seems a huge value.

Now doing some tuning by getting a least squares fit across these 62k positions which is both interesting and pretty quick...

[Robert Pope]

I just ran 800K positions through SF and Abbess and sorted by absolute difference. I haven't had a chance to review them yet, but I'll toss a couple random ones up here. I think this will end up being an instructive exercise...

Abbess: 93 Stockfish: 3978
[d]8/1K6/1P6/8/8/8/8/3k4 w - - fmvn 12
Obviously, Abbess' eval needs some unstoppable pawn code


Abbess: 505 Stockfish: 157
[d] r2kN3/1p1P4/7p/r6P/8/4R3/p4PP1/3R2K1 w - - fmvn 12

[Ferdy]

This is a lot of fun. I downloaded a wodge of Stockfish games from CCRL and used Ferdinands script to generate ~65k EPDs, which reduced to ~62k unique positions. I compared the eval to my engine to find the biggest difference:-

[d]3rq3/p4r1k/bpnppnp1/1N2b1B1/1PP1PNP1/P6Q/6B1/3R1RK1 b - -

My eval is an order of magnitude less !! than SFs eval of 683 which is based largely on the white attack of the black king position. Seems a huge value.

Now doing some tuning by getting a least squares fit across these 62k positions which is both interesting and pretty quick...

One variation I tried is to calculate error this way.
if(my_eval > sf_eval) error = 0
else error = sqrt[(my_eval - sf_eval)^2]
The idea is we consider sf eval as best, if there is a score greater
than that we just consider that as as good as sf eval, making the error zero. I got a slight improvement doing this way rather than always getting
the square of the difference. Other than that I use qsearch value and compare it with the score of the move from Frank's commented pgn files.
Positions from drawn games are not included and positions with move score > 500 cp are also not included.