Comparative nodes per second

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: Comparative nodes per second

Post by bob »

BubbaTough wrote:
bob wrote:
BubbaTough wrote:
Don wrote:
I think you are saying that this exact position is one that an evaluation function should try to evaluate and I don't know why you think we disagree - I didn't say one way or other about whether you should try.

In Komodo we try to do special evaluation when we think we can do it reliably. I don't really know if you can do that position reliably or not. Maybe you can but you didn't specify what class of positions you are talking about. If you mean that EXACT position then it's easy to evaluate, just test for it and score it as a win if you get an exact match.

So you have to specify what specifically is it in that position that you think you can reliably evaluate. Do you mean any position where there is a pawn on the 7th and the opponent doesn't have a pawn on the 7th? What rule covers that STATICALLY without a search? What if the side with the pawn on the 7th can be mated immediately? Now if you impose too many constraints, you make the rule almost useless.

In fact Larry and I decided to work on square of the pawn a while back and we had a very difficult time coming up with RELIABLE rules! We kept adding constraints so that we would not score a win if there was some doubt but after some time we discovered that it's not possible to make this rule perfect, and in fact the harder we tried, the less often the rule could even be used. So this simple rule, to really get right, requires a large decision tree with all sorts of rules - the type of thing you usually don't want to put in a chess program.

Anyway, propose a generally applicable rule that applies to this position and many others and let's see if it's simple or not. What are your rules?
The rule is, 2 connected passed pawns on the 6th (or 7th) can force a queen against a rook if the king is not close enough. Like the square of the pawn rule, you can find situations where it is not a win, but, like the square of the pawn rule, it is winning enough of the time that it is worth using search to find the exceptions rather than to use search to replicate its findings.

Its not uncommon for strong players to sacrifice if they see they can get 2 pawns on the 6th against a rook, without bothering to calculate much further.

-Sam
I think it is dangerous to assume a win and let the search find a refutation if one exists. A deep / wide search can use (still) the horizon effect to its advantage here to cause embarrassing moves in real games. I trust the search more than the eval when we are talking about races, checks, interference squares and such. I suppose it boils down to how much error, and where, are you willing to tolerate?
I think it really only boils down to which is more accurate, more often. If you look up rook vs. 2 connected passed on the 6th in Dvoretsky's endgame manual, you will see all sorts of tricks showing the rook finding a draw against the pawns. But these are really just exceptions to the rule. In a vast majority of cases the pawns win. VAST majority. And I find it embarrassing in real games to miss these, which is a risk if you prune a lot. Any statement like "I suppose it boils down to how much error, and where, are you willing to tolerate?" can always be turned on it head. How much error in your Eval's assumption that the rook is better than the pawns are YOU willing to tolerate, given the embarrassing moves relying on your search may lead you to?

-Sam
After my experiences (bad) with two connected passers on 6th vs rook, I simply want to let the search discover something good, rather than depending on it to discover something bad. Where the eval gets it wrong in a fast game, I might be stuck with that. In the majority of cases, two connected passers on the 6th are not better than a rook. Unless you tightly constrain it such as "opponent ONLY has a rook, king is at LEAST N squares away, King is not hemmed in and subject to perpetual by the rook, and is not getting mated." I'd prefer to avoid quick searches that push the pawns to the 6th, dropping a pair of bishops or something in the process, and assume it is winning...

Won't begin to argue my approach is the right one, just that it is "my approach" here after getting burned a few times on ICC with that connected on 6th eval term.
BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 5:18 am

Re: Comparative nodes per second

Post by BubbaTough »

bob wrote: After my experiences (bad) with two connected passers on 6th vs rook, I simply want to let the search discover something good, rather than depending on it to discover something bad. Where the eval gets it wrong in a fast game, I might be stuck with that. In the majority of cases, two connected passers on the 6th are not better than a rook. Unless you tightly constrain it such as "opponent ONLY has a rook, king is at LEAST N squares away, King is not hemmed in and subject to perpetual by the rook, and is not getting mated." I'd prefer to avoid quick searches that push the pawns to the 6th, dropping a pair of bishops or something in the process, and assume it is winning...

Won't begin to argue my approach is the right one, just that it is "my approach" here after getting burned a few times on ICC with that connected on 6th eval term.
I understand your perspective. The two contraints you mention I usually use are something like "opponent ONLY has a rook, king is at LEAST N squares away". In my opinion, in a large majority of these cases, 2 connected on the 6th ARE better than a rook under these constraints, but I must reluctantly admit it is only an opinion. In studies, mating threats are the most common method of constructing perpetuals, and they do come up in practice, but I believe they are the exception rather than the rule. Here is a position I just made up I consider typical:

[D]r7/5ppk/2PP4/8/8/8/5P2/6K1 b - - 7 30

I have not actually checked if it is a win for the pawns, but since there are no mate threats I think its safe to assume it is :). Note, its very very very rare for this to be a win for the rook, so its darn safe to assume the pawns are work AT LEAST a rook.

-Sam
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Comparative nodes per second

Post by michiguel »

bob wrote:
Uri Blass wrote:
Don wrote:
lucasart wrote:
Uri Blass wrote:Here is another example for the huge positional score of chess programs(probably also includes IvanHoe that I did not install in my computer and at least it includes robbolito0.09w32 and houdini that gives more than +4 for white at depth 1 when the program cannot see the promotion by search because it is hidden by a lot of checks)

Toga1.2.1a gives 1.17 for black at depth 1 but toga is inferior old program
and I believe that all the programs that are at the playing strength of Rybka3 or higher give a big advantage for white when they search to depth 1 inspite of nominal 6 pawns advantage by black.

[D]7k/PP5p/2PP3p/7p/7p/7p/6r1/K7 b - - 0 1
???
it's black's turn to play in your fen, and it's very obvious that black has an unstoppable passed pawn on h3. the eval should already return a big score for black without even a search!
These positions only illustrate what is already well known, that humans don't evaluate positions the same way that computers do but instead reason about the position. I would remind you that in this case it usually involves not just evaluation but look-ahead - you are looking ahead here to know that there will be a checkmate in 2 moves. You didn't process this in your brain statically.

Computers use a version of the same process and it's called null move search. We will never (in my opinion) have computers match the human brain and it may not even be desirable to try - computers are much different and trying to force them to do things the same way is like trying to force a human to run on all fours because it works for Cheeta's.

In your example of course we know that a computers static evaluation function is not very good at sorting out what is most relevant. That can be improved incrementally with a more sophisticated evaluation function for specific cases but a general solution is probably going to make programs a lot weaker - simply because it will slow the program down enormously and only be useful once in a while. Even the position you show is not a problem for any chess program doing 1 or 2 more ply - although it's easily understood that you can compose arbitrarily more and more difficult cases.
I think that the evaluation function of top programs in this position is very good and they evaluate it even after searching to depth 1 when they cannot see the promotion of white(because black has many checks) as advantage for white.
My evaluation of these kinds of positions is not bad, but Crafty doesn't get this one right without a search, because the black rook prevents a pawn race from being evaluated correctly. In this position with two connected passed pawns on 7th, one could evaluate it correctly since a single piece can't stop the promotion unless the king is right there to help. I'd be interested in seeing a program that does just a one ply search and says "white is winning". Because it is really dangerous to evaluate such positions blindly. If black checks and white escapes check, the eval has basically the same position as it had at the root. Now you have to run down the checks to see if after h2 one of the white pawns can promote and somehow prevent the h2 pawn from promoting.

I think this will likely always be a nominal search + evaluation type of position.
Gaviota thinks white is winning from eval. I spent quite a bit of time on these type of positions. It is not straightforward, but not impossible. It took me a long while to nail it. I do not think it adds any elo, but it was a really nice chess challenge.

Miguel
PS: This is an old Gaviota version I have in my office (v0.80). I need to update it :-)

Code: Select all

setboard 7k/PP5p/2PP3p/7p/7p/7p/6r1/K7 b - - 0 1
d
+-----------------+
| . . . . . . . k | [Black]
| P P . . . . . p |
| . . P P . . . p |
| . . . . . . . p |    Castling:
| . . . . . . . p |    ep: -
| . . . . . . . p |
| . . . . . . r . |
| K . . . . . . . |
+-----------------+

score
===> -6.73


analyze
iterative deepening --> start, thread=0
set timer to infinite
       152   1       0.0   -16.04  h2 2.a8=Q+ Kg7
       223   1       0.0    -6.97  Rg1+ 2.Kb2
       230   1:      0.0    -6.97  Rg1+ 2.Kb2
       345   2       0.0      :-(  Rg1+
       597   2       0.0      :-(
      1816   2       0.0      :-(  Rg1+
      1941   2       0.0      :-(
      2649   2       0.0      :-(  Rg1+
      2778   2       0.0      :-(
      3825   2       0.0   -13.77  Rg1+ 2.Kb2 Rg8 3.b8=Q Rxb8+ 4.axb8=Q+
                                   Kg7
      4152   2       0.0   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
      4152   2:      0.0   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
      4241   3       0.0   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
      7201   3:      0.0   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
      7728   4       0.1   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
     10519   4:      0.1   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
     20987   5       0.1   -13.70  Rg8 2.b8=Q Rxb8 3.axb8=Q+ Kg7
     44252   5       0.3      :-)  Rg1+
     52022   5       0.3   -13.63  Rg1+ 2.Ka2 h2 3.b8=Q+ Kg7 4.Qb7+ Kg6
                                   5.c7 h1=R 6.Qe4+ Kf7 7.a8=Q
     55265   5:      0.3   -13.63  Rg1+ 2.Ka2 h2 3.b8=Q+ Kg7 4.Qb7+ Kg6
                                   5.c7 h1=R 6.Qe4+ Kf7 7.a8=Q
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Comparative nodes per second

Post by michiguel »

BubbaTough wrote:
bob wrote: After my experiences (bad) with two connected passers on 6th vs rook, I simply want to let the search discover something good, rather than depending on it to discover something bad. Where the eval gets it wrong in a fast game, I might be stuck with that. In the majority of cases, two connected passers on the 6th are not better than a rook. Unless you tightly constrain it such as "opponent ONLY has a rook, king is at LEAST N squares away, King is not hemmed in and subject to perpetual by the rook, and is not getting mated." I'd prefer to avoid quick searches that push the pawns to the 6th, dropping a pair of bishops or something in the process, and assume it is winning...

Won't begin to argue my approach is the right one, just that it is "my approach" here after getting burned a few times on ICC with that connected on 6th eval term.
I understand your perspective. The two contraints you mention I usually use are something like "opponent ONLY has a rook, king is at LEAST N squares away". In my opinion, in a large majority of these cases, 2 connected on the 6th ARE better than a rook under these constraints, but I must reluctantly admit it is only an opinion. In studies, mating threats are the most common method of constructing perpetuals, and they do come up in practice, but I believe they are the exception rather than the rule. Here is a position I just made up I consider typical:

[D]r7/5ppk/2PP4/8/8/8/5P2/6K1 b - - 7 30

I have not actually checked if it is a win for the pawns, but since there are no mate threats I think its safe to assume it is :). Note, its very very very rare for this to be a win for the rook, so its darn safe to assume the pawns are work AT LEAST a rook.

-Sam
Gaviota thinks white is winning.
Every eval term is a gamble. So, the best option is to place the best bet possible. The absence of an eval term is a gamble too! you are actually betting that the best score for the term is zero. So, not having an eval term is risky too. The problem it is not having the term, the problem is having the wrong score for it.

Miguel

Code: Select all

setboard r7/5ppk/2PP4/8/8/8/5P2/6K1 b - - 7 30
score
===> -3.88
BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 5:18 am

Re: Comparative nodes per second

Post by BubbaTough »

michiguel wrote:
BubbaTough wrote:
bob wrote: After my experiences (bad) with two connected passers on 6th vs rook, I simply want to let the search discover something good, rather than depending on it to discover something bad. Where the eval gets it wrong in a fast game, I might be stuck with that. In the majority of cases, two connected passers on the 6th are not better than a rook. Unless you tightly constrain it such as "opponent ONLY has a rook, king is at LEAST N squares away, King is not hemmed in and subject to perpetual by the rook, and is not getting mated." I'd prefer to avoid quick searches that push the pawns to the 6th, dropping a pair of bishops or something in the process, and assume it is winning...

Won't begin to argue my approach is the right one, just that it is "my approach" here after getting burned a few times on ICC with that connected on 6th eval term.
I understand your perspective. The two contraints you mention I usually use are something like "opponent ONLY has a rook, king is at LEAST N squares away". In my opinion, in a large majority of these cases, 2 connected on the 6th ARE better than a rook under these constraints, but I must reluctantly admit it is only an opinion. In studies, mating threats are the most common method of constructing perpetuals, and they do come up in practice, but I believe they are the exception rather than the rule. Here is a position I just made up I consider typical:

[D]r7/5ppk/2PP4/8/8/8/5P2/6K1 b - - 7 30

I have not actually checked if it is a win for the pawns, but since there are no mate threats I think its safe to assume it is :). Note, its very very very rare for this to be a win for the rook, so its darn safe to assume the pawns are work AT LEAST a rook.

-Sam
Gaviota thinks white is winning.
Every eval term is a gamble. So, the best option is to place the best bet possible. The absence of an eval term is a gamble too! you are actually betting that the best score for the term is zero. So, not having an eval term is risky too. The problem it is not having the term, the problem is having the wrong score for it.

Miguel

Code: Select all

setboard r7/5ppk/2PP4/8/8/8/5P2/6K1 b - - 7 30
score
===> -3.88
Your perspective exactly mirror my own Miguel.

-Sam
diep
Posts: 1822
Joined: Thu Mar 09, 2006 11:54 pm
Location: The Netherlands

Re: Comparative nodes per second

Post by diep »

hi i hacked in textmode some:

How about this and your theories?

[d]8/5p2/2PPpk2/8/8/8/r4P2/6K1 b - - 7 30

I just hacked some - as a chessplayer i'd guess it's a draw. I'd start with Rd2 and chop off the d6 pawn while white promotes.

Yet if i add another pawn:

[d]8/5p2/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

Now it's probably a win for white.

[d]8/5pp1/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

But in the above position i really have no clue whether black can hold it or not... Blindfolded guess would be 1-0 :)
BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 5:18 am

Re: Comparative nodes per second

Post by BubbaTough »

diep wrote:hi i hacked in textmode some:

How about this and your theories?

[d]8/5p2/2PPpk2/8/8/8/r4P2/6K1 b - - 7 30

I just hacked some - as a chessplayer i'd guess it's a draw. I'd start with Rd2 and chop off the d6 pawn while white promotes.

Yet if i add another pawn:

[d]8/5p2/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

Now it's probably a win for white.

[d]8/5pp1/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

But in the above position i really have no clue whether black can hold it or not... Blindfolded guess would be 1-0 :)
Without looking at my code, I am pretty sure the black king is too close to the pawns to trigger my heuristics in these examples. Not sure about Miguel. Are you implying you don't address two pawns on the 6th against a rook in Diep?

-Sam
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Comparative nodes per second

Post by michiguel »

diep wrote:hi i hacked in textmode some:

How about this and your theories?

[d]8/5p2/2PPpk2/8/8/8/r4P2/6K1 b - - 7 30

I just hacked some - as a chessplayer i'd guess it's a draw. I'd start with Rd2 and chop off the d6 pawn while white promotes.

Yet if i add another pawn:

[d]8/5p2/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

Now it's probably a win for white.

[d]8/5pp1/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30

But in the above position i really have no clue whether black can hold it or not... Blindfolded guess would be 1-0 :)
My engine thinks that white is winning in all cases, as you can see from the command "score". It gives the evaluation() score in pawns and 1/256 pawns (the true internal score).

Of course, the first position is a draw because it lacks the knowledge required to understand a R v Q fortress, but that is another issue.

Miguel

Code: Select all

setboard 8/5p2/2PPpk2/8/8/8/r4P2/6K1 b - - 7 30 
d
+-----------------+
| . . . . . . . . | [Black]
| . . . . . p . . |
| . . P P p k . . |
| . . . . . . . . |    Castling: 
| . . . . . . . . |    ep: -
| . . . . . . . . |
| r . . . . P . . |
| . . . . . . K . |
+-----------------+

score
===> -3.31 (-848)

setboard 8/5p2/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30 
d
+-----------------+
| . . . . . . . . | [Black]
| . . . . . p . . |
| . . P P p k . . |
| . . . . . . . . |    Castling: 
| . . . . . . . . |    ep: -
| . . . . . . . . |
| r . . . . P P . |
| . . . . . . K . |
+-----------------+

score
===> -4.40 (-1126)

setboard 8/5pp1/2PPpk2/8/8/8/r4PP1/6K1 b - - 7 30 
d
+-----------------+
| . . . . . . . . | [Black]
| . . . . . p p . |
| . . P P p k . . |
| . . . . . . . . |    Castling: 
| . . . . . . . . |    ep: -
| . . . . . . . . |
| r . . . . P P . |
| . . . . . . K . |
+-----------------+

score
===> -2.99 (-766)

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 5:18 am

Re: Comparative nodes per second

Post by BubbaTough »

michiguel wrote: My engine thinks that white is winning in all cases, as you can see from the command "score". It gives the evaluation() score in pawns and 1/256 pawns (the true internal score).
Interesting. So you statically detect that black cannot approach the white pawns rather than use a simple distance function?

-Sam
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Comparative nodes per second

Post by michiguel »

BubbaTough wrote:
michiguel wrote: My engine thinks that white is winning in all cases, as you can see from the command "score". It gives the evaluation() score in pawns and 1/256 pawns (the true internal score).
Interesting. So you statically detect that black cannot approach the white pawns rather than use a simple distance function?

-Sam
If I remove the e6 pawn, black is winning (see below). I take into account the path of the king, whether is free to go or not. It is not that simple because there are several exceptions.

Miguel

Code: Select all

setboard 8/5pp1/2PP1k2/8/8/8/r4PP1/6K1 b - - 7 30 
d
+-----------------+
| . . . . . . . . | [Black]
| . . . . . p p . |
| . . P P . k . . |
| . . . . . . . . |    Castling: 
| . . . . . . . . |    ep: -
| . . . . . . . . |
| r . . . . P P . |
| . . . . . . K . |
+-----------------+

score
===> 3.30 (845)