bob wrote:For R=3 (first run) this appears to drop 24 Elo after 3,000 games. Will let it run to completion for accuracy. Also have R=4 and R=5 queued up since the paper uses R=4...
One question about your implementation. When you finish the search and store in the transposition table, do you use the original depth or the reduced one?
Reduced one, if you mean the reduced-depth search that follows the null-move fail high. Don't see how you could store anything else and not get gross search inconsistencies...
All I did was reduce the actual "depth" variable at the null-move test code in search.c... reduced-depth is used below that point for everything.
The reason for using original depth would be the same as with normal null move: to be able to use the result if you visit the node again with the same depth. Anyway, I'm not sure the idea is sound. After the null move has failed high, the reduced search acts like a verification search. If it fails high, you return like in verified null move but if it fails low (a zugzwang position), it leaves you with a reduced depth search, whereas in verified null move you would continue searching at normal depth.
The problem is the "depth" value which gets passed thru the recursive search calls. It would be incredibly bad to use "depth" to store, but pass depth-4 to the next ply. The result you are storing is not a "depth" search, but a depth-4 search. If you store depth-4, you should get a hit the next around at the same depth...
I don't follow. When you do the depth-4 search is because you have done a null move search before that failed high, and this normally acts as a replacement of a normal depth search, except that in this case you also did the reduced search that acts as a verification.
I also don't get how storing depth-4 would help you to get a hit next time. For example, if you initially have depth 9, null move fails high, you reduce the depth, fail high again on reduced search and store with depth 5, you won't be able to use the result when you come again to this node with depth 9 and you will do the same work (null move + reduced depth search).
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
I am not certain. A month or two back, when working on 23.3, I experimented with the cutoff point for NM. I used to do normal null-move until there was only a queen left, then turn it off in low plies until I got down to a rook and turned it off completely. My best results were doing full null-move until I reached pawn-only endings where it is turned off. I believe doing it in pawn endings was a -2 or -3 elo change. Not very much. Improves the depth significantly, which helps all the time. But falls into zugzwang positions with some regularity which offsets that depth gain by a little. But it is not a big win. Allowing nulls anywhere is only 1-2 Elo worse than not allowing two consecutive nulls, also, so that was a surprise to me as well. But a fact, at least for Crafty. I still do not allow consecutive nulls as 2 Elo is still 2 Elo. Ditto for pawn endings and null-move
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
I am not certain. A month or two back, when working on 23.3, I experimented with the cutoff point for NM. I used to do normal null-move until there was only a queen left, then turn it off in low plies until I got down to a rook and turned it off completely. My best results were doing full null-move until I reached pawn-only endings where it is turned off. I believe doing it in pawn endings was a -2 or -3 elo change. Not very much. Improves the depth significantly, which helps all the time. But falls into zugzwang positions with some regularity which offsets that depth gain by a little. But it is not a big win. Allowing nulls anywhere is only 1-2 Elo worse than not allowing two consecutive nulls, also, so that was a surprise to me as well. But a fact, at least for Crafty. I still do not allow consecutive nulls as 2 Elo is still 2 Elo. Ditto for pawn endings and null-move
When you say not allowing consecutive null moves do you mean first for white, then for black?
Komodo never does null move unless the static evaluation is >= beta so it's not possible (for Komodo) to do 2 consecutive null moves - unless you mean 2 null moves in a row for the SAME SIDE.
I have tried to do null moves when the score is close to beta similar to how stockfish does it, but this always hurts the program. I'm speculating that since stockfish tries to take advantage of the threat move information it is a good thing for stockfish. Komodo does not try to make use of the threat move.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
I am not certain. A month or two back, when working on 23.3, I experimented with the cutoff point for NM. I used to do normal null-move until there was only a queen left, then turn it off in low plies until I got down to a rook and turned it off completely. My best results were doing full null-move until I reached pawn-only endings where it is turned off. I believe doing it in pawn endings was a -2 or -3 elo change. Not very much. Improves the depth significantly, which helps all the time. But falls into zugzwang positions with some regularity which offsets that depth gain by a little. But it is not a big win. Allowing nulls anywhere is only 1-2 Elo worse than not allowing two consecutive nulls, also, so that was a surprise to me as well. But a fact, at least for Crafty. I still do not allow consecutive nulls as 2 Elo is still 2 Elo. Ditto for pawn endings and null-move
When you say not allowing consecutive null moves do you mean first for white, then for black?
Komodo never does null move unless the static evaluation is >= beta so it's not possible (for Komodo) to do 2 consecutive null moves - unless you mean 2 null moves in a row for the SAME SIDE.
I have tried to do null moves when the score is close to beta similar to how stockfish does it, but this always hurts the program. I'm speculating that since stockfish tries to take advantage of the threat move information it is a good thing for stockfish. Komodo does not try to make use of the threat move.
I mean two consecutive plies. I never had any luck with the static eval > beta, but then I use the hash table trick to avoid null moves that won't fail high according to the hash table, so perhaps those two ideas overlap and that's why it didn't help me.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
I am not certain. A month or two back, when working on 23.3, I experimented with the cutoff point for NM. I used to do normal null-move until there was only a queen left, then turn it off in low plies until I got down to a rook and turned it off completely. My best results were doing full null-move until I reached pawn-only endings where it is turned off. I believe doing it in pawn endings was a -2 or -3 elo change. Not very much. Improves the depth significantly, which helps all the time. But falls into zugzwang positions with some regularity which offsets that depth gain by a little. But it is not a big win. Allowing nulls anywhere is only 1-2 Elo worse than not allowing two consecutive nulls, also, so that was a surprise to me as well. But a fact, at least for Crafty. I still do not allow consecutive nulls as 2 Elo is still 2 Elo. Ditto for pawn endings and null-move
When you say not allowing consecutive null moves do you mean first for white, then for black?
Komodo never does null move unless the static evaluation is >= beta so it's not possible (for Komodo) to do 2 consecutive null moves - unless you mean 2 null moves in a row for the SAME SIDE.
I have tried to do null moves when the score is close to beta similar to how stockfish does it, but this always hurts the program. I'm speculating that since stockfish tries to take advantage of the threat move information it is a good thing for stockfish. Komodo does not try to make use of the threat move.
I mean two consecutive plies. I never had any luck with the static eval > beta, but then I use the hash table trick to avoid null moves that won't fail high according to the hash table, so perhaps those two ideas overlap and that's why it didn't help me.
In my case, I try null move at any ply where:
(1) hash table does not say it is worthless;
(2) not in check
(3) move at previous ply was not a null-move
(4) side to move has at least one non-pawn piece.
(5) remaining depth > 1
When testing search changes (as opposed to eval changes) should we use EGTBs or should we turn them off?
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
Are you saying that you lose 100 ELO if you use R=5 in king and pawn endings? And you lose about 11 if you use R=4 in KP endings?
Komodo never does null move when the game stage is zero - which means that only knights and pawns can be on the board. But I don't remember the data on this - how much it's worth.
Don
Not just "null-move" but the version discussed in the paper mentioned at the beginning of this thread. That is, if null-move fails high, do a reduced depth search rather than returning the fail high condition. Doing that, R=5 cost me 100 elo in a test, where R=4 cost me 11. I found no case where this idea was as good as normal null-move search. Best I got was -6 when tweaking the two depths independently (null-move R value vs the reduction R value if NM fails high). Paper said 4 for both, I tried a mix of several values.
Ok, now I understand.
Ignoring this algorithm for the moment, do you have any sense of how much not doing null move in k+p endings is worth?
An earlier (non-release) version of Komodo had the rule to never do null move in simple endings, where simple is defined as at least 2 bishops (knights are not considered in game phase in komodo.) So if there was only 1 bishop on the board (and perhaps knights and pawns) we would not do null move.
It turned out this was a regression - it had somehow slipped through the cracks and later we noticed that a current version was weaker than an earlier reference version - it took us a week to figure out that this rule was the culprit. It was one of the last things we checked because it seemed to us to be not very important either way - but we were wrong. This was like a 5 or 10 ELO error.
I am not certain. A month or two back, when working on 23.3, I experimented with the cutoff point for NM. I used to do normal null-move until there was only a queen left, then turn it off in low plies until I got down to a rook and turned it off completely. My best results were doing full null-move until I reached pawn-only endings where it is turned off. I believe doing it in pawn endings was a -2 or -3 elo change. Not very much. Improves the depth significantly, which helps all the time. But falls into zugzwang positions with some regularity which offsets that depth gain by a little. But it is not a big win. Allowing nulls anywhere is only 1-2 Elo worse than not allowing two consecutive nulls, also, so that was a surprise to me as well. But a fact, at least for Crafty. I still do not allow consecutive nulls as 2 Elo is still 2 Elo. Ditto for pawn endings and null-move
When you say not allowing consecutive null moves do you mean first for white, then for black?
Komodo never does null move unless the static evaluation is >= beta so it's not possible (for Komodo) to do 2 consecutive null moves - unless you mean 2 null moves in a row for the SAME SIDE.
I have tried to do null moves when the score is close to beta similar to how stockfish does it, but this always hurts the program. I'm speculating that since stockfish tries to take advantage of the threat move information it is a good thing for stockfish. Komodo does not try to make use of the threat move.
I mean two consecutive plies. I never had any luck with the static eval > beta, but then I use the hash table trick to avoid null moves that won't fail high according to the hash table, so perhaps those two ideas overlap and that's why it didn't help me.
In my case, I try null move at any ply where:
(1) hash table does not say it is worthless;
(2) not in check
(3) move at previous ply was not a null-move
(4) side to move has at least one non-pawn piece.
(5) remaining depth > 1
When testing search changes (as opposed to eval changes) should we use EGTBs or should we turn them off?
thanks,
Alvaro
I don't use 'em in my cluster testing, too big a pain to copy them to each individual node and then remove them after a match is done... I doubt it makes any difference.
bob wrote:one more bit of info. Using this approach, and allowing null-move search with pawn-only endings is worth about -100 Elo, so this is not much of a verification good news test.
Just realized I reported that wrong. Using R=4 and doing null-move in KP endings is a -11 Elo deal in Crafty. Going to R=5 drops it to -100, again doing this in KP endings (my normal null-move only triggers if the side on move has a single piece or more...)
An idea I had was to allow null move if the side on move has no pieces if it is significantly down in material (I would guess more than a piece). The idea is that if you're lost anyway, it probably doesn't take zugzwang to beat you.
Once the current batch of tests are done, I'll give that a check. In pawn-only endgames or not???
Since you already do null moves when the side on move has a piece, try also doing null moves when the side on move has no pieces and is significantly down in material.
OK, two more questions:
(1) do original null-move or this new idea?
(2) what is "significantly down in material"? piece? rook? queen?
1. Do your standard null move and in addition do a standard null move when the side to move has no pieces and is down significantly in material.
2. I am guessing significantly means more than a piece, but you could try it with a larger number to see if it works for you at all and tune it later.
Once again, nothing helped. doing it with no pieces and down a rook still hurts elo, surprisingly. You can go low enough that the loss is zero, but only because the alpha bound becomes so low that the null-move is never or almost never done...
