Page 1 of 3
cutechess-cli: not restarting an engine because of tt
Posted: Sat Jul 22, 2017 11:59 pm
by flok
Hi,
It looks like my program plays way worse when it is not restarted. Most likely this is due to a tt change I did a couple of weeks ago.
I would like to test this of course. Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.
So I wonder: does anyone know a way around this? I've got 4 versions to test.
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 12:18 am
by AndrewGrant
I reset History, Pawn Table, and Trans Table between searches. I leave the killer moves in tact, as well as counter moves.
If you actually notice a big difference due to not restarting, I imagine you have a bug of sorts.
Are you using a different set of zorbist keys between games?
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 12:20 am
by flok
No same zobrist keys every time.
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 12:26 am
by AndrewGrant
I'm sure if you try zeroing out all of your tables between games you will find no losses.
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 12:44 am
by Ras
flok wrote:It looks like my program plays way worse when it is not restarted.
I just reset TT, history and PV if it's a new game or a different position.
That's a bit tricky with UCI because you have to work around a stateless protocol while maintaining state, but still possible:
- after every move the engine makes, store the position hash and the ply counter.
- if you get a "go", first check that the ply counter is exactly one more than it was after the engine made its last move.
- if so, take back the last ply. Check if the position hash is identical to what it was after the engine made its last move.
- if you have a match, you can keep hash tables, history and PV for move sorting. Otherwise, reset everything.
Then redo the ply you undid and launch the search.
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 12:48 am
by flok
This is my replacement scheme:
Code: Select all
int useSubIndex = -1, worstDepth = 99999;
for(int i=0; i<N_TE_PER_HASH_GROUP; i++)
{
if (e[i].hash == hash) {
useSubIndex = i;
break;
}
if (e[i].age != age || e[i].flags == NOTVALID || e[i].depth < worstDepth) {
worstDepth = e[i].depth;
useSubIndex = i;
}
}
// here the bin with index "useSubIndex" is replaced
"age" is increased between moves and notvalid is set when the program is started (and the tt initialized)
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 1:00 am
by flok
Ras wrote:flok wrote:It looks like my program plays way worse when it is not restarted.
I just reset TT, history and PV if it's a new game or a different position.
[/quote]
That should be easy to test. Will do so!
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 4:46 am
by lucasart
flok wrote:Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.
Cutechess CLI does NOT restart engines, if you use it correctly...
It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 7:31 am
by flok
lucasart wrote:flok wrote:Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.
Cutechess CLI does NOT restart engines, if you use it correctly...
It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".
Can you show me a correct example?
I usually do this:
Code: Select all
cutechess-cli \
-engine cmd=/usr/local/bin/dorpsgek proto=xboard name=dorpsgek \
-engine cmd=./E_trunk.sh proto=uci name=trunk \
-engine cmd=./E_trunk-age.sh proto=uci name=trunk-age \
-engine cmd=./E_4171.sh proto=uci name=4171 \
-engine cmd=./E_4173.sh proto=uci name=4173 \
-engine cmd=./E_4182.sh proto=uci name=4182 \
-engine cmd=./E_4183.sh proto=uci name=4183 \
-concurrency $CONC \
-each dir=$DIR tc=40/10+0.25 -rounds 500 -recover -repeat -tournament gauntlet -pgnout out/$HOST.pgn -site "$HOST" | tee log/$HOST.log
Re: cutechess-cli: not restarting an engine because of tt
Posted: Sun Jul 23, 2017 8:39 am
by hgm
Ras wrote:I just reset TT, history and PV if it's a new game or a different position.
That's a bit tricky with UCI because you have to work around a stateless protocol while maintaining state, but still possible:
- after every move the engine makes, store the position hash and the ply counter.
- if you get a "go", first check that the ply counter is exactly one more than it was after the engine made its last move.
- if so, take back the last ply. Check if the position hash is identical to what it was after the engine made its last move.
- if you have a match, you can keep hash tables, history and PV for move sorting. Otherwise, reset everything.
Then redo the ply you undid and launch the search.
This would only work when the engine plays itself, right? Othersise you would have to take back two ply. And it doesn't work at all when the engine ponders.
Why would you want to clear the TT on a new game anyway? Empty entries are not more helpful than entires from another game. As long as you are still in the same variant, that is. Aging would make the entries from the previous search be considered empty pretty quickly anyway.
A simpler method would be to remember the last two position-move commands, and see how much they have in common. If the sum of the length of the non-common parts is below a certain limit (like 20 characters), you can assume the same game. That would prevent clearing of tables on a ponder miss, or a takeback.