cutechess-cli: not restarting an engine because of tt

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
flok

cutechess-cli: not restarting an engine because of tt

Post by flok » Sat Jul 22, 2017 9:59 pm

Hi,

It looks like my program plays way worse when it is not restarted. Most likely this is due to a tt change I did a couple of weeks ago.
I would like to test this of course. Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.

So I wonder: does anyone know a way around this? I've got 4 versions to test.

AndrewGrant
Posts: 472
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: cutechess-cli: not restarting an engine because of tt

Post by AndrewGrant » Sat Jul 22, 2017 10:18 pm

I reset History, Pawn Table, and Trans Table between searches. I leave the killer moves in tact, as well as counter moves.

If you actually notice a big difference due to not restarting, I imagine you have a bug of sorts.

Are you using a different set of zorbist keys between games?

flok

Re: cutechess-cli: not restarting an engine because of tt

Post by flok » Sat Jul 22, 2017 10:20 pm

No same zobrist keys every time.

AndrewGrant
Posts: 472
Joined: Tue Apr 19, 2016 4:08 am
Location: U.S.A
Full name: Andrew Grant
Contact:

Re: cutechess-cli: not restarting an engine because of tt

Post by AndrewGrant » Sat Jul 22, 2017 10:26 pm

I'm sure if you try zeroing out all of your tables between games you will find no losses.

Ras
Posts: 1140
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: cutechess-cli: not restarting an engine because of tt

Post by Ras » Sat Jul 22, 2017 10:44 pm

flok wrote:It looks like my program plays way worse when it is not restarted.
I just reset TT, history and PV if it's a new game or a different position.

That's a bit tricky with UCI because you have to work around a stateless protocol while maintaining state, but still possible:

- after every move the engine makes, store the position hash and the ply counter.
- if you get a "go", first check that the ply counter is exactly one more than it was after the engine made its last move.
- if so, take back the last ply. Check if the position hash is identical to what it was after the engine made its last move.
- if you have a match, you can keep hash tables, history and PV for move sorting. Otherwise, reset everything.

Then redo the ply you undid and launch the search.

flok

Re: cutechess-cli: not restarting an engine because of tt

Post by flok » Sat Jul 22, 2017 10:48 pm

This is my replacement scheme:

Code: Select all

        int useSubIndex = -1, worstDepth = 99999;

        for&#40;int i=0; i<N_TE_PER_HASH_GROUP; i++)
        &#123;
                if &#40;e&#91;i&#93;.hash == hash&#41; &#123;
                        useSubIndex = i;
                        break;
                &#125;

                if &#40;e&#91;i&#93;.age != age || e&#91;i&#93;.flags == NOTVALID || e&#91;i&#93;.depth < worstDepth&#41; &#123;
                        worstDepth = e&#91;i&#93;.depth;
                        useSubIndex = i;
                &#125;
        &#125;

// here the bin with index "useSubIndex" is replaced
"age" is increased between moves and notvalid is set when the program is started (and the tt initialized)

flok

Re: cutechess-cli: not restarting an engine because of tt

Post by flok » Sat Jul 22, 2017 11:00 pm

Ras wrote:
flok wrote:It looks like my program plays way worse when it is not restarted.
I just reset TT, history and PV if it's a new game or a different position.
[/quote]

That should be easy to test. Will do so!

User avatar
lucasart
Posts: 3037
Joined: Mon May 31, 2010 11:29 am
Full name: lucasart
Contact:

Re: cutechess-cli: not restarting an engine because of tt

Post by lucasart » Sun Jul 23, 2017 2:46 am

flok wrote:Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.
Cutechess CLI does NOT restart engines, if you use it correctly...

It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.

flok

Re: cutechess-cli: not restarting an engine because of tt

Post by flok » Sun Jul 23, 2017 5:31 am

lucasart wrote:
flok wrote:Usually I use cutechess-cli for testing (because it is command line so I can easily start it on all nodes of the test cluster) but using strace I saw that it restarts every program after an iteration which gives me a clean transposition table every time.
Cutechess CLI does NOT restart engines, if you use it correctly...

It just sends "ucinewgame" between games. If you want to clear HT between games, that's where you should do it: upon receipt of "ucinewgame".

Can you show me a correct example?

I usually do this:

Code: Select all

cutechess-cli \
        -engine cmd=/usr/local/bin/dorpsgek proto=xboard name=dorpsgek \
        -engine cmd=./E_trunk.sh proto=uci name=trunk \
        -engine cmd=./E_trunk-age.sh proto=uci name=trunk-age \
        -engine cmd=./E_4171.sh proto=uci name=4171 \
        -engine cmd=./E_4173.sh proto=uci name=4173 \
        -engine cmd=./E_4182.sh proto=uci name=4182 \
        -engine cmd=./E_4183.sh proto=uci name=4183 \
        -concurrency $CONC \
        -each dir=$DIR tc=40/10+0.25 -rounds 500 -recover -repeat -tournament gauntlet -pgnout out/$HOST.pgn -site "$HOST" | tee log/$HOST.log

User avatar
hgm
Posts: 23610
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: cutechess-cli: not restarting an engine because of tt

Post by hgm » Sun Jul 23, 2017 6:39 am

Ras wrote:I just reset TT, history and PV if it's a new game or a different position.

That's a bit tricky with UCI because you have to work around a stateless protocol while maintaining state, but still possible:

- after every move the engine makes, store the position hash and the ply counter.
- if you get a "go", first check that the ply counter is exactly one more than it was after the engine made its last move.
- if so, take back the last ply. Check if the position hash is identical to what it was after the engine made its last move.
- if you have a match, you can keep hash tables, history and PV for move sorting. Otherwise, reset everything.

Then redo the ply you undid and launch the search.
This would only work when the engine plays itself, right? Othersise you would have to take back two ply. And it doesn't work at all when the engine ponders.

Why would you want to clear the TT on a new game anyway? Empty entries are not more helpful than entires from another game. As long as you are still in the same variant, that is. Aging would make the entries from the previous search be considered empty pretty quickly anyway.

A simpler method would be to remember the last two position-move commands, and see how much they have in common. If the sum of the length of the non-common parts is below a certain limit (like 20 characters), you can assume the same game. That would prevent clearing of tables on a ponder miss, or a takeback.

Post Reply