LC0 and tablebases

[Werewolf]

So I've got a bit of a weird result here, conditions below:

LC0 11089 (used in CCCC) without TB
LC0 11089 with Syzygy TB

1060 Nvidia.

after 100 games it was +6 -2 =92 to the non-TB version.
TB were enabled by directing to them in the UCI options and they do seem to work.

The non-TB version doesn't seem to use them although Fritz does have syzygy TB in the GUI enabled. Is it possible both were using TB somehow or don't they add much to LC0 yet?

Also, running the above against SF Dev (latest) on a single core with TB I get 58% in favour of SF Dev, which seems a poor result for LC0.

[Werewolf]

I'm virtually certain one of the LC0s is not using tablebases. If someone knows a tricky ending which requires TBs to find the winning move - that would really help to distinguish these two apart in terms of endgame ability.

Also - is my result vs SF Dev on one core (4.2 GHz Intel) about right? I was expecting LC0 to score above 50% on a 1060 GeForce.

[kasinp]

Long time ago I put this position together to help me determine if TB access was working.

Without 6-men tablebases engines (including latest Lc0) report a large negative score for white. With TB the obvious Rxh3 returns 0.00.
If you are testing for 5-men tablebases then simply remove the black pawn h4.

[d]8/4k3/8/2q5/7p/R6p/6P1/7K w - - 0 1

PK

[Werewolf]

Long time ago I put this position together to help me determine if TB access was working.

Without 6-men tablebases engines (including latest Lc0) report a large negative score for white. With TB the obvious Rxh3 returns 0.00.
If you are testing for 5-men tablebases then simply remove the black pawn h4.

[d]8/4k3/8/2q5/7p/R6p/6P1/7K w - - 0 1

PK

Perfect, thank you. Tablebases are working exactly as I hoped.

Unfortunately, that means (on my relatively short test admittedly) TBs are not helping LC0 much. I was expecting about +50 elo which I'd expect to see in a 100 game match. In fact the side with TB lost.

[jp]

You used that test position to check both, did you?

That's an interesting result. If you could somehow do another match starting from equal endgame positions (e.g. 12-piece) that would be interesting. Maybe the +6 were all decided long before the endgame.

[Werewolf]

You used that test position to check both, did you?

Yes.

That's an interesting result. If you could somehow do another match starting from equal endgame positions (e.g. 12-piece) that would be interesting. Maybe the +6 were all decided long before the endgame.

Yes that is possible. I just assumed that over 100 games eventually the side with TB would prevail. Perhaps I need to run a longer match.

If you know of a 12 man starting position set, I could try that also.

[jp]

Did you use an opening book for the 100 games?

[Werewolf]

Did you use an opening book for the 100 games?

Yes, Noomen short to avoid repeated games

[jp]

Noomen short sounds good.

On endgame test sets, http://www.top-5000.nl/testsets.htm
has some endgame test sets. I have no idea about them, but they may be what we're after.

It has:
"Eigenmann
Popular Endgame testset."

and

"End Game
4 end game testsets. Recommendation sunepos.pgn with annotations."


Maybe you can set up a match with two games for each position in one of these testsets.

[Albert Silver]

So I've got a bit of a weird result here, conditions below:

LC0 11089 (used in CCCC) without TB
LC0 11089 with Syzygy TB

1060 Nvidia.

after 100 games it was +6 -2 =92 to the non-TB version.
TB were enabled by directing to them in the UCI options and they do seem to work.

The non-TB version doesn't seem to use them although Fritz does have syzygy TB in the GUI enabled. Is it possible both were using TB somehow or don't they add much to LC0 yet?

Also, running the above against SF Dev (latest) on a single core with TB I get 58% in favour of SF Dev, which seems a poor result for LC0.

If Fritz has TB enabled in the GUI, then whenever they reach a TB ending, all moves are played with TBs by both sides. Unless you also disabled this in the match settings.