Devlog of Leorik

[Mike Sherwin]

It might be a TT bug, storing the wrong score.

It could be that my downloaded copy is corrupt. If it is that would be a first.

Nope ruled that out. I downloaded Leorik-2.4 (again) and it has the exact same behavior. There is definitely a serious bug and it looks like a TT bug. In this position I'm looking at.
5k2/R7/4K1p1/5PPp/7P/8/2r5/8 b - - 0 86
Both Leorik-2.4 and Leorik.KiSS evaluate the position as 0.00 and gives no pv. It just looks like a TT bug.

Leorik-2.4:
1 00:00 31 31k -4.82 Rc2-e2+
2 00:00 61 61k -4.72 Rc2-e2+ Ke6-f6
3 00:00 206 206k -4.99 Rc2-e2+ Ke6-f6 g6xf5
4 00:00 561 561k 0.00 Kf8-g8
5 00:00 617 617k 0.00 Kf8-g8
6 00:00 673 673k 0.00 Kf8-g8
7 00:00 773 773k 0.00 Kf8-g8
8 00:00 866 866k 0.00 Kf8-g8
9 00:00 1k 1,316k 0.00 Kf8-g8
10 00:00 2k 1,710k 0.00 Kf8-g8
11 00:00 2k 2,416k 0.00 Kf8-g8
12 00:00 3k 3,005k 0.00 Kf8-g8
13 00:00 4k 3,970k 0.00 Kf8-g8
14 00:00 5k 4,741k 0.00 Kf8-g8
15 00:00 6k 6,226k 0.00 Kf8-g8
16 00:00 8k 7,538k 0.00 Kf8-g8
17 00:00 11k 5,297k 0.00 Kf8-g8
18 00:00 13k 6,627k 0.00 Kf8-g8
19 00:00 18k 6,030k 0.00 Kf8-g8
20 00:00 22k 5,444k 0.00 Kf8-g8
21 00:00 28k 5,522k 0.00 Kf8-g8
22 00:00 32k 4,565k 0.00 Kf8-g8
23 00:00 40k 4,429k 0.00 Kf8-g8
24 00:00 46k 4,199k 0.00 Kf8-g8
25 00:00 60k 4,276k 0.00 Kf8-g8
26 00:00 70k 4,116k 0.00 Kf8-g8
27 00:00 91k 4,121k 0.00 Kf8-g8
28 00:00 116k 4,155k 0.00 Kf8-g8
29 00:00 137k 4,024k 0.00 Kf8-g8
30 00:00 159k 3,796k 0.00 Kf8-g8
31 00:00 197k 3,708k 0.00 Kf8-g8
32 00:00 230k 3,546k 0.00 Kf8-g8
33 00:00 299k 3,436k 0.00 Kf8-g8
34 00:00 360k 3,361k 0.00 Kf8-g8
35 00:00 538k 3,517k 0.00 Kf8-g8
36 00:00 717k 3,515k 0.00 Kf8-g8
37 00:00 1,281k 3,702k 0.00 Kf8-g8
38 00:00 1,805k 3,729k 0.00 Kf8-g8
39 00:00 2,759k 3,822k 0.00 Kf8-g8
40 00:01 4,330k 3,866k 0.00 Kf8-g8
41 00:02 8,165k 3,952k 0.00 Kf8-g8
42 00:03 13,946k 4,020k 0.00 Kf8-g8
43 00:05 22,642k 4,044k 0.00 Kf8-g8
44 00:08 32,804k 4,065k 0.00 Kf8-g8
45 00:12 51,826k 4,051k 0.00 Kf8-g8
46 00:21 88,022k 4,085k 0.00 Kf8-g8
47 00:33 137,341k 4,087k 0.00 Kf8-g8
48 00:48 197,510k 4,113k 0.00 Kf8-g8
49 01:13 300,689k 4,116k 0.00 Kf8-g8
50 01:46 444,706k 4,160k 0.00 Kf8-g8

[dangi12012]

I added sparse pext to C++ for fun.
https://github.com/Gigantua/Chess_Moveg ... Sparse.hpp

Unfortunately it produces incorrect results? Can you take a look - maybe it corresponds to above issues.
Or I made translation errors.

Code: Select all

Occupy: 510                                                                                                                                           .XXXXXXX                                                                                                                                              X.......                                                                                                                                              ........                                                                                                                                              ........                                                                                                                                              ........
........
........
........

Solution:
X.X.....
XXX.....
.X.X....
.X..X...
.X...X..
.X....X.
.X.....X
.X......

Error:
X.X.....
X.X.....
X..X....
X...X...
X....X..
X.....X.
X......X
X.......

in Main.cpp #define PextSparse_ -> (1)
Performance preview: ~around fancy magic

[Mike Sherwin]

Sorry for the drama. It is just I've been working toward getting KiSS competitive for roughly 16 years. And when an unexplained negative change in fortune occured it made me quite upset.

Okay, I've had a little difficulty modifying the single dimensional array math since I'm not used to having to do that. But finally I was able to get it right. I hope.

Code: Select all

using System.Runtime.CompilerServices;
using static Leorik.Core.Bitboard;

namespace Leorik.Core.Slider
{
    //This class implements Mike Sherwin's Kindergarten Super SISSY Bitboards where SISSY stands for Split Index Sub Set Yielding
    //...henceforth we shall abreviate it as KiSS <3
    public static class KiSS
    {
        static readonly ulong[] dSubset = new ulong[64 * 64];
        static readonly ulong[] aSubset = new ulong[64 * 64];
        static readonly ulong[] vSubset = new ulong[64 * 64];
        static readonly ulong[] hSubset = new ulong[64 * 64];

        const int FILE_A = 0;
        const int FILE_H = 7;
        const int RANK_1 = 0;
        const int RANK_8 = 7;
        const ulong FILE_A2_A7 = 0x0001010101010100;
        const ulong DIAGONAL_C2_H7 = 0x0080402010080400;

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static ulong BishopAttacks(ulong occupation, int square)
        {
            ulong dIndex = (occupation & Blocker.DiagonalMask[square]) * FILE_A2_A7 >> 57;
            ulong aIndex = (occupation & Blocker.AntiDiagonalMask[square]) * FILE_A2_A7 >> 57;
            return dSubset[square * 64 + (int)dIndex] | aSubset[square * 64 + (int)aIndex];
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static ulong RookAttacks(ulong occupation, int square)
        {
            ulong hIndex = occupation >> (square & 0b111000 | 1) & 63;
            ulong vIndex = (occupation >> (square & 0b000111) & FILE_A2_A7) * DIAGONAL_C2_H7 >> 58;
            return hSubset[square * 64 + (int)hIndex] | vSubset[square * 64 + (int)vIndex];
        }

        //Table initialization

        static KiSS()
        {
            //Init Subsets
            for (int square = 0; square < 64; square++)
                for (int index = 0; index < 64; index++)
                {
                    dSubset[square * 64 + index] = GetDiagonalSubset(square, index);
                    aSubset[square * 64 + index] = GetAntiDiagonalSubset(square, index);
                    hSubset[square * 64 + index] = GetHorizontalSubset(square, index);
                    vSubset[square * 64 + index] = GetVerticalSubset(square, index);
                }
        }

        static ulong GetDiagonalSubset(int square, int index)
        {
            ulong blockers = GetBlockers(square, index);
            ulong result = 0;
            for (int sq = square; IsFree(blockers, sq) && File(sq) < FILE_H && Rank(sq) < RANK_8; sq += 9)
                result |= 1UL << sq + 9;
            for (int sq = square; IsFree(blockers, sq) && File(sq) > FILE_A && Rank(sq) > RANK_1; sq -= 9)
                result |= 1UL << sq - 9;
            return result;
        }

        static ulong GetAntiDiagonalSubset(int square, int index)
        {
            ulong blockers = GetBlockers(square, index);
            ulong result = 0;
            for (int sq = square; IsFree(blockers, sq) && File(sq) > FILE_A && Rank(sq) < RANK_8; sq += 7)
                result |= 1UL << sq + 7;
            for (int sq = square; IsFree(blockers, sq) && File(sq) < FILE_H && Rank(sq) > RANK_1; sq -= 7)
                result |= 1UL << sq - 7;
            return result;
        }

        static ulong GetHorizontalSubset(int square, int index)
        {
            ulong blockers = GetBlockers(square, index);
            ulong result = 0;
            for (int sq = square; IsFree(blockers, sq) && File(sq) < FILE_H; sq++)
                result |= 1UL << sq + 1;
            for (int sq = square; IsFree(blockers, sq) && File(sq) > FILE_A; sq--)
                result |= 1UL << sq - 1;
            return result;
        }

        static ulong GetVerticalSubset(int square, int index)
        {
            ulong blockers = GetVerticalBlockers(square, index);
            ulong result = 0;
            for (int sq = square; IsVerticalFree(blockers, sq) && Rank(sq) < RANK_8; sq += 8)
                result |= 1UL << sq + 8;
            for (int sq = square; IsVerticalFree(blockers, sq) && Rank(sq) > RANK_1; sq -= 8)
                result |= 1UL << sq - 8;
            return result;
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        static bool IsFree(ulong mask, int square) => (mask & 1UL << File(square)) == 0;

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        static ulong GetBlockers(int square, int index) => (ulong)(index << 1) & ~(1UL << File(square));

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        static bool IsVerticalFree(ulong mask, int square) => (mask & 1UL << square - File(square)) == 0;

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        static ulong GetVerticalBlockers(int square, int index)
        {
            //Place the 6 'index' bits FEDCBA like this, leave standing square ZERO 
            //  - - - - - - - - 
            //  A - - - - - - - 
            //  B - - - - - - - 
            //  C - - - - - - - 
            //  D - - - - - - - 
            //  E - - - - - - - 
            //  F - - - - - - - 
            //  - - - - - - - - 
            ulong blockers = 0;
            for (int i = 0, shift = 48; i < 6; i++, shift -= 8)
                if ((index & 1 << i) > 0) //index bit is set?
                    if (shift != square - File(square)) //don't block standing square
                        blockers |= 1UL << shift;
            return blockers;
        }

        //~~~ DEBUG ~~~

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static ulong _BishopAttacks(ulong occupation, int square)
        {
            ulong dIndex = (occupation & Blocker.DiagonalMask[square]) * FILE_A2_A7 >> 57;
            ulong aIndex = (occupation & Blocker.AntiDiagonalMask[square]) * FILE_A2_A7 >> 57;
            return GetDiagonalSubset(square, (int)dIndex) | GetAntiDiagonalSubset(square, (int)aIndex);
        }

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static ulong _RookAttacks(ulong occupation, int square)
        {
            ulong hIndex = occupation >> (square & 0b111000 | 1) & 63;
            ulong vIndex = (occupation >> (square & 0b000111) & FILE_A2_A7) * DIAGONAL_C2_H7 >> 58;
            return GetHorizontalSubset(square, (int)hIndex) | GetVerticalSubset(square, (int)vIndex);
        }
    }
}

My compiles still use net6 because I could not get net8 to work.
I have some numbers from analyze mode for the original starting position.

• ply Classic KiSS one KiSS four
  20 4396k 4421 4448
  21 4432 4440 4496
  22 4419 4434 4482
  23 4423 4424 4488
  24 4425 4440 4493
  25 4452 4455 4527

Once the bugs are fixed I'll run some test.

Though it probably does not make much of a difference.

[lithander]

Sorry for the drama. It is just I've been working toward getting KiSS competitive for roughly 16 years. And when an unexplained negative change in fortune occured it made me quite upset.

I totally understand your frustrations. I mirror them to some extend, even. I just don't want to be accused of treason when all I did was implement that algorithm in Leorik. I optimized it even to the best of my abilities for C# to give it a fair chance.

Though it probably does not make much of a difference.

The four array version is nicer to read. If it's a little faster, too, I'll push it to the branch next update.

I added sparse pext to C++ for fun.
https://github.com/Gigantua/Chess_Moveg ... Sparse.hpp

Unfortunately it produces incorrect results? Can you take a look - maybe it corresponds to above issues.
Or I made translation errors.

I didn't spot anything at a cursory glance. An idea would be that Chess_Lookup::GetSliderD1Cond() might return a different result than Classic.DiagonalSubset()?

I'll have a look later but currently my top priority is finding the bug causing these suicidal tendencies in Leorik's play that Mike has pointed out.

I can reproduce the wrong moves to some extend. But I can't understand how my reproduction would happen in an actual game. If you initialize the the engine with "position startpos move1 move2 move3" then the position achieved after playing each move is stored in the TT with a score of zero. The idea is that we've seen the position before in the game and repeating it would just lead towards a three-fold-repetition.
But if you don't clear the TT after setting up a position with "position startpos move1 move2 move3" (which I don't) and you set one up with "position startpos move1 move2" afterwards excluding move3 then the search will still find move3 in the TT with a score of 0. But this time it's not a repetition and the score is plain wrong.

In this position [fen]8/8/4kp2/2K3p1/4Pp1p/5P1P/R5P1/3r4 w - - 20 60[/fen] a2a7 is a bad move.

Code: Select all

Leorik 2.4 Net8 Classic
position startpos moves e2e4 c7c5 b1c3 e7e6 g1f3 b8c6 d2d4 c5d4 f3d4 d8c7 c1e3 a7a6 f1d3 g8f6 e1g1 f8d6 g1h1 h7h5 d4c6 d7c6 h2h3 f6g4 d1d2 g4e3 d2e3 d6f4 e3c5 b7b6 c5a3 f4d6 a3b3 e8g8 a2a4 h5h4 c3e2 e6e5 d3c4 b6b5 a4b5 c6b5 c4d5 c8b7 c2c4 a8b8 b3d3 b5c4 d5c4 d6c5 e2c3 c5d4 c4a6 b7a6 a1a6 b8b2 c3d5 c7c2 d5e7 g8h7 d3f3 c2e2 f3f5 h7h8 f1a1 b2b1 a1b1 e2a6 f2f3 a6h6 b1d1 d4b2 d1d7 h6c1 h1h2 c1f4 f5f4 e5f4 d7b7 b2d4 e7c6 d4f2 b7b2 f8c8 b2f2 c8c6 f2a2 g7g6 h2g1 h8g7 g1f2 g7f6 f2e2 f6e6 e2d3 c6d6 d3e2 f7f6 a2b2 g6g5 b2c2 d6b6 e2d3 b6b1 c2a2 b1b3 d3c4 b3b6 c4c5 b6d6 a2b2 d6d1 c5c4 d1c1 c4d4 c1c6 b2a2 c6d6 d4c5 d6d1 a2a7
go depth 20
info depth 1 score cp 61 nodes 24 nps 8000 time 3 pv e6e5
info depth 2 score cp 65 nodes 166 nps 16600 time 10 pv d1d2 a7a6
info depth 3 score cp 97 nodes 258 nps 23454 time 11 pv d1d2 a7a6 e6e5
info depth 4 score cp 181 nodes 779 nps 70818 time 11 pv d1d2 c5c6 d2c2 c6b6
info depth 5 score cp 201 nodes 1476 nps 134181 time 11 pv d1d2 c5b6 d2b2 b6a6 b2g2
info depth 6 score cp 193 nodes 2755 nps 229583 time 12 pv d1d2 c5c6 d2g2 a7c7 g2c2 c6b6
info depth 7 score cp 205 nodes 6406 nps 492769 time 13 pv d1d2 a7a8 d2g2 a8e8 e6f7 e8d8 f7e6
info depth 8 score cp 181 nodes 9239 nps 659928 time 14 pv d1d2 a7a8 d2g2 a8e8 e6f7 e8d8 f7e6 c5c6
info depth 9 score cp 199 nodes 11840 nps 789333 time 15 pv d1d2 a7a8 d2g2 a8e8 e6f7 e8b8 g2d2 b8b6 f7g6
info depth 10 score cp 221 nodes 19767 nps 1098166 time 18 pv d1d2 a7a8 d2g2 a8e8 e6f7 e8a8 g2f2 a8a3 f7e6 a3a6
info depth 11 score cp 236 nodes 29292 nps 1394857 time 21 pv d1d2 a7a8 d2g2 a8e8 e6f7 e8b8 g2f2 b8b3 f2c2 c5d5 f7g6
info depth 12 score cp 198 nodes 95238 nps 2442000 time 39 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6e7 c4c5 g2c2 c5d5
info depth 13 score cp 210 nodes 110640 nps 2514545 time 44 pv d1d2 a7a6 e6e5 c5c4 d2c2 c4d3 c2g2 a6a5 e5e6 a5a6 e6e7 d3d4 g2c2
info depth 14 score cp 232 nodes 140826 nps 2657094 time 53 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 a6a7 f7g6 a7a8 g2d2 c4c5
info depth 15 score cp 233 nodes 174188 nps 2764888 time 63 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 a6a7 f7g6 a7a8 g2d2 c4c5 f6f5
info depth 16 score cp 271 nodes 338974 nps 3167981 time 107 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 a6a7 f7g6 a7a6 g2g3 e4e5 g3f3 a6f6
info depth 17 score cp 277 nodes 423306 nps 3159000 time 134 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 a6a7 f7g6 a7a6 g2g3 e4e5 g3f3 a6f6 g6h5
info depth 18 score cp 378 nodes 611552 nps 3287913 time 186 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 a6a7 f7g6 a7a6 g2g3 e4e5 g3h3 e5f6 h3f3 c4b5
info depth 19 score cp 368 nodes 746811 nps 3219012 time 232 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 e4e5 f6e5 c4d5 g2g3 d5e5 g3h3 e5f5 h3f3 a6a7 f7f8
info depth 20 score cp 356 nodes 967761 nps 3225870 time 300 pv d1d2 a7a6 e6e5 c5c4 d2g2 a6a5 e5e6 a5a6 e6f7 e4e5 f6e5 c4d5 g2g3 d5e5 g3f3 e5f5 f3h3 a6a7 f7f8 f5g5
bestmove d1d2
position startpos moves e2e4 c7c5 b1c3 e7e6 g1f3 b8c6 d2d4 c5d4 f3d4 d8c7 c1e3 a7a6 f1d3 g8f6 e1g1 f8d6 g1h1 h7h5 d4c6 d7c6 h2h3 f6g4 d1d2 g4e3 d2e3 d6f4 e3c5 b7b6 c5a3 f4d6 a3b3 e8g8 a2a4 h5h4 c3e2 e6e5 d3c4 b6b5 a4b5 c6b5 c4d5 c8b7 c2c4 a8b8 b3d3 b5c4 d5c4 d6c5 e2c3 c5d4 c4a6 b7a6 a1a6 b8b2 c3d5 c7c2 d5e7 g8h7 d3f3 c2e2 f3f5 h7h8 f1a1 b2b1 a1b1 e2a6 f2f3 a6h6 b1d1 d4b2 d1d7 h6c1 h1h2 c1f4 f5f4 e5f4 d7b7 b2d4 e7c6 d4f2 b7b2 f8c8 b2f2 c8c6 f2a2 g7g6 h2g1 h8g7 g1f2 g7f6 f2e2 f6e6 e2d3 c6d6 d3e2 f7f6 a2b2 g6g5 b2c2 d6b6 e2d3 b6b1 c2a2 b1b3 d3c4 b3b6 c4c5 b6d6 a2b2 d6d1 c5c4 d1c1 c4d4 c1c6 b2a2 c6d6 d4c5 d6d1
go depth 10
info depth 1 score cp -1 nodes 27 nps 27000 time 0 pv a2a7
info depth 2 score cp 0 nodes 56 nps 56000 time 1 pv a2a7
info depth 3 score cp 0 nodes 106 nps 106000 time 1 pv a2a7
info depth 4 score cp 0 nodes 171 nps 171000 time 1 pv a2a7
info depth 5 score cp 0 nodes 251 nps 125500 time 2 pv a2a7
info depth 6 score cp 0 nodes 335 nps 167500 time 2 pv a2a7
info depth 7 score cp 0 nodes 476 nps 238000 time 2 pv a2a7
info depth 8 score cp 0 nodes 623 nps 207666 time 3 pv a2a7
info depth 9 score cp 0 nodes 2372 nps 790666 time 3 pv a2a7
info depth 10 score cp 0 nodes 3114 nps 778500 time 4 pv a2a7
bestmove a2a7

But I'm not aware of a GUI going "back" a move when using the engine? Maybe it could happen when the GUI wants to use the engine to compute something else in addition to the best move?

[Mike Sherwin]

The problem has gone away with a larger increment 200ms -> 500ms. I'm exactly 250 games into a 5000 game test. I've watched at least 50 of the games and have not noticed the problem even once. My theory is that LMR in real short lines is just not seeing necessary moves like having to give up a rook for a pawn in three moves to prevent queening.

[Guenther]

The way to solve this is to add a pawn hash table so you don't have to recheck the pawn structure every time, and then add an evaluation term for passed pawns. These are pawns that cannot be blocked or captured by enemy pawns.

This is too obvious a blunder to explain it with a too coarse evaluation function and I have pawn structure eval already including pawn hash table.

As far as I can see move 90 was where Leorik blunders by playing b7b3.

But if I hand that FEN to the engine directly I get a reasonable output:

Code: Select all

Leorik 2.4 Net8 Classic
position fen 8/1R6/2K5/8/8/2k5/2p5/8 w - - 8 90
go depth 20
info depth 1 score cp -694 nodes 49 nps 24500 time 2 pv b7b3
info depth 2 score cp -738 nodes 98 nps 14000 time 7 pv b7b3 c3b3
info depth 3 score cp 29 nodes 545 nps 68125 time 8 pv b7c7 c2c1b c7d7
info depth 4 score cp -22 nodes 830 nps 92222 time 9 pv b7c7 c2c1q c6d7 c3b2
info depth 5 score cp 88 nodes 1821 nps 202333 time 9 pv b7c7 c2c1q c6d7 c3b2 c7c1
info depth 6 score cp 18 nodes 2493 nps 249300 time 10 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1
info depth 7 score cp 44 nodes 3585 nps 325909 time 11 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e6
info depth 8 score cp 26 nodes 4444 nps 370333 time 12 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e6 c1d2
info depth 9 score cp 26 nodes 5415 nps 416538 time 13 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7e6
info depth 10 score cp 0 nodes 6500 nps 464285 time 14 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7e6 d2e1
info depth 11 score cp 0 nodes 8844 nps 552750 time 16 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7d6 d2e1 d6e6
info depth 12 score cp 0 nodes 10743 nps 631941 time 17 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7d6 d2e2 d6e6 e2e1
info depth 13 score cp 0 nodes 18418 nps 837181 time 22 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7d6 d2e2 d6e7 e2e1 e7e6
info depth 14 score cp 0 nodes 24858 nps 920666 time 27 pv b7c7 c2c1q c6d7 c3b2 c7c1 b2c1 d7e7 c1d2 e7d6 d2e2 d6e7 e2d2 e7d

Mike, if you observed more of these kind of games please send them via PM. That would be very helpful because so far I have failed to dig up more instances of this kind of bug from my logs.

Btw... the FEN-tag seems to be broken for me. E.g. [fen]8/1R6/2K5/8/8/2k5/2p5/8 w - - 8 90[/fen]

Well, the question is more, why there should be so much depth 1 moves at all? Depths that need just a few ms, even on my slow hardware
and this with an increment of 200ms, that seems absurd to me. A depth 1 move under normal circumstances could only be triggered by 'easy' move detection, or being extremely low on time. (but this cannot be the reason here due to the inc., 200ms is plenty nowadays)

I would never trust Arena ultra bullet games anyway. (actually I don't trust it in all aspects ;) and I know so much incredible bugs of it)
BTW it cannot even reflect the correct time control, because it always rounds to one sec precision and 200ms is simply zero for it.
Thus it writes 5+0 in the pgn header instead of 5+0.2.
Should be also the reason why no move times are recorded at all.

(Another reason could be that the machine used was overloaded with over work during the game and randomly caused this)

I bet you would never see this, if the match is done with CuteChess or WB with the same tc on your machine.

column style w/o useless main variation for visibility (depth 1 bold font)
[Event "RomiRomi"]
[Site "DESKTOP-HFVHK2B"]
[Date "2023.04.07"]
[Round "8"]
[White "Leorik.Classic"]
[Black "Leorik.KiSS"]
[Result "0-1"]
[BlackElo "2000"]
[ECO "D07"]
[Opening "QGD"]
[Time "00:55:57"]
[Variation "Chigorin, 3.cxd5 Main Line, 7.Bxc3"]
[WhiteElo "2000"]
[TimeControl "5+0"]
[Termination "normal"]
[PlyCount "189"]
[WhiteType "human"]
[BlackType "human"]

1. d4 d5
2. c4 Nc6
3. cxd5 Qxd5
4. e3 e5
5. Nc3 Bb4
6. Bd2 Bxc3
7. Bxc3 exd4
8. Ne2 Bg4
9. f3 O-O-O
10. Nxd4 Be6
11. Qb3 {+1.16/11} Nxd4 {-1.10/14}
12. Qxd5 {+0.86/16} Nc2+ {-1.19/16}
13. Kd2 {+0.89/16} Rxd5+ {-1.28/16}
14. Kxc2 {+0.92/16 1} Nf6 {-1.08/13}
15. e4 {+1.41/15} Rg5 {-0.98/13}
16. h4 {+1.08/14 1} Rg3 {-1.85/14}
17. Be1 {+1.62/13} Rg6 {-1.93/17}
18. h5 {+1.78/16} Rg5 {-2.17/17}
19. h6 {+2.51/16} Rg6 {-2.57/16}
20. hxg7 {+3.36/15} Rg8 {-3.12/15}
21. Bc3 {+3.38/15} R8xg7 {-3.33/15}
22. g4 {+3.66/16} Rg8 {-3.34/15}
23. g5 {+3.94/16} Nd7 {-4.25/14}
24. f4 {+4.07/14} Nc5 {-4.53/14}
25. f5 {+4.07/15} Rxg5 {-3.99/15}
26. fxe6 {+4.73/15} fxe6 {-4.93/15}
27. Re1 {+5.21/15} Rf8 {-5.13/14}
28. Rxh7 {+5.03/14} Rf2+ {-5.16/15}
29. Kc1 {+5.38/15} Rf3 {-5.61/14}
30. Be2 {+5.54/15} Rf8 {-6.11/14}
31. Bd4 {+6.11/15} Nxe4 {-6.09/16}
32. Bc4 {+6.09/15} Nd6 {-5.94/16}
33. Bxe6+ {+5.87/16} Kb8 {-5.91/16}
34. Bf6 {+5.95/15} Rg6 {-5.48/15}
35. Be7 {+5.55/14} Re8 {-5.80/17}
36. Bd5 {+5.60/14} Nc8 {-5.83/14}
37. Kc2 {+5.67/15} Rxe7 {-5.70/15}
38. Rhxe7 {+5.76/17} Nxe7 {-5.76/16}
39. Rxe7 {+5.53/16} c6 {-5.68/15}
40. Be4 {+5.66/14} Rh6 {-5.67/15}
41. a4 {+6.02/15} a5 {-6.03/16}
42. Re5 {+5.80/17} Rh4 {-5.67/17}
43. Kd3 {+5.61/18} Rh3+ {-5.67/19}
44. Kc4 {+5.66/18} Rh4 {-5.67/20}
45. b3 {+5.64/18} Kc7 {-5.54/18}
46. Kd4 {+5.54/19} Kb6 {-5.41/19}
47. Re7 {+5.54/19} Rg4 {-5.54/18}
48. Kc4 {+5.55/19} Rf4 {-5.47/19}
49. Kc3 {+5.56/20} Kc5 {-5.54/18}
50. Kd3 {+5.56/20} Kb6 {-5.54/18}
51. Kc4 {+5.54/19 1} Rh4 {-5.47/20}
52. Re6 {+5.54/19} Rf4 {-5.49/20}
53. Re7 {+5.67/1} Rh4 {0.00/35}
54. Kd4 {+5.91/1} Rg4 {0.00/31}
55. Ke5 {+6.04/1} Rg3 {-3.04/17}
56. Bc2 {+3.05/16} Rc3 {-3.36/18}
57. Bf5 {+3.58/17} Rxb3 {-3.19/18}
58. Kd6 {+3.39/17} Ra3 {-3.25/18}
59. Bc8 {+3.48/7} Rd3+ {-2.94/19}
60. Ke5 {+3.11/1} Rd8 {-2.94/17}
61. Bf5 {+2.94/16} Rd5+ {-2.63/17}
62. Kf6 {+2.38/18} Rd4 {-2.14/17}
63. Bc8 {+2.47/17} Rxa4 {-1.99/17}
64. Rxb7+ {+2.47/15} Kc5 {-2.14/16}
65. Re7 {+3.93/1} Rc4 {-3.41/1}
66. Ra7 {+2.30/16} Kb4 {-2.16/16}
67. Rb7+ {+2.30/17} Kc5 {-2.16/17}
68. Be6 {+1.46/16} Rc1 {-1.35/17}
69. Ra7 {+1.46/14} Kb4 {-1.35/15}
70. Rd7 {+3.22/1} a4 {-2.40/1}
71. Rb7+ {+1.00/16} Kc5 {-1.11/16}
72. Ke7 {+0.99/16} a3 {-0.81/17}
73. Ra7 {+1.02/17} Rc3 {-0.77/17}
74. Rd7 {+0.89/17} Kb4 {-1.16/1}
75. Bf7 {+1.19/1} c5 {-0.77/1}
76. Rb7+ {+1.56/17} Ka5 {-1.22/17}
77. Kd6 {+2.09/18} Rc2 {-1.67/17}
78. Kc6 {+2.50/18} Rb2 {-0.72/18}
79. Bb3 {+0.72/17} Rxb3 {-0.26/19}
80. Rxb3 {+0.72/15} a2 {-1.75/1}
81. Ra3+ {+3.22/19} Kb4 {-0.26/20}
82. Rxa2 {+0.44/21} c4 {0.00/22}
83. Rb2+ {+0.26/20} Ka3 {0.00/27}
84. Rb7 {+8.48/1} c3 {-7.26/1}
85. Rd7 {+7.44/1} c2 {-4.47/1}
86. Ra7+ {0.00/26} Kb2 {0.00/28}
87. Rb7+ {0.00/28} Ka2 {0.00/29}
88. Ra7+ {0.00/24} Kb2 {0.00/27}
89. Rb7+ {+3.64/1} Kc3 {-3.56/1}
90. Rb3+ {-6.94/1} Kxb3 {+7.38/1}
91. Kd7 {-18.84/16} c1=Q {+18.84/15}
92. Ke7 {-18.84/14} Qg5+ {+20.21/15}
93. Ke6 {-19.13/15} Kc4 {+M7/14}
94. Kf7 {-M5/7} Kd3 {+18.50/4}
95. Ke8 {-16.81/1 White resigns}
0-1

[lithander]

I agree with Günther that the examples Mike found are really dumb moves that the engine would find within a handful of plies and the engine typically searches 10 plies or more in 20ms. With an increment >100ms it should never lose on time unless the computer stalls the process for a long time. An example is if you plug in new hardware like an USB stick and you hear that Plug&Play sound. In that instances I've had Leorik lose on time before, but never under "normal" conditions in cutechess-cli.

If it happens on other GUIs too the TT bug still seems a likely a candidate I can not rule out. I have to go through more of my games and look for game-changing blunders that are easy to avoid with just a few plies of search.

For the reproduction case I gave above I think the best fix is to just not store positions from the move-history as 0 in the TT at all. I can't get them out again easily and I don't want to clear the TT when a position is set because normally it helps to have the results from the search two plys ago still around. So it follows that the TT is not a good place to store history. I've changed my repetition detection code to use array lookups:

Code: Select all

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private bool IsRepetition(int ply)
        {
            ulong hash = Positions[ply].ZobristHash;
            //start with the positions we've been searching
            for (int i = ply - 4; i >= 0; i -= 2)
            {
                if (Positions[i].ZobristHash == hash)
                    return true;

                //captures and pawn moves reset the halfmove clock for the purpose of enforcing the 50-move rule and also make a repetition impossible
                if (Positions[i].HalfmoveClock <= 1)
                    return false;
            }
            //continue with the full history of positions from the startpos
            int start = _moveHistory.Count - 1 - (ply & 1);
            for (int i = start; i >= 0; i -= 2)
            {
                if (_moveHistory[i].Hash == hash)
                    return true;

                //captures and pawn moves reset the halfmove clock for the purpose of enforcing the 50-move rule and also make a repetition impossible
                if (_moveHistory[i].Clock <= 1)
                    return false;
            }

            return false;
        }

Looks expensive but I guess it isn't in practice. You'll early-out with the HalfmoveClock trick pretty frequently and would rarely even get to the 2nd for-loop.

But now that I thought about the issue I feel like the problem might be more fundamental: you can never be sure if the score you got on a position might be influenced by the 50-move counter or the 3-fold rule leading to a significant change in the evaluation. (Or can you?) And if you store the position with that eval in the TT it might not be a true reflection of the position reached in a different context.

Do other programmers encode the HalfmoveClock in their Zobrist hash to prevent that? Are there other techniques? It's a potential problem I never really thought about and I admit that I don't grasp the implications fully at this point.

[Guenther]

I agree with Günther that the examples Mike found are really dumb moves that the engine would find within a handful of plies and the engine typically searches 10 plies or more in 20ms. With an increment >100ms it should never lose on time unless the computer stalls the process for a long time. An example is if you plug in new hardware like an USB stick and you hear that Plug&Play sound. In that instances I've had Leorik lose on time before, but never under "normal" conditions in cutechess-cli.

If it happens on other GUIs too the TT bug still seems a likely a candidate I can not rule out. I have to go through more of my games and look for game-changing blunders that are easy to avoid with just a few plies of search.

For the reproduction case I gave above I think the best fix is to just not store positions from the move-history as 0 in the TT at all. I can't get them out again easily and I don't want to clear the TT when a position is set because normally it helps to have the results from the search two plys ago still around. So it follows that the TT is not a good place to store history. I've changed my repetition detection code to use array lookups:

Code: Select all

        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        private bool IsRepetition(int ply)
        {
            ulong hash = Positions[ply].ZobristHash;
            //start with the positions we've been searching
            for (int i = ply - 4; i >= 0; i -= 2)
            {
                if (Positions[i].ZobristHash == hash)
                    return true;

                //captures and pawn moves reset the halfmove clock for the purpose of enforcing the 50-move rule and also make a repetition impossible
                if (Positions[i].HalfmoveClock <= 1)
                    return false;
            }
            //continue with the full history of positions from the startpos
            int start = _moveHistory.Count - 1 - (ply & 1);
            for (int i = start; i >= 0; i -= 2)
            {
                if (_moveHistory[i].Hash == hash)
                    return true;

                //captures and pawn moves reset the halfmove clock for the purpose of enforcing the 50-move rule and also make a repetition impossible
                if (_moveHistory[i].Clock <= 1)
                    return false;
            }

            return false;
        }

Looks expensive but I guess it isn't in practice. You'll early-out with the HalfmoveClock trick pretty frequently and would rarely even get to the 2nd for-loop.

But now that I thought about the issue I feel like the problem might be more fundamental: you can never be sure if the score you got on a position might be influenced by the 50-move counter or the 3-fold rule leading to a significant change in the evaluation. (Or can you?) And if you store the position with that eval in the TT it might not be a true reflection of the position reached in a different context.

Do other programmers encode the HalfmoveClock in their Zobrist hash to prevent that? Are there other techniques? It's a potential problem I never really thought about and I admit that I don't grasp the implications fully at this point.

Well I don't think it is a TT bug, you sure have noticed that move 90 in Mikes game example was played with the same eval! at depth 1 as in your
attempt for reproducing the move and I get the same too on my machine. With an eval of -6.94 it would surely extend the search to a higher
depth, if everything was 'normal' in this search. We don't even know, which search times Arena sends under certain circumstances...
(and it was never intended for really fast games - also you have to fiddle already with move animation etc., even under CuteChessGUI e.g.
in case you use it for extreme bullet)
May be if I have time in the evening I will try 100 games (Leorik vs. Leorik) under the CuteChessGUI with move animation off and check the depths later.
100 games should be enough, as Mike reported it happened already quite a lot in his aborted may be 50 games test.
I don't know though about the hash setting, but I will stay with default size, normally it should be probably lowered for 5+0.2 even more on a
slow hardware like mine ;)

[lithander]

I just played 6500 games at 5s each with no increment. This is putting a lot of time pressure on the engine but I didn't see a single time forfeit. And when you look how deep the engine searches there were only a few shallow searches in the log:

depth 3: 16 (all from the same game)
depth 4: 22 (most from the same game as depth 3)
depth 5: 75
depth 6: 200
depth 7: 927
depth 8: 3805
depth 9: 18615
depth 10: 69199
depth 11: 136152
depth 12: 127125
depth 13: 144414
depth 14: 107345
depth 15: 80483
depth 16: 54959
depth 17: 35432
depth 18: 21505
depth 19: 14088
depth 20: 8504

So even under rather extreme time pressure 99,9% of the moves were searched to a depth that should prevent obvious blunders. (but of course I didn't watch the actual games, so no idea if there were blunders or not)

[lithander]

My engine still uses PieceSquareTables for the eval. I have added a few tables for pawn-specific features like Connected, Protected and Passed Pawns and a mobility bonus but the principle remains the same. A common strategy to improve a simple PST based evaluation that was mentioned a few times on these forums is to swap out the in the PSTs relative to the king positions.

With all permutations of the two king's positions you'd end up with ~4000 different sets and some of the (white-king-sqr, black-king-sqr) pairs would be extremely rare and hard to find in the training data. So I thought I'd just create sets of 64 PSTs and focus on just one of the kings at a time. Later I would see how I could combine the two sets again.

The problem is that both the black and the white pieces share the same tables. If I select only positions where the white King has castled short then the black side's king would be all over the place and I wouldn't want the black pieces to mess up my king-on-g1 subset. So the first thing I did is change my tuner to keep black and white pieces in seperate piece square tables.

Now another problem is that some king positions are still relatively rare. I had only a few thousand positions with white king on a8 for example. To utilize my collection of selfplay games fully I don't completely ignore black positions. Instead if the black king would be positioned on the right square I flip the board so that black becomes white. That roughly doubles the amount of eligible positions.

Some subsets are still vastly larger than others but I thought I had enough to do a quick test and tune a set of PSTs for each of the 64 white king positions. And because it meant only tiny modifications to the code I did the same for black (the opponent), too.

Now I had 128 separate sets of PSTs but not yet a good intuition of what they encoded. While swapping out the tables at runtime would be a possibility I was hoping that I could find ways to encode the patterns in a more efficient way. How exactly? That would depend on the nature of the patterns. I was also worrying that my modestly sized set of labeled positions would not be enough for the more exotic tables and that the exotic sets are mostly encoding noise.

In other words, before moving forward I wanted to visualize my data! And without further ado, here's the visualization!



How to read the plot? Each little colorful square tile is equivalent to an entry in a normal PieceSquare table. Which piece is written on top (starting with Pawns) and which square is indicated by the dark blue pixel in the tile. Two side-by-side squares are the midgame and the endgame entry. Now, what would normally be just one number is now 64 numbers because I have tuned 64 different sets of PSTs, each tuned on a subset of my training data where all positions have the king on the same specific square. And that king's square coresponds exactly to one of the pixels in the small 8x8 tile. So it's basically a little chess board.

If all PSTs would have the same value then all pixels would be of the same color. Yellow is the baseline and it's what this piece would be worth on that square in my current evaluation e.g. a PST trained on all positions. So if a pixel is now red that means the piece in question is actually worth less than "normal" when the king is on that square. If it's green then the king on that position makes the piece more valuable. The difference between pure green and pure red is 512 pixel-steps which is equivalent to 512 cp. But again: the color encodes only the delta from the baseline not the true value of the piece.

How common a king-square is is also visualized: in the two extra mostly green tiles right at the top. You can see that Leorik really loves to castle-short and that the queen side is totally underrepresented.

The entire endeavor definitely gave me a few insights like that but not a clear path forward with improving the actual evaluation. I have a few ideas but if you see something in the plot and know how you'd try to capture it in the evaluation without just adding orders of magnitude more tables let me know!