Something new, something borrowed

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

User avatar
vittyvirus
Posts: 646
Joined: Wed Jun 18, 2014 2:30 pm
Full name: Fahad Syed

Something new, something borrowed

Post by vittyvirus »

I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
-> For pinned sliders, '&' the slider attack masks with LineBetween[sq][own_king_square] for completely legal move set.
-> A pinned knight can't legally move

My aim wasn't to build the fastest perft program out there, and there is still lot of speed-up possible in this program, although I have never myself seen a program that beats it (but I believe there are many). My aim was merely to experiment with ideas I had, and to write a base for my completely original engine which would not use any copyrighted code (such as Pradu's). Also, I had an idea to pre-store moves inside MoveLists, and to index MoveLists using magic bitboards. I knew this would screw cache, and not pay off too much, but as I said, my aim was to experiment. I came to know that Gerd Isenberg had similar ideas in the past, but that didn't stop me. I used this code to find the magics (not really, the following code is a rather heavily evolved one)

Code: Select all

inline bool is_good_mlist_magic(bitboard_t magic, bitboard_t Atk,
                                std&#58;&#58;vector<bool>& used, count_t shift&#41;
&#123;
  const size_t MaxSize = 1 << &#40;64 - shift&#41;;
  assert&#40;used.size&#40;) >= MaxSize&#41;;
  for &#40;size_t i = 0; i < MaxSize; ++i&#41;
	used&#91;i&#93; = false;
  bitboard_t subset = 0;
  do &#123;
    uint32_t h = uint32_t&#40;&#40;magic * subset&#41; >> shift&#41;;
    assert&#40;h < MaxSize&#41;;
	if &#40;used&#91;h&#93; == true&#41; return false;
	used&#91;h&#93; = true;
  &#125; while &#40;subset = &#40;subset - Atk&#41; & Atk&#41;;	// Transverse all subsets
  return true;
&#125;

// Finds a magic which would hash all subsets of 'atk' uniquely into 64-'shift' bits.
inline bitboard_t find_mlist_magic&#40;count_t shift, bitboard_t atk&#41;
&#123;
  bitboard_t m;
  std&#58;&#58;vector<bool> temp&#40;1 << &#40;64-shift&#41;);	// Feel free to use an array
  for (;;) &#123;
	m = Random&#58;&#58;rand64_few_bits&#40;);
	if &#40;is_good_mlist_magic&#40;m, atk, temp, shift&#41;) break;
  &#125;
  return m;
&#125;
At first I tried using separate MoveLists for Rank, File, Diag, AntiDiag etc. but I realized that I should rather use a separate MoveList for Rook and Bishop. It was computationally infeasible for my laptop to find optimal 14-bit rook movelist magics, so I decided to go for 15 bits. This made the total table size 58 MB for Rooks, and 2 MB for Bishops! I then realized that the more sparsely populated the attack masks are, the less efficient this technique gets. So here are some runs for comparison with qperft and a version that doesn't use move lists (with some other optimizations) (some trucation done manually)
1. The start position (a position featuring sparsely populated slider attack masks). Yaka with movelists performed 1.26x faster than qperft, and without movelists performed 2.86x faster.

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=           20 ( 0.000 sec&#41;
perft&#40; 2&#41;=          400 ( 0.000 sec&#41;
perft&#40; 3&#41;=         8902 ( 0.000 sec&#41;
perft&#40; 4&#41;=       197281 ( 0.000 sec&#41;
perft&#40; 5&#41;=      4865609 ( 0.047 sec&#41;
perft&#40; 6&#41;=    119060324 ( 1.156 sec&#41;
perft&#40; 7&#41;=   3195901860 &#40;31.080 sec&#41;

Yaka 0.0 x64 by Syed Fahad
perft 7
b1a3&#58; 120142144
...truncated...
h2h4&#58; 138495290
Took 24626 ms for 3195901860 nodes, 129777 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
perft 7
b1a3&#58; 120142144
...truncated...
h2h4&#58; 138495290
Took 10874 ms for 3195901860 nodes, 293903 knps
2. Kiwipete. Yaka with movelists performed 1.40x faster, and without movelists performed 2.93x faster.

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=           48 ( 0.000 sec&#41;
perft&#40; 2&#41;=         2039 ( 0.000 sec&#41;
perft&#40; 3&#41;=        97862 ( 0.000 sec&#41;
perft&#40; 4&#41;=      4085603 ( 0.047 sec&#41;
perft&#40; 5&#41;=    193690690 ( 1.437 sec&#41;
perft&#40; 6&#41;=   8031647685 &#40;65.519 sec&#41;

Yaka 0.0 x64 by Syed Fahad
position kiwipete
perft 6
f3d3&#58; 164583144
...truncated...
g2g4&#58; 135208177
Took 46955 ms for 8031647685 nodes, 171049 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
position kiwipete
perft 6
a1b1&#58; 160413321
...truncated...
g2g4&#58; 135208177
Took 22354 ms for 8031647685 nodes, 359293 knps

3. [d] 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
Here, Yaka performed 1.24x faster, and without movelists it performed only 1.97x faster.

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=           14 ( 0.000 sec&#41;
perft&#40; 2&#41;=          191 ( 0.000 sec&#41;
perft&#40; 3&#41;=         2812 ( 0.000 sec&#41;
perft&#40; 4&#41;=        43238 ( 0.016 sec&#41;
perft&#40; 5&#41;=       674624 ( 0.000 sec&#41;
perft&#40; 6&#41;=     11030083 ( 0.172 sec&#41;
perft&#40; 7&#41;=    178633661 ( 2.078 sec&#41;
perft&#40; 8&#41;=   3009794393 &#40;40.283 sec&#41;

Yaka 0.0 x64 by Syed Fahad
position fen 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
perft 8
b4b1&#58; 334237659
...truncated...
g2g4&#58; 229481475
Took 32560 ms for 3007928693 nodes, 92381 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
position fen 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
perft 8
b4b1&#58; 334237659
...truncated...
g2g4&#58; 229481475
Took 20476 ms for 3007928693 nodes, 146900 knps

4. [d] r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1
In this position, Yaka with movelists performed 1.42x faster, and without movelists it's 3.31x faster!

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=            6 ( 0.000 sec&#41;
perft&#40; 2&#41;=          264 ( 0.000 sec&#41;
perft&#40; 3&#41;=         9467 ( 0.000 sec&#41;
perft&#40; 4&#41;=       422333 ( 0.000 sec&#41;
perft&#40; 5&#41;=     15833292 ( 0.141 sec&#41;
perft&#40; 6&#41;=    706045033 ( 7.641 sec&#41;
perft&#40; 7&#41;=  27209691363 &#40;260.328 sec&#41;

Yaka 0.0 x64 by Syed Fahad
position fen r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1
perft 7
g1h1&#58; 5992804777
f1f2&#58; 4558995210
b4c5&#58; 3227348274
f3d4&#58; 5092346265
c4c5&#58; 3409832969
d2d4&#58; 4928363868
Took 182889 ms for 27209691363 nodes, 148777 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
position fen r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1
perft 7
g1h1&#58; 5992804777
f1f2&#58; 4558995210
b4c5&#58; 3227348274
f3d4&#58; 5092346265
c4c5&#58; 3409832969
d2d4&#58; 4928363868
Took 78677 ms for 27209691363 nodes, 345840 knps
5. rnbqkb1r/pp1p1ppp/2p5/4P3/2B5/8/PPP1NnPP/RNBQK2R w KQkq - 0 6
In this position, Yaka with movelists is only 1.19x faster, but Yaka without movelists is 2.69x faster.

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=           42 ( 0.000 sec&#41;
perft&#40; 2&#41;=         1352 ( 0.000 sec&#41;
perft&#40; 3&#41;=        53392 ( 0.001 sec&#41;
perft&#40; 4&#41;=      1761505 ( 0.017 sec&#41;
perft&#40; 5&#41;=     70202861 ( 0.629 sec&#41;
perft&#40; 6&#41;=   2362704901 &#40;20.089 sec&#41;

Yaka 0.0 x64 by Syed Fahad
position fen rnbqkb1r/pp1p1ppp/2p5/4P3/2B5/8/PPP1NnPP/RNBQK2R w KQkq - 0 6
perft 6
d1d2&#58; 61274029
...truncated...
h2h4&#58; 59632702
Took 16981 ms for 2362704901 nodes, 139138 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
position fen rnbqkb1r/pp1p1ppp/2p5/4P3/2B5/8/PPP1NnPP/RNBQK2R w KQkq - 0 6
perft 6
d1d2&#58; 61274029
...truncated...
h2h4&#58; 59632702
Took 7470 ms for 2362704901 nodes, 316292 knps

I sometimes wish I could speed-up Stockfish's or Texel's move generator, but I can not contribute on GitHub (since internet is very slow here). Also, I'd leave a message for Peter Osturland: there's a _lot_ of speed-up possible in Texel's move generation, evaluation and search, and for sure, Texel is going to give a strong competition to Yaka in future years (sorry if that was an exceptionally bad joke).
=====================================================
To download these versions and test yourself: https://sites.google.com/site/sydfhd/projects/yaka
jwes
Posts: 778
Joined: Sat Jul 01, 2006 7:11 am

Re: Something new, something borrowed

Post by jwes »

vittyvirus wrote:I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
Try this position
[d] 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
User avatar
vittyvirus
Posts: 646
Joined: Wed Jun 18, 2014 2:30 pm
Full name: Fahad Syed

Re: Something new, something borrowed

Post by vittyvirus »

jwes wrote:
vittyvirus wrote:I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
Try this position
[d] 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
Well, Yaka fails in this position after perft 7. Can you tell me a position where there are more than three moves by pinned pawns?
User avatar
vittyvirus
Posts: 646
Joined: Wed Jun 18, 2014 2:30 pm
Full name: Fahad Syed

Re: Something new, something borrowed

Post by vittyvirus »

jwes wrote:
vittyvirus wrote:I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
Try this position
[d] 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
Well, Yaka fails in this position after perft 7. Can you tell me a position where there are more than three moves by pinned pawns?
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Something new, something borrowed

Post by zullil »

jwes wrote:
vittyvirus wrote:I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
Try this position
[d] 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
I get these:

Code: Select all

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 6
Leaf nodes = 35635077
Time taken = 428 ms

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 7
Leaf nodes = 290063345
Time taken = 18810 ms

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 8
Leaf nodes = 17665826996
Time taken = 217499 ms
User avatar
vittyvirus
Posts: 646
Joined: Wed Jun 18, 2014 2:30 pm
Full name: Fahad Syed

Re: Something new, something borrowed

Post by vittyvirus »

zullil wrote:
jwes wrote:
vittyvirus wrote:I had a lot of (perhaps not original) ideas for a fast move generator. Regarding legal move generation:
-> There are no more than three legal pinned pawns moves in a position, one in each direction.
Try this position
[d] 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
I get these:

Code: Select all

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 6
Leaf nodes = 35635077
Time taken = 428 ms

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 7
Leaf nodes = 290063345
Time taken = 18810 ms

ProcyonLeo-2&#58; ~/Documents/Chess/Kirby&#93; ./perft
FEN string = 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -
Depth = 8
Leaf nodes = 17665826996
Time taken = 217499 ms
These seem to be correct:

Code: Select all

frcperft 1.0, &#40;C&#41; 2008-2011 AJ Siemelink
single threaded, no hashing, mode=FAST, extract=BUILTIN, count=BUILTIN

   +---+---+---+---+---+---+---+---+
 8 |*r*|*n*|*b*|*q*|*k*|*b*|*n*|*r*|
   +---+---+---+---+---+---+---+---+
 7 |*p*|*p*|*p*|*p*|*p*|*p*|*p*|*p*|
   +---+---+---+---+---+---+---+---+
 6 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 5 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 4 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 3 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 2 |&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|&#40;P&#41;|
   +---+---+---+---+---+---+---+---+
 1 |&#40;R&#41;|&#40;N&#41;|&#40;B&#41;|&#40;Q&#41;|&#40;K&#41;|&#40;B&#41;|&#40;N&#41;|&#40;R&#41;|
   +---+---+---+---+---+---+---+---+
     a   b   c   d   e   f   g   h

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
interactive mode, type 'help'+Enter for help
% fen 1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - -

   +---+---+---+---+---+---+---+---+
 8 |   |&#40;Q&#41;|   |   |   |   |   |&#40;Q&#41;|
   +---+---+---+---+---+---+---+---+
 7 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 6 |   |   |   |*p*|   |*p*|   |   |
   +---+---+---+---+---+---+---+---+
 5 |   |   |&#40;R&#41;|*p*|*k*|*p*|&#40;R&#41;|   |
   +---+---+---+---+---+---+---+---+
 4 |   |   |&#40;P&#41;|*p*|*p*|*p*|&#40;P&#41;|   |
   +---+---+---+---+---+---+---+---+
 3 |   |   |&#40;Q&#41;|&#40;P&#41;|&#40;Q&#41;|&#40;P&#41;|&#40;B&#41;|   |
   +---+---+---+---+---+---+---+---+
 2 |   |   |   |   |   |   |   |   |
   +---+---+---+---+---+---+---+---+
 1 |   |   |   |   |&#40;K&#41;|   |   |   |
   +---+---+---+---+---+---+---+---+
     a   b   c   d   e   f   g   h

1Q5Q/8/3p1p2/2RpkpR1/2PpppP1/2QPQPB1/8/4K3 b - - 0 1
% perft 8
perft 1              3       0.00s      1.$ mnps 8566.7 ticks/move
perft 2            202       0.00s      1.$ mnps   56.5 ticks/move
perft 3           1130       0.00s      1.$ mnps   91.7 ticks/move
perft 4          74235       0.00s      1.$ mnps    5.5 ticks/move
perft 5         567103       0.00s      1.$ mnps   38.2 ticks/move
perft 6       35635077       0.06s    565.6 mnps    3.9 ticks/move
perft 7      290063345       3.30s     88.0 mnps   27.2 ticks/move
perft 8    17665826996      28.78s    613.8 mnps    3.9 ticks/move
%
PS I remember once you said that I'd give up coding a movegen that passes perft test. I think I'll have to say everyone a sorry for how I behaved that time. I really was a kid, and had no idea of what I'm saying. Peace.
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Something new, something borrowed

Post by zullil »

vittyvirus wrote: PS I remember once you said that I'd give up coding a movegen that passes perft test.
A year has gone by, so I don't remember specifics. I do remember urging you to start writing code, rather than talking about writing code!

In any case, you seem to have succeeded in writing a correct (and fast!) move generator. :D
zullil
Posts: 6442
Joined: Tue Jan 09, 2007 12:31 am
Location: PA USA
Full name: Louis Zulli

Re: Something new, something borrowed

Post by zullil »

vittyvirus wrote:[Can you tell me a position where there are more than three moves by pinned pawns?
[D]4k3/4r3/8/8/8/2q3b1/3PPP2/4K3 w - - 0 1

:D
ibid
Posts: 89
Joined: Mon Jun 13, 2011 12:09 pm

Re: Something new, something borrowed

Post by ibid »

vittyvirus wrote:3. [d] 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
Here, Yaka performed 1.24x faster, and without movelists it performed only 1.97x faster.

Code: Select all

Quick Perft by H.G. Muller
Perft mode&#58; No hashing, bulk counting in horizon nodes

perft&#40; 1&#41;=           14 ( 0.000 sec&#41;
perft&#40; 2&#41;=          191 ( 0.000 sec&#41;
perft&#40; 3&#41;=         2812 ( 0.000 sec&#41;
perft&#40; 4&#41;=        43238 ( 0.016 sec&#41;
perft&#40; 5&#41;=       674624 ( 0.000 sec&#41;
perft&#40; 6&#41;=     11030083 ( 0.172 sec&#41;
perft&#40; 7&#41;=    178633661 ( 2.078 sec&#41;
perft&#40; 8&#41;=   3009794393 &#40;40.283 sec&#41;

Yaka 0.0 x64 by Syed Fahad
position fen 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
perft 8
b4b1&#58; 334237659
...truncated...
g2g4&#58; 229481475
Took 32560 ms for 3007928693 nodes, 92381 knps

Yaka &#40;no movelists&#41; 0.0 x64 by Syed Fahad
position fen 8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1
perft 8
b4b1&#58; 334237659
...truncated...
g2g4&#58; 229481475
Took 20476 ms for 3007928693 nodes, 146900 knps

Did you notice you are getting the wrong answer here? I downloaded the no movelists version of yaka and it appears to get perft 5+ wrong.

A comparison of the no movelists yaka with the development version of gperft on my machine with these 5 positions (in ms)

Code: Select all

              yaka   gperft
1. perft 7   12285     3726
2. perft 6   24505    12746
3. perft 8   21975*   10047
4. perft 7   82845    53074
5. perft 6    8073     3204
User avatar
stegemma
Posts: 859
Joined: Mon Aug 10, 2009 10:05 pm
Location: Italy
Full name: Stefano Gemma

Re: Something new, something borrowed

Post by stegemma »

Is this a possible position? White has promoted 3 pawns, so black can have captured only 2 of them. Black must have captured 2 pawns + 4 pieces, to put pawns there but white has lost only N+N+B... so:

- black must have captured 2 pawns + 4 pieces
- white must have lose no more than 2 pawns + 3 pieces

If I'm not wrong, the position itself is impossible.