SPCC: Testruns of BrainFish 160610 and 160712 finished

[pohl4711]

Some people said, that the Elo-gain of Brainfish in my Testruns is too big (around +80 Elo), because Brainfish is "only" Stockfish with optimized opening-lines. But keep in mind, that the Ceberellum Library is calculated backwards. So each Library-move is based on very deep calculations, based on evaluations, which were done many plies ahead. Because of that, each Library-move is hundreds of Elo stronger, than a move, normally calculated and played by a chessengine. And thats the reason why only some played Library-moves in a complete game can raise the Elo of Brainfish so much above Stockfish (and additionally each Library-moves saves thinking time, which can be used by the engine later in the game for more node-crunching, which gains some more Elos).
But on the other hand, it has to be said, that the Ceberellum Library was fed with the 500 opening-positions, which I use for my testruns. And with games from older testruns of Stockfish, which used of course the same opening positions. So, Brainfish was tested "in a perfect world"...But the idea of that testruns were to measure the full potential of the Ceberellum Library. So, if Brainfish is tested with other opening-lines, the Elo-gain can (and should be) somewhat smaller.
And if you use some artificial, cruel opening lines, which are not included in the Library, Brainfish will gain no Elo and will play like a normal Stockfish.

[pohl4711]

Meanwhile we have another nice testresult of Brainfish, against asmFish, the superfast assembler-version of Stockfish...

1000 GAME GAUNTLET TEST
i3 380M | 2 @ 2.53 | Win7 x64 | 6 GB Ram
PB & TB~No | HT~On | Core~1 | ELO~Ordo 1.2
GUI~CuteChess | HASH~8MB | BOOK~TopGM_2move.pgn
TIME~20" + .2" (~63)

# PLAYER : RATING ERROR PLAYED (%) D(%)
1 Brainfish 160724 : 3235 7 1000 59.8 61.0
2 SF 160722ASMPD : 3165 7 1000 40.2 61.0

+70 Elo stronger...

Really impressive. Because asmFish is around +20 Elo stronger, than the normal speed Stockfish (and Brainfish is not faster than Stockfish)
and because the used openings are very short (only 2 moves deep).

[Guenther]

Meanwhile we have another nice testresult of Brainfish, against asmFish, the superfast assembler-version of Stockfish...

1000 GAME GAUNTLET TEST
i3 380M | 2 @ 2.53 | Win7 x64 | 6 GB Ram
PB & TB~No | HT~On | Core~1 | ELO~Ordo 1.2
GUI~CuteChess | HASH~8MB | BOOK~TopGM_2move.pgn
TIME~20" + .2" (~63)

# PLAYER : RATING ERROR PLAYED (%) D(%)
1 Brainfish 160724 : 3235 7 1000 59.8 61.0
2 SF 160722ASMPD : 3165 7 1000 40.2 61.0

+70 Elo stronger...

Really impressive. Because asmFish is around +20 Elo stronger, than the normal speed Stockfish (and Brainfish is not faster than Stockfish)
and because the used openings are very short (only 2 moves deep).

Note that it is practically 'trained' against Stockfish with your opening positions, which means performance against other programs and other openings NOT known beforehand should be much less.

It would need much more tests against other opponents and independent openings to get reliable results about how much cerebellum would increase the strength.

Guenther

[Ozymandias]

At 5' + 3'' you would witness halving of the elo gain, at LTC elo gain migth be minimal.

That would depend on how deep, the evaluations used in the library are. The point at which the book becomes a liability could be before 5'+3", or much later, depending on that factor.

Really impressive. Because asmFish is around +20 Elo stronger, than the normal speed Stockfish (and Brainfish is not faster than Stockfish)
and because the used openings are very short (only 2 moves deep).

Unless I'm confused, using a shorter book actually favours Brainfish, which starts taking benefit of its own book sooner.

[pohl4711]

Meanwhile we have another nice testresult of Brainfish, against asmFish, the superfast assembler-version of Stockfish...

1000 GAME GAUNTLET TEST
i3 380M | 2 @ 2.53 | Win7 x64 | 6 GB Ram
PB & TB~No | HT~On | Core~1 | ELO~Ordo 1.2
GUI~CuteChess | HASH~8MB | BOOK~TopGM_2move.pgn
TIME~20" + .2" (~63)

# PLAYER : RATING ERROR PLAYED (%) D(%)
1 Brainfish 160724 : 3235 7 1000 59.8 61.0
2 SF 160722ASMPD : 3165 7 1000 40.2 61.0

+70 Elo stronger...

Really impressive. Because asmFish is around +20 Elo stronger, than the normal speed Stockfish (and Brainfish is not faster than Stockfish)
and because the used openings are very short (only 2 moves deep).

Note that it is practically 'trained' against Stockfish with your opening positions, which means performance against other programs and other openings NOT known beforehand should be much less.

It would need much more tests against other opponents and independent openings to get reliable results about how much cerebellum would increase the strength.

Guenther

That testrun against asmfish was not done be me. And Brainfish was not trained with these opening-pgn file (TopGM_2move.pgn), which the tester used.