SALC V5 openings and books launched

[pohl4711]

I cant read this format.

But even the EPD-file of the SALC V5 endpositions is 1.66 MByte in size. So a 0.667 MByte book can not be correct.

[tpoppins]

PGN-extract catches exact doubles but not "twins", i.e. lines that end in the same position via different move orders. To track those down you need a different set of tools, namely a combination of several programs from Norm Pollock's PGN and EPD sets.

I just finished a custom 10-move edit of Frank's new FEOBOS (strictly for CCRL testing). The official contempt-3 PGN has 33,009 lines. After truncating the PGN to 10 moves and removing duplicates with PGN-Extract I got around 31,000 lines. I put that PGN through Norm's tools and found more than 1,500 lines ending in the same position. After manually removing them (there may be a way to automate it if you put enough time to go carefully through the readmes for both the PGN and the EPD tool sets, but my head was too tired so I took the long-but-easy path) I got exactly 29,340 lines.

I'm not saying SALC5 got 18,000 "twins" (I suspect a user error there) but if you haven't put them through some tools comparable to Norm's, chances are that there are more of them in there than you might think.

BTW, I already completed three gauntlets using your Half-closed-10-move set, two at 40/4 and one at 40/40. At this point I don't see much difference from the 6-move FEOBOSv10 edit I used over the last couple of months in terms of results, except slightly lower draw rate and a lower number of games shorter than 20 moves (this latter can be explained by the lines in SALC5 being longer as well). I like variety and I like the idea behind your book, so I will continue using it and post another comparison once I get a decent number of games (perhaps 10,000 at least).

Extensive testing with engines other than the Top Three should also expose lines that may turn out to be not quite in tune with the uncompromising spirit of the idea behind SALC5. I found one such line already during the first gauntlet (yes, I actually look at some of the games played sometimes):

[pgn][Event "SCTR 1.1f 64-bit 4CPU 40/4 Gauntlet"]
[Site "CCRL"]
[Date "2018.02.11"]
[Round "14"]
[White "SCTR 1.1f 64-bit 4CPU"]
[Black "Gull 3.1 64-bit 4CPU"]
[Result "1/2-1/2"]
[ECO "C34"]
[PlyCount "33"]
[TimeControl "40/120:0"]

1. e4 {book} e5 {book} 2. f4 {book} exf4 {book} 3. Nf3 {book} Nc6 {book} 4. Bc4
{book} Bc5 {book} 5. d4 {book} Bb6 {book} 6. Bxf4 {book} d6 {book} 7. Nc3 {book
} Nf6 {book} 8. Qd2 {book} O-O {book} 9. O-O-O {book} Bg4 {book} 10. h3 {book}
Bxf3 {book} 11. gxf3 {0.00/22 5.6s} Nxd4 {-0.04/16 1.8s} 12. Qg2 {0.00/25 5.5s}
Ne6 {-0.06/16 3.2s} 13. Bd2 {0.00/26 5.4s} Nh5 {0.00/16 3.2s} 14. Qg4 {0.00/27
5.3s} Nf6 {0.00/18 1.1s} 15. Qg2 {0.00/29 5.2s} Nh5 {0.00/18 1.9s} 16. Qg4 {
0.00/28 5.1s} Nf6 {0.00/17 0.088s} 17. Qg2 {0.00/38 5.0s, Draw by 3-fold
repetition} 1/2-1/2

[Event "SCTR 1.1f 64-bit 4CPU 40/4 Gauntlet"]
[Site "CCRL"]
[Date "2018.02.11"]
[Round "14"]
[White "Gull 3 64-bit 4CPU"]
[Black "SCTR 1.1f 64-bit 4CPU"]
[Result "1/2-1/2"]
[ECO "C30"]
[PlyCount "43"]
[TimeControl "40/120:0"]

1. e4 {book} e5 {book} 2. f4 {book} Qh4+ {book} 3. g3 {book} Qe7 {book} 4. d3 {
book} d6 {book} 5. Nf3 {book} Nc6 {book} 6. Bg2 {book} Nf6 {book} 7. O-O {book}
Bg4 {book} 8. c3 {book} Qd7 {book} 9. Qb3 {book} O-O-O {book} 10. a4 {book} Nh5
{book} 11. a5 {0.08/15 4.9s} a6 {-0.42/19 5.2s} 12. d4 {0.00/14 1.6s} exf4 {
-0.22/21 4.7s} 13. gxf4 {-0.10/17 12} Bh3 {0.00/25 5.7s} 14. Bxh3 {0.00/18 4.6s
} Qxh3 {0.00/28 5.3s} 15. d5 {0.00/19 4.5s} Ne7 {0.00/28 5.2s} 16. Ng5 {
0.00/19 4.9s} Qg4+ {0.00/29 5.2s} 17. Kh1 {0.00/19 0} f6 {0.00/29 5.0s} 18. Nf7
{0.00/21 5.3s} Qh3 {0.00/30 4.9s} 19. Kg1 {0.00/20 3.4s} Qg4+ {0.00/31 4.8s}
20. Kh1 {0.00/23 4.9s} Qh3 {0.00/31 4.7s} 21. Kg1 {0.00/21 2.9s} Qg4+ {0.00/32
4.6s} 22. Kh1 {0.00/24 7.2s, Draw by 3-fold repetition} 1/2-1/2

[/pgn]

Perhaps that's not something you going to find if you only use SF and Komodo. Both engines here are 3000+ Elo so such results cannot be dismissed out of hand. So it seems to me that Frank's methodology has a point, after all.

Anyway, to me it looks like both of you guys work towards a similar goal using different approaches, and both are doing a great job. Variety is the spice of life and I hope it continues to be that way. There is no reason one could not alternate between the two books for different tests.

Thank you, Stefan.

[Graham Banks]

I cant read this format.

But even the EPD-file of the SALC V5 endpositions is 1.66 MByte in size. So a 0.667 MByte book can not be correct.

My largest ChessGUI book has almost 26,000 lines and is just over 3.2mb in size.
That should be roughly the size of your book too.
I'm wondering whether ChessGUI is still having an issue reading some of the games.

[MikeB]

I just released my brand new SALC V5 openings and books.

SALC means "S"hort "A"nd "L"ong "C"astling: white and black castling to opposite sides (if white played 0-0, black played 0-0-0. If white played 0-0-0, black played 0-0).
When using SALC-openings, the chance for attacks towards the opponent king is much higher than using normal opening-books. Because of this, computerchess using SALC openings, will bring more action and fun to watch (and a measureable lower number of draws), because the faster the computers get, the higher the quality of computerchess get and the higher the draw-rate in engine-engine-matches get...so the computerchess is in danger to die the "draw-death" in the near future. So, using SALC openings will give computerchess a future beyond playing only draws or using strange and incorrect gambit-openings for a lower draw-rate!

What is new in SALC V5 (compared to older SALC V1 - V3) ? SALC V5 contains half-closed SALC-positions, only. And that leads to measureable lower draw-rates, compared to older SALC-versions - take a look at the testing-results on my website. So, it is strongly recommended to use the new SALC V5 openings from now, only and not the older SALC-versions.

These are the half-closed filters:
3) On d-line or e-line at least one white and one black pawn (=one of both center-lines closed)
4) no pawn-capture on the center-squares (e4,d4,e5,d5) possible (means: not allowed: (white pawn on e4 and black pawn on d5) or (white pawn on d4 and black pawn on e5) - so, the position cannot get fully open after 1 or 2 played moves by the engines.
5) no pawn-free d-line, when both queens are on d-line. So, the queens cannot capture each other after 1 or 2 played moves by the engines.

The idea is, that in these half-closed positions, the probability of fast and many capturing-moves is much lower, so it should took more time (and moves) to reach drawish endgame-positions. So, the probability of an interesting and long midgame should get higher...

Checkout my website for testing results and further information and download SALC V5 there.

http://www.sp-cc.de/salc-openings.htm


Stefan (SPCC)

Hi Stefan,

I think this is very interesting , but I'm not 100% convinced that this is the best method to either tune an engine or to evaluate an engine. It does clearly reduce draws and make for a much more entertaining pgn file to review. But it seems that you if one is using positions engines which are more tactical than the average chess position - how are you going to tune an engine or even give credit to an engine for the the quiet moves , when the attacking moves predominate the positions more than the positional moves?

I think the engines that do very well tactically, SF for example , will do very well using these positions and those engines that known to be more positional in design will do less well and the bigger difference in Elo is faux, since a entire class of positions are not being tested. What happens when the engine plays an engine that plays more positional in a real tournament where the operator actually controls the book being used - and it may very steer the engine into a line that it rarely sees? So I would not get too ramped up about stressing the statistical value of these books and openings, but entertainment value, yes, it's Tier 1, A+ for entertainment value and most enthusiasts will enjoy your books openings very much, as I do. Thank you.

Regards,
Michael

[Graham Banks]

I cant read this format.

But even the EPD-file of the SALC V5 endpositions is 1.66 MByte in size. So a 0.667 MByte book can not be correct.

My largest ChessGUI book has almost 26,000 lines and is just over 3.2mb in size.
That should be roughly the size of your book too.
I'm wondering whether ChessGUI is still having an issue reading some of the games.

I'll have to wait for Matthias to comment.
I've created the book four times now, each time with the same result.

[pohl4711]

PGN-extract catches exact doubles but not "twins", i.e. lines that end in the same position via different move orders. To track those down you need a different set of tools, namely a combination of several programs from Norm Pollock's PGN and EPD sets.
.

Thats false,. Pgn-extract has an option called fuzzydepth. Then it looks only on the position after a number of plies.

You can check the EPD-file of SALC V5 endpositions for doubles with an editor for double-lines, if you dont believe, that pgnextract filtered all doubled endpositions out. All 24996 positions are unique.
Or use this online-tool, copy the SALC V5-EPDs out of an editor in there with drag & drop.

http://textmechanic.com/text-tools/basi ... ate-lines/


Here the manual of pgnextract:
.

Positional duplicates match
This flag allows a match on the basis of board position at the indicated number of plies or the end of the game. The flag is followed by the ply depth at which matches are to be considered. The value 0 is used to request matching at the end of games. It should always be used in combination with at least one of: -d/--duplicates, -D/--noduplicates, -U.

In contrast to the --duplicates matching, the match does not consider the move sequence used to reach the match position.

For example:

pgn-extract --fuzzydepth 40 -D games.pgn
would suppress from the output multiple copies of games reaching identical board positions after 40 ply.

[Dann Corbit]

The book Balsa_v2401.cgb has exactly 40,532 distinct interior and exterior nodes.

There are exactly 2366 book exit points.

[tpoppins]

PGN-extract catches exact doubles but not "twins", i.e. lines that end in the same position via different move orders. To track those down you need a different set of tools, namely a combination of several programs from Norm Pollock's PGN and EPD sets.
.

Thats false,. Pgn-extract has an option called fuzzydepth. Then it looks only on the position after a number of plies.

You can check the EPD-file of SALC V5 endpositions for doubles with an editor for double-lines, if you dont believe, that pgnextract filtered all doubled endpositions out. All 24996 positions are unique.

I didn't know about --fuzzydeph, thank you for the explanation.

[Fulvio]

I'm not saying SALC5 got 18,000 "twins" (I suspect a user error there) but if you haven't put them through some tools

I just checked and all the final positions in SALC_V5_hc_10m.pgn are unique.
This is a short SCID script that creates a EPD file with all the final positions and the relative repetition count:

Code: Select all

package require Tcl 8.6
package require Tk  8.6

set fName [tk_getOpenFile -filetypes { {".pgn" ".PGN"} } -title "Open PGN file"]
if {$fName == ""} { exit }

set baseId [sc_base open PGN "$fName"]
set nGames [sc_base numGames $baseId]
for &#123;set i 1&#125; &#123;$i <= $nGames&#125; &#123;incr i&#125; &#123;
	sc_game load $i
	sc_move end
	incr positions&#40;&#91;sc_pos fen&#93;)
&#125;

set file_output &#91;open "$fName.epd" w&#93;
set sort_by_count &#91;lsort -stride 2 -index 1 -integer &#91;array get positions&#93;&#93;
foreach &#123;fen count&#125; "$sort_by_count" &#123;
	puts $file_output "$fen noop $count;"
&#125;
close $file_output

tk_messageBox -message "Successfully written $fName.epd"
exit

[Graham Banks]

I cant read this format.

But even the EPD-file of the SALC V5 endpositions is 1.66 MByte in size. So a 0.667 MByte book can not be correct.

My largest ChessGUI book has almost 26,000 lines and is just over 3.2mb in size.
That should be roughly the size of your book too.
I'm wondering whether ChessGUI is still having an issue reading some of the games.

I'll have to wait for Matthias to comment.
I've created the book four times now, each time with the same result.

Really strange.

The Balsa_v2401.cgb and FOEBOSv20.1.cgb books seem to be okay.
It's only the SALCv5.02.pgn I have an issue with.

I stripped the pgn file using Chessbase Reader 2017 and still get the same result.