Gaviota tablebases, Probing Code Release (Finally)

Discussion of chess software programming and technical issues.

Moderator: Ras

User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Gaviota tablebases, Probing Code Release (Finally)

Post by michiguel »

http://sites.google.com/site/gaviotache ... e/releases

Enjoy, it comes with zero support :-)
You are free to do whatever the MIT License allow you to do, which is almost everything.

Version is v0.1, meaning, I hope there is no bugs, but if there is any, let me know and v1.0 won't have them. Some other features are intended to be added before 1.0. In any case, this is what I am using currently in Gaviota and works well.

It was a painful job to excise the whole code from Gaviota to release this as a module. The code is still messy and should be considered a useful prototype. It works (or at least it is supposed to), but do not go to look for beauty in it. I feel quite embarrassed about it.

The performance is acceptable, IMHO, but there are lots of room for improvement. In fact, some functions were designed on purpose to be simple so they will work the first time. Still, some of those were not optimized and are primitive. Since the bottleneck is HD access, you will probably won't notice it. The cache system is quite efficient, IMHO though, which is important.

If there is anything good about this, is the function tb_probe_soft, which will probably set this apart from the Nalimov scheme. In addition, the compression of the scheme 4 is better.

Good luck with them!

Miguel
PS: The generator code release will come later.
User avatar
Greg Strong
Posts: 388
Joined: Sun Dec 21, 2008 6:57 pm
Location: Washington, DC

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by Greg Strong »

Thank you!!! Look forward to playing around with it!
Michel
Posts: 2292
Joined: Mon Sep 29, 2008 1:50 am

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by Michel »

Thanks!

An open source implementation of tablebases was much needed.

Is there a reference for the API somewhere or should one just look at the source code?
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by michiguel »

Michel wrote:Thanks!

An open source implementation of tablebases was much needed.

Is there a reference for the API somewhere or should one just look at the source code?
This is the API inside gtb-probe.h

Code: Select all

/*
This Software is distributed with the following X11 License,
sometimes also known as MIT license.
 
Copyright (c) 2010 Miguel A. Ballicora

 Permission is hereby granted, free of charge, to any person
 obtaining a copy of this software and associated documentation
 files (the "Software"), to deal in the Software without
 restriction, including without limitation the rights to use,
 copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the
 Software is furnished to do so, subject to the following
 conditions:

 The above copyright notice and this permission notice shall be
 included in all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 OTHER DEALINGS IN THE SOFTWARE.
*/

#if !defined(H_GTBPROBE)
#define H_GTBPROBE
#ifdef __cplusplus
extern "C" {
#endif
/*>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>*/

#include <stdlib.h>

#define tb_MAXPATHLEN 1024

/*----------------------------------*\
|            CONSTANTS
\*----------------------------------*/

enum TB_mask_values { tb_RESMASK = 3, tb_INFOMASK = 7, tb_PLYSHIFT = 3 };

enum TB_return_values {   
   tb_DRAW = 0,
   tb_WMATE = 1,
   tb_BMATE = 2,
   tb_FORBID = 3,
   tb_UNKNOWN = 7
};

enum TB_pieces    {
   tb_NOPIECE, tb_PAWN, tb_KNIGHT, tb_BISHOP, tb_ROOK, tb_QUEEN, tb_KING
};

enum TB_sides {
   tb_WHITE_TO_MOVE, tb_BLACK_TO_MOVE
};

enum TB_squares {
   tb_A1, tb_B1, tb_C1, tb_D1, tb_E1, tb_F1, tb_G1, tb_H1,
   tb_A2, tb_B2, tb_C2, tb_D2, tb_E2, tb_F2, tb_G2, tb_H2,
   tb_A3, tb_B3, tb_C3, tb_D3, tb_E3, tb_F3, tb_G3, tb_H3,
   tb_A4, tb_B4, tb_C4, tb_D4, tb_E4, tb_F4, tb_G4, tb_H4,
   tb_A5, tb_B5, tb_C5, tb_D5, tb_E5, tb_F5, tb_G5, tb_H5,
   tb_A6, tb_B6, tb_C6, tb_D6, tb_E6, tb_F6, tb_G6, tb_H6,
   tb_A7, tb_B7, tb_C7, tb_D7, tb_E7, tb_F7, tb_G7, tb_H7,
   tb_A8, tb_B8, tb_C8, tb_D8, tb_E8, tb_F8, tb_G8, tb_H8,
   tb_NOSQUARE
};

enum TB_castling {
   tb_NOCASTLE = 0,
   tb_WOO  = 8,
   tb_WOOO = 4,    
   tb_BOO  = 2,
   tb_BOOO = 1
};

enum TB_compression_scheme {
	tb_UNCOMPRESSED, tb_CP1, tb_CP2, tb_CP3, tb_CP4 
};

/*----------------------------------*\
|         	FUNCTIONS
\*----------------------------------*/

extern void			tb_init   (int verbosity, int compression_scheme, char **paths);

extern void			tb_restart(int verbosity, int compression_scheme, char **paths);

extern void			tb_done (void);

extern int /*bool*/	tb_probe_hard 
							(unsigned stm, 
			 				 unsigned epsq,
							 unsigned castles,
			 				 const unsigned *inp_wSQ, 
			 				 const unsigned *inp_bSQ,
			 				 const unsigned char *inp_wPC, 
			 				 const unsigned char *inp_bPC,
			 				 /*@out@*/ unsigned *tbinfo, 
			 				 /*@out@*/ unsigned *plies);

extern int /*bool*/	tb_probe_soft 
							(unsigned stm, 
			 				 unsigned epsq,
							 unsigned castles,
			 				 const unsigned *inp_wSQ, 
			 				 const unsigned *inp_bSQ,
			 				 const unsigned char *inp_wPC, 
			 				 const unsigned char *inp_bPC,
			 				 /*@out@*/ unsigned *tbinfo, 
			 				 /*@out@*/ unsigned *plies);

extern int /*bool*/	tb_is_initialized (void);

/*----------------------------------*\
|         	cache
\*----------------------------------*/

extern int /*bool*/	tbcache_init (size_t cache_mem);
extern void			tbcache_done (void);
extern int /*bool*/	tbcache_is_on (void);

/*----------------------------------*\
|         	STATS
\*----------------------------------*/

/*
For maximum portability, some stats are provided
in two 32 bits integers rather than a single 64 bit
number. For intance, prob_hard_hits[0] contains only the
less significant 32 bits (bit 0 to 31), and prob_hard_hitsp[1] the
most significant ones (32 to 63). The number can be recreated
like this
uint64_t x = (uint64_t)probe_hard_hits[0] | ((uint64_t)probe_hard_hits[1] << 32);
The user has the responsibility to use the proper 64 bit integers.
*/

struct TB_STATS {
	long unsigned int probe_hard_hits[2];
	long unsigned int probe_hard_miss[2];
	long unsigned int probe_soft_hits[2];
	long unsigned int probe_soft_miss[2];
	long unsigned int bytes_read[2];
	long unsigned int files_opened;
	long unsigned int blocks_occupied;
	long unsigned int blocks_max;	
	long unsigned int comparisons;
};

extern void			tbstats_reset (void);
extern void 		tbstats_get (struct TB_STATS *stats);

/*----------------------------------*\
|         	PATH MANAGEMENT
\*----------------------------------*/

extern char ** 		tbpaths_init(void);
extern char ** 		tbpaths_add(char **ps, char *newpath);
extern char ** 		tbpaths_done(char **ps);

extern const char *	tbpaths_getmain (void);

/*<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<*/
#ifdef __cplusplus
}
#endif
#endif
And this is how to use it. I think that it is self-explanatory but please let me know what it is not. I tried to incorporate all the suggestions I got in December.

I still think that I need to provide what is guaranteed to be kept or not (it is tb_A1 guaranteed to be 0 always? etc.) but I did not do that because: I hope I will do it for v1.0 and lazyness. Shooting for perfection would have delayed this even longer. I will improve things as we move along.

Miguel

Code: Select all

/*
This Software is distributed with the following X11 License,
sometimes also known as MIT license.
 
Copyright (c) 2010 Miguel A. Ballicora

 Permission is hereby granted, free of charge, to any person
 obtaining a copy of this software and associated documentation
 files (the "Software"), to deal in the Software without
 restriction, including without limitation the rights to use,
 copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the
 Software is furnished to do so, subject to the following
 conditions:

 The above copyright notice and this permission notice shall be
 included in all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 OTHER DEALINGS IN THE SOFTWARE.
*/

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "gtb-probe.h"

int main (int argc, char *argv[])
{
	/*--------------------------------------*\
	|   Probing info to provide
	\*--------------------------------------*/

	int	stm;				/* side to move */
	int	epsquare;			/* target square for an en passant capture */
	int	castling;			/* castling availability, 0 => no castles */
	unsigned int  ws[17];	/* list of squares for white */
	unsigned int  bs[17];	/* list of squares for black */
	unsigned char wp[17];	/* what white pieces are on those squares */
	unsigned char bp[17];	/* what black pieces are on those squares */

	/*--------------------------------------*\
	|   Probing info requested
	\*--------------------------------------*/

	int tb_available;			/* 0 => FALSE, 1 => TRUE */
	unsigned info = tb_UNKNOWN;	/* default, no tbvalue */
	unsigned pliestomate;	

	/*--------------------------------------*\
	|   Initialization info to be provided
	\*--------------------------------------*/

	int verbosity = 0;		/* initialization 0 = non-verbose, 1 = verbose */
	int	scheme = tb_CP4;	/* compression scheme to be used */
	char ** paths;		/* paths where files will be searched */
	size_t cache_size = 32*1024*1024; /* 32 MiB in this example */


	/*----------------------------------*\
	|	Return version
	\*----------------------------------*/

	#include "version.h"
	#include "progname.h"
	if (argc > 1 && 0==strcmp(argv[1],"-v")) {
		printf ("%s %s\n",PROGRAM_NAME,VERSION);
		return 0;
	}

	/*--------------------------------------*\
	|   Initialization:
	|   Include something like this at
	|   the beginning of the program.   
	\*--------------------------------------*/

	paths = tbpaths_init();
	paths = tbpaths_add (paths, "gtb/gtb4");
	paths = tbpaths_add (paths, "gtb/gtb3");
	paths = tbpaths_add (paths, "gtb/gtb2");
	paths = tbpaths_add (paths, "gtb/gtb1");

	tb_init (verbosity, scheme, paths);

	tbcache_init(cache_size);

	tbstats_reset();

	/*--------------------------------------*\
	|
	|   ASSIGNING POSITIONAL VALUES for
	|   one probing example
	|   
	\*--------------------------------------*/

#if 0

	/* FEN: 1r6/6k1/8/8/8/8/1P6/1KR5 w - - 0 1 */

	stm      = tb_WHITE_TO_MOVE;/* 0 = white to move, 1 = black to move */
	epsquare = tb_NOSQUARE;		/* no ep available */
	castling = tb_NOCASTLE;		/* no castling available */

	ws[0] = tb_B1;
	ws[1] = tb_C1;
	ws[2] = tb_B2;
	ws[3] = tb_NOSQUARE;		/* it marks the end of list */

	wp[0] = tb_KING;
	wp[1] = tb_ROOK;
	wp[2] = tb_PAWN;
	wp[3] = tb_NOPIECE;			/* it marks the end of list */

	bs[0] = tb_G7;
	bs[1] = tb_B8;
	bs[2] = tb_NOSQUARE;		/* it marks the end of list */

	bp[0] = tb_KING;
	bp[1] = tb_ROOK;
	bp[2] = tb_NOPIECE;			/* it marks the end of list */

#else

	/* FEN: 8/8/8/4k3/8/8/8/KR6 w - - 0 1 */

	stm      = tb_WHITE_TO_MOVE;/* 0 = white to move, 1 = black to move */
	epsquare = tb_NOSQUARE;		/* no ep available */
	castling = tb_NOCASTLE;		/* no castling available */

	ws[0] = tb_A1;
	ws[1] = tb_B1;
	ws[2] = tb_NOSQUARE;		/* it marks the end of list */

	wp[0] = tb_KING;
	wp[1] = tb_ROOK;
	wp[2] = tb_NOPIECE;			/* it marks the end of list */

	bs[0] = tb_E5;
	bs[1] = tb_NOSQUARE;		/* it marks the end of list */

	bp[0] = tb_KING;
	bp[1] = tb_NOPIECE;			/* it marks the end of list */

#endif

	/*--------------------------------------*\
	|
	|      		PROBING TBs
	|   
	\*--------------------------------------*/

	tb_available = tb_probe_hard (stm, epsquare, castling, ws, bs, wp, bp, &info, &pliestomate);

	if (tb_available) {

		if (info == tb_DRAW)
			printf ("Draw\n");
		else if (info == tb_WMATE && stm == tb_WHITE_TO_MOVE)
			printf ("White mates, plies=%u\n", pliestomate);
		else if (info == tb_BMATE && stm == tb_BLACK_TO_MOVE)
			printf ("Black mates, plies=%u\n", pliestomate);
		else if (info == tb_WMATE && stm == tb_BLACK_TO_MOVE)
			printf ("Black is mated, plies=%u\n", pliestomate);
		else if (info == tb_BMATE && stm == tb_WHITE_TO_MOVE)
			printf ("White is mated, plies=%u\n", pliestomate);         
		else {
			printf ("FATAL ERROR, This should never be reached\n");
			exit(EXIT_FAILURE);
		}
	} else {
		printf ("Tablebase info not available\n");   
	}

	/*--------------------------------------*\
	|	Clean up at the end of the program
	\*--------------------------------------*/

	tbcache_done();

	tb_done();

	paths = tbpaths_done(paths);

	/*--------------------------------------*\
	|         		Return
	\*--------------------------------------*/

	if (tb_available)
		return EXIT_SUCCESS;
	else
		return EXIT_FAILURE;
} 
Dann Corbit
Posts: 12777
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by Dann Corbit »

michiguel wrote:http://sites.google.com/site/gaviotache ... e/releases

Enjoy, it comes with zero support :-)
You are free to do whatever the MIT License allow you to do, which is almost everything.

Version is v0.1, meaning, I hope there is no bugs, but if there is any, let me know and v1.0 won't have them. Some other features are intended to be added before 1.0. In any case, this is what I am using currently in Gaviota and works well.

It was a painful job to excise the whole code from Gaviota to release this as a module. The code is still messy and should be considered a useful prototype. It works (or at least it is supposed to), but do not go to look for beauty in it. I feel quite embarrassed about it.

The performance is acceptable, IMHO, but there are lots of room for improvement. In fact, some functions were designed on purpose to be simple so they will work the first time. Still, some of those were not optimized and are primitive. Since the bottleneck is HD access, you will probably won't notice it. The cache system is quite efficient, IMHO though, which is important.

If there is anything good about this, is the function tb_probe_soft, which will probably set this apart from the Nalimov scheme. In addition, the compression of the scheme 4 is better.

Good luck with them!

Miguel
PS: The generator code release will come later.
This is utterly fantastic news. This is good news for:
1. Those who requested the use of Nalimov tablebase files and for *years* got as many replies as they got from the Egyptian Sphynx.
2. Those who want to add tablebase files but for which Nalimov files are not an option (e.g. license is incompatible).
3. Those who want to study tablebase files (though it will be much more useful when the generation code is released).

I do herefore make a beseaching plea to chess programmers:
Standardize on the Gaviota tablebase files, or (at least) have an interface to them as an alternative. I don't want 9 different collections of 6 man EGTB files on my hard drives and neither does anyone else. If you want to write your own tablebase files, I think it is a wonderful idea, and I support that. But think of the poor slobs who have to install these disk chewing collections on their poor, straining hard drives. A terabyte will soon be under $100? Sure. But 7 man files are right around the corner.

Give us this choice, pretty please, down on my knees, with sugar and cream on it and a cherry.
;-)
User avatar
Zach Wegner
Posts: 1922
Joined: Thu Mar 09, 2006 12:51 am
Location: Earth

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by Zach Wegner »

Hello Miguel,

Thanks! Looks nice. I'm glad that an open source TB is now available, I hope it will replace Nalimov with time.

I had a quick look through the source, and I'm curious how much code was handwritten and how much was automatically generated. Specifically the hundreds of indexing functions. How much time did it take to write too?

Regards,
Zach
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by michiguel »

Zach Wegner wrote:Hello Miguel,

Thanks! Looks nice. I'm glad that an open source TB is now available, I hope it will replace Nalimov with time.

I had a quick look through the source, and I'm curious how much code was handwritten and how much was automatically generated. Specifically the hundreds of indexing functions. How much time did it take to write too?

Regards,
Zach
Everything was manually typed, except the table when I rewrote it to avoid confusions. I started slowly and it took a long time (I started the project fall 2008). Of course, I completely underestimated how much work this was (of course I did not work on this full time). Another reason is that I changed many of them as I made progress and I realized I should do them differently. There is no question that for the 6-pc all of them will have to be generated by a code generator. Once I know what to do, It will be insane to do it manually and utterly boring. Writing the generator will be fun :-)

Miguel
Aaron Becker
Posts: 292
Joined: Tue Jul 07, 2009 4:56 am

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by Aaron Becker »

Looks great, I'll be experimenting with adding support to Daydreamer. FYI, some compilers (at least gcc 4.0.1) don't work correctly with variables that start with two underscores, because those symbols are reserved for system use. This breaks wrap.c and hzip.c, but is easily fixed by renaming the offending variables.

edit: One question. For UCI engines, would you recommend a standard set of options to control tablebase use? I'm sure end users would be appreciative if there was some standardization on this point.
kingliveson

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by kingliveson »

Miguel, congratulations on a fine work well done.

Franklin
bnemias
Posts: 373
Joined: Thu Aug 14, 2008 3:21 am
Location: Albuquerque, NM

Re: Gaviota tablebases, Probing Code Release (Finally)

Post by bnemias »

I wanted to express my thanks on this as well. It's a much needed and appreciated effort.