Frustrations of a mediocre chess programmer. What next?

[Chan Rasjid]

Hello Jesper,

I wonder how many of you have had the same frustrating experience as me.

When i started writing my engine, Pupsi, it was steady improvements for roughly a year and a half! It was wonderful! Thrilling! Exciting!

For SnailChess, if the excitement of getting close to Rybka ever come, it's rating is 700elo.

If Pupsi is at about elo 2500 - 2550, then my comments below would not be for you.

I) There is no secret in writing a chess program of GM level, say 2500, as there are open sources like Crafty, Fruit, Glaurung, Slowchess etc.. so in this sense it is very easy. The difficult part is only the coding.

The significant parts of a chess program are:-

Search
=====
1) root search
2) full search
3) QS
4) move ordering
5) transposition tables (or + pawn hashing)
6) null move
7) SEE
8) extension, reduction, pruning ,etc

Evaluation
======
whatever.

Hypothesis:- if everything above is "well done" and follow what are conservatively recommended, the program will have about and elo of at least 2500.

1) If each of the above 8 parts of search are implemented less than "optimal", and with simple assumption:-
3% penalty,Final Strength = 0.97 ^ 8 x 2500 = 1959.
2% penalty,Final Strength = 0.98 ^ 8 x 2500 = 2126
What if pupsi does not have SEE! ( x 0.95)

What if evaluation means a multiple of 0.9 to 1.12?
2500 * 0.9 = 2250
2500 * 1.12= 2800

At least such calculations show how badly bugs may kick in! Or whether evaluation is critical.

2) testing search - if we can do a cut and paste graft of Crafty's evaluation into our program and just play it against Crafty, it is almost a certain test of an engines's search. I think it is rather difficult and I have not found a way to do so.

3) testing evaluation - this part should be simple. We graft our evaluation into Fruit and then it would be crystal clear if it is ok.

Best Regards,
Rasjid

[bob]

I am not sure Winboard would be much faster; it might all be OS delays.

I always use Winboard_x under PSWBTM to set up gauntlets. There you can see the results while the match is running.

I do not know what is PSWBTM

practically I am not interested in book in my testing so my testing is from the noomen positions(I would like to increase the number of positions so I can get 1000 games or in chess960 1920 games from all positions.

I would like a tool that allows me to run the same match hopefully with 1000 games(I need to decide about pgn file) and repeat it with different number of nodes

I can start at 1000 nodes per move and later at 2000 nodes per move and later at 3000 nodes per move when I can follow the results.

I think that an interface that only save the games as pgn and show the results of every match with no graphic can be good for programmers.

Uri

I have such an interface, which I use on our cluster daily. I have not released it as it was never intended to be used by others. I have found some issues that I had to work around in the testing I do here. My interface does not try to keep up with the game, and legal/illegal moves. I decided to "trust Crafty" after running into a few problems with other programs (ArasanX 9 seems to confuse the "result strings" and will resignn but report the result as though it won. This caused some head-scratching at first, but I finally just decided to trust Crafty's result since it has been tested for years and is correct.

All my interface does is pass moves back and forth, and keep up with time used ala' xboard so that it can tell both engines how much time they have left, and it will call games due to flag when appropriate. I don't have any games that end on illegal move time losses so that hasn't been a problem for the opponents I regularly test against.

[jesper_nielsen]

Hello Jesper,

I wonder how many of you have had the same frustrating experience as me.

When i started writing my engine, Pupsi, it was steady improvements for roughly a year and a half! It was wonderful! Thrilling! Exciting!

For SnailChess, if the excitement of getting close to Rybka ever come, it's rating is 700elo.

If Pupsi is at about elo 2500 - 2550, then my comments below would not be for you.

I) There is no secret in writing a chess program of GM level, say 2500, as there are open sources like Crafty, Fruit, Glaurung, Slowchess etc.. so in this sense it is very easy. The difficult part is only the coding.

The significant parts of a chess program are:-

Search
=====
1) root search
2) full search
3) QS
4) move ordering
5) transposition tables (or + pawn hashing)
6) null move
7) SEE
8) extension, reduction, pruning ,etc

Evaluation
======
whatever.

!SNIP!

Best Regards,
Rasjid

Pupsi is not at the 2500 level. Yet.

Well I have all the parts! So now it is just a matter of bugfixing, and optimising them it seems!

In Pupsi's case, I think the search part is "overdone", and the evaluation part is "underdone". So one aspect to look at is how to remove the unnecessary parts of the search implementation.

But it looks like most people agree that the most important thing is to be bug free.

So bugfixing, bugfixing and more bugfixing!

Kind regards,
Jesper

[Onno Garms]

One more idea to spot bugs. Not very practical, but it would be certainly extremely useful.

Explain your algorithms in natural language to another programmer. Including weights for the evaluation, when precisely to prune etc. Don't give him your source. Let him implement the same algorithms. If you use bitboards, he should use mailbox, if you search recursively he should use a loop. When he is done, compare the searches, which should produce absolutely identical results. It's practically certain that they won't. Find the first difference, fix it, and go for the next difference. Apart from missunderstandings in your description you will find a large number of bugs in both engines.

This is probably the best way to paralelize the work at a chess engine!

[Pradu]

One more idea to spot bugs. Not very practical, but it would be certainly extremely useful.

Explain your algorithms in natural language to another programmer. Including weights for the evaluation, when precisely to prune etc. Don't give him your source. Let him implement the same algorithms. If you use bitboards, he should use mailbox, if you search recursively he should use a loop. When he is done, compare the searches, which should produce absolutely identical results. It's practically certain that they won't. Find the first difference, fix it, and go for the next difference. Apart from missunderstandings in your description you will find a large number of bugs in both engines.

This is probably the best way to paralelize the work at a chess engine!

The way you program the engine itself could be different. For example move ordering when using bitboards and move ordering when using mailbox will be different so results won't be the same. But after fixing cases like that, it seems a good idea.