TalkChess.com

Thought I would move this since the other thread has gotten pretty long, as well as getting off-topic a bit. I have now fixed my referee program so that it can properly adjudicate games as won/lost/drawn. It maintains a board state (borrowed code from crafty but rewrote move generator to avoid the magic stuff to keep the code short. Since speed is not an issue, it just directly generates moves in bitboards rather than doing rotated lookups or magic stuff. I had to do this because not all programs provide accurate "Result" commands and can't be trusted. It can now ignore them and play anybody vs anybody. Only thing that is not allowed is draw offers. I had too many problems with programs not handling that correctly and decided to just disable the "offer draw" stuff and let the game continue. It is a forced draw if the 3-fold repetition is hit, or the 50 move rule, or insufficient material, regardless of whether one side claims a draw or not, which keeps the test games to a reasonable length.

I have run a partial test with the 6 programs so far. I am going to make 4 runs overnight, each opponent plays 160 games against every other opponent. And then I will do that 4 times to see how things look. I'm going to show the data two different ways, as the preliminary results are interesting. I will produce 4 sets of output from BayesElo where everybody plays everybody, and then 4 sets of output using only the Crafty vs everybody else PGN files, to see what happens. Remember that there are a couple of issues. One is what is the Elo spread from best to worst, and then what is the stability like. I should be able to post all of this in the morning if my shell script doesn't have a glaring error I missed.

More to follow...

Here are some partial results just for information: First batch is everybody vs everybody, second batch is just crafty vs each opponent... The two samples were made at the same time, for reference. Has a ways to go until it is done. I will then run this again, but with the "big" match. And will eventually stop running the all vs all, since once each non-crafty has played all the others, those versions do not change and there is little use in re-running them over and over. I will just take those PGN files and add them to the crafty vs everyone so that I just have to run crafty vs everyone after the first run to get all the PGN.

opponent 21.7 is crafty 21.7 for reference, we wanted to keep a 21.7 version in to see how the new version is doing. 22.2 is pretty incomplete with pieces of the eval chopped out...

The most importaint thing is that you make the referee save the core or machine it was running on and the time it was played, next to the starting position, players and result. You could do this in the PGN tags, so that it can be extracted later, or write it in a more compact form, so that aberrant results can be scrutinized for ' cause of death'. Rather than having to rely on an "it did not work, tell me what's wrong " approach.

bob wrote:Thought I would move this since the other thread has gotten pretty long, as well as getting off-topic a bit. I have now fixed my referee program so that it can properly adjudicate games as won/lost/drawn. It maintains a board state (borrowed code from crafty but rewrote move generator to avoid the magic stuff to keep the code short. Since speed is not an issue, it just directly generates moves in bitboards rather than doing rotated lookups or magic stuff. I had to do this because not all programs provide accurate "Result" commands and can't be trusted. It can now ignore them and play anybody vs anybody. Only thing that is not allowed is draw offers. I had too many problems with programs not handling that correctly and decided to just disable the "offer draw" stuff and let the game continue. It is a forced draw if the 3-fold repetition is hit, or the 50 move rule, or insufficient material, regardless of whether one side claims a draw or not, which keeps the test games to a reasonable length.

I have run a partial test with the 6 programs so far. I am going to make 4 runs overnight, each opponent plays 160 games against every other opponent. And then I will do that 4 times to see how things look. I'm going to show the data two different ways, as the preliminary results are interesting. I will produce 4 sets of output from BayesElo where everybody plays everybody, and then 4 sets of output using only the Crafty vs everybody else PGN files, to see what happens. Remember that there are a couple of issues. One is what is the Elo spread from best to worst, and then what is the stability like. I should be able to post all of this in the morning if my shell script doesn't have a glaring error I missed.

More to follow...

Here are some partial results just for information: First batch is everybody vs everybody, second batch is just crafty vs each opponent...

Code: Select all

2291 game(s) loaded, 0 game(s) with unknown result ignored.
Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung 2-epsilon/5    73   19   19   763   61%   -15   18% 
   2 opponent-21.7           26   18   18   759   54%    -5   23% 
   3 Fruit 2.1                5   19   19   754   51%    -2   15% 
   4 Glaurung 1.1 SMP       -15   19   19   764   47%     2   16% 
   5 Crafty-22.2            -33   18   18   757   45%     6   22% 
   6 Arasan 10.0            -56   19   19   785   42%    11   10% 

760 game(s) loaded, 0 game(s) with unknown result ignored.
Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung 2-epsilon/5   115   44   42   153   69%   -31   17% 
   2 Fruit 2.1               64   42   41   149   63%   -31   18% 
   3 Glaurung 1.1 SMP        48   42   41   152   61%   -31   16% 
   4 opponent-21.7           20   38   37   152   58%   -31   41% 
   5 Crafty-22.2            -31   19   19   760   45%     6   22% 
   6 Arasan 10.0           -216   43   45   154   26%   -31   16% 

The two samples were made at the same time, for reference. Has a ways to go until it is done. I will then run this again, but with the "big" match. And will eventually stop running the all vs all, since once each non-crafty has played all the others, those versions do not change and there is little use in re-running them over and over. I will just take those PGN files and add them to the crafty vs everyone so that I just have to run crafty vs everyone after the first run to get all the PGN.

opponent 21.7 is crafty 21.7 for reference, we wanted to keep a 21.7 version in to see how the new version is doing. 22.2 is pretty incomplete with pieces of the eval chopped out...

Based on this results it may be interesting to look at the games of arasan to see if there is something wrong with them.

Arasan had 26% after 154 games(probably 40/154) when it only played against Crafty in the first 154 games and it has 42% after 785 games.

I do not believe that arasan is strong enough to score more than 40% against programs like fruit and glaurung and it seems that something is wrong in your test(maybe you are wrong in adjudicating part of the games of arasan against other opponents).

Uri

Uri Blass wrote:

bob wrote:Thought I would move this since the other thread has gotten pretty long, as well as getting off-topic a bit. I have now fixed my referee program so that it can properly adjudicate games as won/lost/drawn. It maintains a board state (borrowed code from crafty but rewrote move generator to avoid the magic stuff to keep the code short. Since speed is not an issue, it just directly generates moves in bitboards rather than doing rotated lookups or magic stuff. I had to do this because not all programs provide accurate "Result" commands and can't be trusted. It can now ignore them and play anybody vs anybody. Only thing that is not allowed is draw offers. I had too many problems with programs not handling that correctly and decided to just disable the "offer draw" stuff and let the game continue. It is a forced draw if the 3-fold repetition is hit, or the 50 move rule, or insufficient material, regardless of whether one side claims a draw or not, which keeps the test games to a reasonable length.

I have run a partial test with the 6 programs so far. I am going to make 4 runs overnight, each opponent plays 160 games against every other opponent. And then I will do that 4 times to see how things look. I'm going to show the data two different ways, as the preliminary results are interesting. I will produce 4 sets of output from BayesElo where everybody plays everybody, and then 4 sets of output using only the Crafty vs everybody else PGN files, to see what happens. Remember that there are a couple of issues. One is what is the Elo spread from best to worst, and then what is the stability like. I should be able to post all of this in the morning if my shell script doesn't have a glaring error I missed.

More to follow...

Here are some partial results just for information: First batch is everybody vs everybody, second batch is just crafty vs each opponent...

Code: Select all

2291 game(s) loaded, 0 game(s) with unknown result ignored.
Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung 2-epsilon/5    73   19   19   763   61%   -15   18% 
   2 opponent-21.7           26   18   18   759   54%    -5   23% 
   3 Fruit 2.1                5   19   19   754   51%    -2   15% 
   4 Glaurung 1.1 SMP       -15   19   19   764   47%     2   16% 
   5 Crafty-22.2            -33   18   18   757   45%     6   22% 
   6 Arasan 10.0            -56   19   19   785   42%    11   10% 

760 game(s) loaded, 0 game(s) with unknown result ignored.
Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung 2-epsilon/5   115   44   42   153   69%   -31   17% 
   2 Fruit 2.1               64   42   41   149   63%   -31   18% 
   3 Glaurung 1.1 SMP        48   42   41   152   61%   -31   16% 
   4 opponent-21.7           20   38   37   152   58%   -31   41% 
   5 Crafty-22.2            -31   19   19   760   45%     6   22% 
   6 Arasan 10.0           -216   43   45   154   26%   -31   16% 

The two samples were made at the same time, for reference. Has a ways to go until it is done. I will then run this again, but with the "big" match. And will eventually stop running the all vs all, since once each non-crafty has played all the others, those versions do not change and there is little use in re-running them over and over. I will just take those PGN files and add them to the crafty vs everyone so that I just have to run crafty vs everyone after the first run to get all the PGN.

opponent 21.7 is crafty 21.7 for reference, we wanted to keep a 21.7 version in to see how the new version is doing. 22.2 is pretty incomplete with pieces of the eval chopped out...

Based on this results it may be interesting to look at the games of arasan to see if there is something wrong with them.

Arasan had 26% after 154 games(probably 40/154) when it only played against Crafty in the first 154 games and it has 42% after 785 games.

I do not believe that arasan is strong enough to score more than 40% against programs like fruit and glaurung and it seems that something is wrong in your test(maybe you are wrong in adjudicating part of the games of arasan against other opponents).

Uri

The adjudication is not wrong. I have played several thousand games and compared results to the output of Crafty's search. The rules are simple:

(1) if one side has no legal moves and is in check, he is checkmated and loses (loss).

(2) if one side has no legal moves and is not in check, he is stalemated (draw).

(3) if the position is repeated for the third time with the same side on move, it is a draw by repetition.

(4) if the position satisfies the 50-move rule, it is a draw.

(5) if the position has no pawns and only one minor piece per side, it is a draw, and yes I know there is an exception. I've never seen it happen in a real game and find it reasonable to say draw.

(6) if one side resigns, it is a loss.

I played several thousand test games between Crafty and the 5 opponents, I then grabbed the Result tag, and the last value from Crafty's search, and correlated them by hand. Since crafty is always +=good for white, it was easy to verify that the side with the evaluation edge always got the edge except for a few draws, in cases where Crafty thought it was worse but the opponent repeated or allowed a 50-move draw, for example.

Once I get a complete set of data, I'll put the whole thing on my ftp box if you want to look at the results for yourself. No program can end the game in any way except by resigning, which is an instant loss. Otherwise the referee detects the mates and such itself.

Wow! Your referee program is nearly as smart as WinBoard!

hgm wrote:Wow! Your referee program is nearly as smart as WinBoard!

It's smarter. It can run on a cluster without adding the overhead of remote board displays (xboard for unix). Where winboard can't even run on a cluster like this because there's no place to display a graphical board.

running 540 games at a time on the big cluster produces a _ton_ of network traffic to display 540 chess boards, animation on or not. This introduces a ton of random delays due to network congestion at fast time controls. That's why I wrote the thing in the first place.

OK, naturally all didn't go quite as expected. Here are the first four runs using the round-robin approach, which does stabilize the ratings somewhat better than previous tests:
the part that failed was the attempt to gather the above data for just crafty vs the world, leaving out the world-vs-world round-robin data so that we could compare old and new approaches across 4 runs. I will re-run the test with the fix included. But I am saving the PGN from the last run each time and that lets me provide this output for the last run above, which is just crafty vs the 5 opponents, without the opponents playing each other...
The first thing that obviously stands out is that the effective Elos are significantly different when only using C vs world games, rather than a full round-robin. However, I want to get all the data for 4 runs so am starting it again. These seem to take about an hour to run based on the timestamps given above, so I should have good data later this afternoon.


I received an email from a mathematician overnight that had some interesting insight into our discussions. Since they could be taken as a bit insulting to some posters in the discussion, I have invited him to come and participate directly. He has some insights into why "some" of the posters are a bit off-the-target with discussions about dependency and correlation, but I will save that for him to present if he takes me up on my invitation. The suggestions he proposes will certainly generate some discussion since they were not things that immediately jumped out to me as potential improvements...

bob wrote:

hgm wrote:Wow! Your referee program is nearly as smart as WinBoard!

It's smarter. It can run on a cluster without adding the overhead of remote board displays (xboard for unix). Where winboard can't even run on a cluster like this because there's no place to display a graphical board.

running 540 games at a time on the big cluster produces a _ton_ of network traffic to display 540 chess boards, animation on or not. This introduces a ton of random delays due to network congestion at fast time controls. That's why I wrote the thing in the first place.

I do not understand your confidence that you have no bugs in yout referee program.

Results clearly suggest that you have some bug and it will be possible to find out after we look at the pgn.

Uri

Uri Blass wrote:

bob wrote:

hgm wrote:Wow! Your referee program is nearly as smart as WinBoard!

It's smarter. It can run on a cluster without adding the overhead of remote board displays (xboard for unix). Where winboard can't even run on a cluster like this because there's no place to display a graphical board.

running 540 games at a time on the big cluster produces a _ton_ of network traffic to display 540 chess boards, animation on or not. This introduces a ton of random delays due to network congestion at fast time controls. That's why I wrote the thing in the first place.

I do not understand your confidence that you have no bugs in yout referee program.

Results clearly suggest that you have some bug and it will be possible to find out after we look at the pgn.

Uri

Did you read what I wrote. I played a couple of thousand games between Crafty and each possible opponent. I then correlated the Result tag from the PGN to the last search value Crafty produced. If the result was 1-0 and the score was > +8, then that passed. If the result was 0-1 and the score was < -9 then that passed. If the result was 1/2-1/2 and the score satisfied -.01 <= v <= .01, then that passed. A few failed, but were validated by hand. for example, Crafty was playing white against Glaurung2, and had a score of -.43, and we got a 3-fold repetition draw result. Crafty agreed that it was a repetition event hough it was not expecting it. Apparently Glaurung2 thought it was worse and went for the repeat, crafty thought it was worse and thought the opponent would not repeat. Result was acceptable.

That is why I an convinced the referee is not producing bogus results. I am in the process of making sure there are no issues in non-crafty games right now, as I did not check that. But then I thought that crafty can handle most any kind of move input. Coordinate (e2e4) or SAN (e4, Nf3). I am not sure about the other programs and am going to make sure that there are no time forfeits or complaints about illegal moves. Flags falling are not all that uncommon in 1+1 games (Crafty hardly ever loses one, but some older programs that I am not using have done this quite frequently) and so long as the flag falls because the program is thinking, it is OK. If it falls because the program though the last move was illegal and ignored it, that needs attention.

Here's a sample PGN for one game:
The "Site" tag identifies the node. The generic host name is in the form compute-rack-node where there are 4 racks (1-4) and 32 nodes per rack. we do have a couple of oddball nodes that are named compute-0-0 and compute-0-1. same exact configurations, but in the central rack where the head and IBRIX stuff is. The Date tag should be self-explanatory. I do think I am going to run each move thru Crafty's SAN output routine (which is in the referee) to get rid of that ugly d4d5 sort of move stuff, which will make the PGN easier for me personally to read.