sprt tourney manager

[Guenther]

I did a quick test with amoeba vs. ZuriChess and to my surprise I ran into a timing problem.
According to the PGN and confirmed by the taskmanager it seemed that amoeba had a ratio of around 20:5 cpu time per core.
ZuriChess used much less time than the allowed 0.1s per move.
I guess you translate that 0.1s back to millisec integer values for sending?

Code: Select all

--time|-t <movetime>     time &#40;const seconds&#41; to play a move &#40;default 0.1s&#41;

[Event "Tourney"]
[Site "??"]
[Date "2017.3.3"]
[Round "1"]
[White "zurichess jura"]
[Black "Amoeba 2.2.w64"]
[Result "0-1"]
1. c4 c5 2. Nf3 Nf6 3. Nc3 Nc6 4. d4 cxd4 5. Nxd4 e6 6. Ndb5 d5 7. cxd5
{-0.10/7 0.019} Nxd5 {0.00/10 0.084} 8. e4 {0.52/8 0.015} Nxc3 {-0.02/11 0.095}
9. Qxd8+ {0.36/7 0.017} Kxd8 {-0.07/10 0.092} 10. Nxc3 {0.25/8 0.017} Bc5
{-0.17/10 0.095} 11. Bf4 {0.27/8 0.029} e5 {-0.21/9 0.096} 12. Bg3 {0.66/7
0.024} Bg4 {-0.29/8 0.067} 13. f3 {0.72/7 0.017} Be6 {-0.29/8 0.068} 14. Nd5
{0.30/6 0.016} f6 {-0.38/8 0.085} 15. Rc1 {0.22/6 0.018} Bd4 {-0.04/9 0.090}
16. Bf2 {0.37/6 0.017} Kd7 {-0.22/9 0.096} 17. Bd3 {0.75/6 0.021} Kd6 {0.08/9
0.095} 18. Ke2 {0.33/6 0.026} a6 {0.15/7 0.062} 19. Rhe1 {0.59/6 0.017} h6
{0.37/9 0.092} 20. Kf1 {0.79/5 0.020} Bxd5 {0.56/10 0.080} 21. exd5 {0.27/7
0.020} Nb4 {0.61/11 0.095} 22. Bf5 {0.86/8 0.017} Bxf2 {0.80/11 0.085} 23. Kxf2
{-0.08/9 0.025} Nxa2 {0.83/12 0.095} 24. Rc4 {-0.24/9 0.023} a5 {0.97/11 0.079}
25. Rd1 {-0.30/9 0.024} Nb4 {0.93/11 0.095} 26. Ke3 {-0.15/9 0.040} b5 {1.12/11
0.058} 27. Rc3 {0.03/7 0.014} g6 {1.25/11 0.090} 28. Be6 {-0.25/8 0.014} Ra7
{1.19/11 0.095} 29. h4 {-0.25/7 0.017} f5 {1.19/10 0.095} 30. f4 {-0.56/7
0.032} Re8 {1.14/10 0.095} 31. fxe5+ {-0.73/7 0.026} Kxe5 {1.14/10 0.096} 32.
Kf3 {-0.75/8 0.030} Rf8 {1.04/9 0.095} 33. Rc5 {-0.34/7 0.018} Rb8 {1.22/10
0.095} 34. h5 {-0.57/7 0.017} gxh5 {1.61/10 0.063} 35. Re1+ {-0.85/8 0.028} Kd6
{1.81/9 0.059} 36. Rc3 {-0.88/9 0.017} Rg7 {2.06/10 0.088} 37. g3 {-1.10/6
0.023} Rf8 {2.19/10 0.095} 38. Rd1 {-1.12/8 0.027} Rg4 {2.71/10 0.086} 39. Ra3
{-1.35/7 0.016} a4 {2.85/12 0.090} 40. Rc3 {-2.30/8 0.014} h4 {2.86/12 0.076}
41. gxh4 {-2.27/9 0.021} Rxh4 {2.99/13 0.095} 42. Rd2 {-2.39/8 0.020} Rc4
{3.01/13 0.095} 43. Rxc4 {-2.61/8 0.017} bxc4 {3.16/14 0.095} 44. Ke3 {-3.11/8
0.023} f4+ {3.40/12 0.095} 45. Kf3 {-3.51/9 0.020} a3 {3.23/13 0.095} 46. bxa3
{-2.09/9 0.018} c3 {6.66/14 0.095} 47. Rd4 {-2.53/8 0.014} c2 {7.32/14 0.072}
48. Rc4 {-6.50/9 0.020} Nd3 {7.62/14 0.073} 49. Ke4 {-7.64/8 0.019} c1=Q
{8.25/15 0.078} 50. Rxc1 {-8.66/10 0.031} Nxc1 {8.55/15 0.095} 51. Kf3 {-9.16/9
0.021} Nd3 {9.83/12 0.071} 52. Ke4 {-9.16/8 0.018} Ne5 {10.75/14 0.065} 53. Bh3
{-10.32/9 0.014} f3 {11.46/14 0.075} 54. Ke3 {-11.66/9 0.039} Ra8 {11.46/16
0.095} 55. Kf2 {-12.11/8 0.016} Rxa3 {11.64/13 0.066} 56. Bf1 {-12.65/10 0.048}
Ra2+ {11.65/13 0.059} 57. Ke3 {-12.71/10 0.023} Ng4+ {12.42/13 0.071} 58. Kf4
{-11.91/9 0.022} Nh2 {12.75/14 0.063} 59. Bd3 {-12.40/8 0.016} f2 {12.97/14
0.058} 60. Ke4 {-12.96/8 0.015} f1=Q {13.14/13 0.087} 61. Bxf1 {-13.48/9 0.017}
Nxf1 {13.28/12 0.078} 62. Kf4 {-13.57/8 0.052} Kxd5 {13.45/10 0.085} 63. Kg4
{-13.93/8 0.020} Ne3+ {13.56/10 0.095} 64. Kf3 {-14.12/8 0.018} Nf5 {13.61/12
0.065} 65. Kf4 {-13.95/9 0.019} Rf2+ {13.61/12 0.095} 66. Kg4 {-14.12/10 0.016}
Ke4 {13.62/12 0.095} 67. Kh5 {-14.96/8 0.020} Rh2+ {13.71/11 0.095} 68. Kg6
{-15.91/9 0.021} Rg2+ {21.95/11 0.064} 69. Kf6 {-28.17/11 0.025} Rg7 {+M17/15
0.095} 70. Ke6 {-28.17/11 0.017} h5 {+M11/9 0.095} 71. Kf6 {-M10/12 0.056} h4
{+M9/9 0.071} 72. Ke6 {-M8/11 0.017} h3 {+M7/7 0.000} 73. Kf6 {-M6/14 0.015} h2
{+M5/5 0.000} 74. Ke6 {-M4/30 0.014} h1=Q {+M3/3 0.000} 75. Kf6 {-M2/63 0.003}
Qh6# {+M1/1 0.000} {Black wins} 0-1

Guenther

[abulmo2]

I guess you translate that 0.1s back to millisec integer values for sending?

Yes. The uci engine should receive the command "go movetime 100". The floating point time in seconds is converted to milliseconds in the line 139 of the file https://github.com/abulmo/amoeba/blob/m ... /tourney.d

Code: Select all

	ms = cast (int) (1000 * t + 0.5);

I give a quick look at zurichess engine code, and I think there is a problem in the line 139¹ of the following file: https://bitbucket.org/zurichess/zuriche ... ew-default

Code: Select all

	// If there are still many moves to go, don't use all the time.
	tc.limit /= time.Duration(min(tc.MovesToGo, 5))

As I understand it, Zurichess allots the time to at least 5 moves... so it should be penalized when a GUI or a tournament manager sends the 'go movetime <ms>' command.

In a future version, I will try to use more common time control.

¹ no mistake, both important lines have the same number

[Guenther]

I guess you translate that 0.1s back to millisec integer values for sending?

Yes. The uci engine should receive the command "go movetime 100". The floating point time in seconds is converted to milliseconds in the line 139 of the file https://github.com/abulmo/amoeba/blob/m ... /tourney.d

Code: Select all

	ms = cast (int) (1000 * t + 0.5);

I give a quick look at zurichess engine code, and I think there is a problem in the line 139¹ of the following file: https://bitbucket.org/zurichess/zuriche ... ew-default

Code: Select all

	// If there are still many moves to go, don't use all the time.
	tc.limit /= time.Duration(min(tc.MovesToGo, 5))

As I understand it, Zurichess allots the time to at least 5 moves... so it should be penalized when a GUI or a tournament manager sends the 'go movetime <ms>' command.

In a future version, I will try to use more common time control.

¹ no mistake, both important lines have the same number :-)

Thanks for the quick reply. More usual time controls are welcome.

[Michel]

Thanks,

I am running some simulations (for the pentanomial model with opening bias). So far the evidence suggests that for a SPRT(0,5) (logistic elo bounds) alpha=beta=0.05, when using the original LLR approximation, the real alpha,beta are in the interval [0.502,0.498] which is extremely accurate. Note that I am using overshoot correction to account for the fact that the jumps in LLR are discrete (overshoot correction amounts to using slightly narrower bounds than the usual [-ln(19),ln(19)] obtained by the SPRT formula).

So it seems that the approximated LLRs can be used safely. I would still like to find a theoretical way to predict the real alpha,beta. Simulations are very costly to use on a routine basis.

[Michel]

I would still like to find a theoretical way to predict the real alpha,beta. Simulations are very costly to use on a routine basis.

It seems that "Non-linear renewal theory" is applicable. Chapter 9 in

http://www.springer.com/gp/book/9780387961347

[Laskos]

Thanks,

I am running some simulations (for the pentanomial model with opening bias). So far the evidence suggests that for a SPRT(0,5) (logistic elo bounds) alpha=beta=0.05, when using the original LLR approximation, the real alpha,beta are in the interval [0.502,0.498] which is extremely accurate. Note that I am using overshoot correction to account for the fact that the jumps in LLR are discrete (overshoot correction amounts to using slightly narrower bounds than the usual [-ln(19),ln(19)] obtained by the SPRT formula).

So it seems that the approximated LLRs can be used safely. I would still like to find a theoretical way to predict the real alpha,beta. Simulations are very costly to use on a routine basis.

I have to remember a bit, correct me if I am wrong. "Real" alpha,beta are Type I, II errors, right? These are very close to alpha,beta in this LLR approximation if you set elo=elo0 to determine Type I error, and elo=elo1 to determine Type II in simulations. That's what you do when getting [0.0502,0.0498]? For general elo, I remember I had an empirical result linking Type I and Type II errors to (alpha, beta, elo, elo0, elo1) as:

1/(1+exp{C*(elo-average[elo0,elo1])/(elo1-elo0)}),

where C depends on alpha,beta. It IIRC was valid for inside the interval too. It might be similar to what you want to get.

[Michel]

I meant "true" alpha, beta indeed in the sense of true Type I, II error probabilities.

It is not hard to see that asymptotically when using the approximated LLR as if it were the true LLR (which exists but we do not know it), we have Type I error=alpha, Type II error=beta. The question is how to obtain a non-asymptotic result.