Hello Ed:
Rebel wrote:I have a couple of questions about the LOS table at the CPW.
http://chessprogramming.wikispaces.com/LOS+Table
1. Are bigger tables available (2000-5000) games?
2. Or even better what's the formula of the table?
3. If a change you make is about a small estimated elo gain then at which LOS percentage do you decide to terminate a match? Arrived here there are 2 possibilities:
a) a negative match score. Say you have played 1000 games so far and you have a -17 score. The LOS table states a 74% reliability. Is that enough for you to abort the match and dump the change you made?
b) a positive match score. After 1000 games you have a +25 score indicating a reliable percentage of 83%. What do you decide, take the 83% for granted and count the change as an improvement or do you test further?
I do exactly what Lucas said: I calculate mean and standard deviation of a normal distribution using the match statistics (wins, draws and loses) and I calculate a probability.
Back in June I PM'ed Gerd Isenberg, who sent my PM to Edmund Moshammer: I calculated larger tables with this Fortran 95 code, assuming a draw ratio of 32%:
Code: Select all
program LOS_
implicit none
integer, parameter :: partitions = 200000
integer :: n, i, max, j
real(KIND=3) :: score, sigma, three_sqrt_of_two_pi, S1, S2, x, h, h2, k_for_LOS, LOS, wins
write(*,*)
write(*,'(A)') 'Number of games:'
write(*,*)
read(*,*) n
write(*,*)
if (n <= 0) then
write(*,'(A)') 'Incorrect number of games.'
write(*,*)
write(*,'(A)') 'Please close and try again. Press Enter to exit.'
read(*,'()')
stop
end if
max = int(3.24d0*sqrt(n + 0d0)) ! This limit avoids to calculate lots of LOS of the order of 100%, when n is big enough.
three_sqrt_of_two_pi = 3d0*sqrt(2d0*acos(-1d0))
open(unit=111, file='log.txt', status='unknown', action='write')
write(111,'(A,I6)') 'Number of games: ', n
write(111,'(A)') 'Draw ratio: 32% (fixed).'
write(111,*)
write(111,'(A)') 'Wins - loses: LOS:'
do j = 0, max
wins = 3.4d-1*n + 5d-1*j ! Draw ratio = 32%.
score = wins/n + 1.6d-1 ! Draw ratio = 32%.
sigma = sqrt((score*(1d0 - score) - 8d-2)/n)
! The formula for calculating sigma was taken from this thread (first seen in post #22):
! http://immortalchess.net/forum/showthread.php?t=2237
k_for_LOS = abs(5d-1 - score)/sigma
LOS = 0d0
h = k_for_LOS/partitions ! Start of the composite Simpson's rule for approximate erf function.
h2 = h + h
x = -h
S1 = 0d0
do i = 1, partitions-1, 2
x = x + h2
S1 = S1 + exp(-5d-1*x*x)
end do
x = 0d0
S2 = 0d0
do i = 2, partitions-2, 2
x = x + h2
S2 = S2 + exp(-5d-1*x*x)
end do
LOS = 5d-1 + h*(1d0 + 4d0*S1 + 2d0*S2 + exp(-5d-1*k_for_LOS*k_for_LOS))/three_sqrt_of_two_pi
write(111,'(A,I4,A,F6.2,A)') ' ', j, ' ', 1d-2*nint(1d4*LOS,KIND=3), ' %' ! j = wins - loses
end do
close(111)
write(*,'(A)') 'End of the calculations.'
write(*,*)
end program LOS_
So, answering to your questions:
1.- They were available for 2000, 5000, 10000, 20000, 50000 and 100000 games because I uploaded them in the PM, but this link expired in July. My method is an approximation which becomes closer with larger number of games, but with even 2000 games my numbers are clearly very close to Edmund's ones (Edmund is the author of that LOS table in CPW).
2.- I do the following:
Code: Select all
score = (wins + draws/2)/(wins + draws + loses)
sigma = sqrt{score*(1 - score) - draw_ratio/4]/(wins + draws + loses)}
k = (0.5 - score)/sigma
I calculate erf function from -infinity to k, then I call this number LOS. A look to the code can be good.
3.- I understand LOS as the probability of being wrong in the assumption of supposing an engine better than the other when in reality is just the opposite. This probability is min.(LOS, 1 - LOS), but of course it is an own thought that does not have to be right.
3.- a) 74.27% is for the engine with +17; the engine with -17 has a LOS value of 100 - 74.27 = 25.73% more less (more than 1/4 of probability of being wrong, which is a lot).
3.- b) The probability of being wrong is 100 - 83.11 = 16.89% more less, which still is very high IMHO. I like long tests, so I would continue a little more... but you see that I am not being rigurous here: it is only a guess.
------------------------
If you like, I can recompute these larger tables or even I can release a programme when you can fix the number of games and the draw ratio to get a notepad with the aspect of the LOS table hosted at CPW.
Just for curiosity, I computed and printed LOS values for 1000 games in a few seconds (I think less than ten) and I obtain LOS ~ 74.28% for +17 and LOS ~ 83.13% for +25; both values are very close to the official ones of CPW.
Regards from Spain.
Ajedrecista.