Seeking Stats Script/program

[mjlef]

This site has a nice simple ELO calculator, which shows ELO difference based on wins, losses and draws:

http://www.3dkingdoms.com/chess/elo.htm

Is there something similar that displays the probability of one program being stronger than another? It would take into account the game scores and total number of games? What I would like is something I can feed results to and it would state something like "Based on your game results, there is a 98% probability that program A is stronger than program B". If someone good in stats could decribe how to calculate this, maybe I will make it into a web page and host it. I am tired of guessing if a program change is meaningful or not!

Mark

[cwb]

This one might be useful:

http://www.ascotti.org/programming/ches ... ngines.htm

[mjlef]

Thanks. Very helpful

[jswaff]

Tom Likens (Djinn) gave me a very nice perl script a year or so ago. I don't know that I'm at liberty to hand it out, so you'll have to contact him if you're interested. But, here's a sample output: (Ok, don't laugh at the results.)

Code: Select all

james@smeagol ~/prophet/scripts $ ./pgn prophet-gnu505.pgn

==========================================================
           Total played: 40 (unique games: 40)
           Note, only unique games are scored.
==========================================================
          Player  Wins  Losses  Draws  Score    Percent
==========================================================
  GNU Chess 5.05   19     15      6   22.0/40  (55.00%)
         prophet   15     19      6   18.0/40  (45.00%)

ELO Diff: 34.86

--
James

[Guetti]

I can recommend Bayeselo from Remi Coulom.
To see a description just visit his site:

http://remi.coulom.free.fr/Bayesian-Elo/

[David Hotham]

Chi-squared test. Here's a nice site that does the sums for you: http://www.graphpad.com/quickcalcs/chisquared1.cfm

(Put categories: win, lose, draw. Observed values obviously per your results. Expected values should be equal numbers of wins and losses, adding up to the actual number of decisive games; and for the expected number of draws use the actual number of draws).

Note that this does not give you the probability that one engine is stronger than another. Rather it gives the probability of getting results like the results that you did get, if the two engines were of equal strength. If this probability is sufficiently small you may conclude that the engines are not of equal strength.