I can only assume it is I who lack the proper understanding of how the ratings are calculated, but watching the IPON results of Critter 1.4, I began to wonder why its performance was 2978 after 2106 games. I took the 22 performances, added them up, and then divided them by 22 and came up with 3000.59, so why is the total performance 2978?
Scoring 77.3% means about 214 elo above opponents (this comes from ELO tables or formula, not lineal).
the average opponent ELO is (rounding) 2765 ELO, if we add them we get: 2765 + 214 = 2979 ELO
There is a discussion about this in a topic called "Komodo 4 at IPON results" in the tournament forum.
Critter 1.4 match seems to be stalled now. Last time I checked, score was 2978 after 2162 games but as I'm writing this, there is no more ad interim results displayed.
From what I observed from several IPON unfolding matches, for whatever reason the provisional score seems to drop after a few hundred games.
It would be ironic that Critter 1.4 score exactly matches Komodo 4 when the profile of play of both programs is very different (from this match, Critter being stronger with the strongest opponents but somewhat inconsistent with weaker ones).
Anyway, we are clearly in the diminishing return phase from the latest version for both programs.
At current rate of progress, we'll need Komodo 7 and Critter 1.6 to bypass Houdini 2/1.5
This demonstrates the engineering ability of Mr Houdart or the luck he had in tuning the parameters of Houdini 1.5
melajara wrote:Critter 1.4 match seems to be stalled now. Last time I checked, score was 2978 after 2162 games but as I'm writing this, there is no more ad interim results displayed.
From what I observed from several IPON unfolding matches, for whatever reason the provisional score seems to drop after a few hundred games.
It would be ironic that Critter 1.4 score exactly matches Komodo 4 when the profile of play of both programs is very different (from this match, Critter being stronger with the strongest opponents but somewhat inconsistent with weaker ones).
Anyway, we are clearly in the diminishing return phase from the latest version for both programs.
At current rate of progress, we'll need Komodo 7 and Critter 1.6 to bypass Houdini 2/1.5
This demonstrates the engineering ability of Mr Houdart or the luck he had in tuning the parameters of Houdini 1.5
I don't think so. Although IPON only shows a 14 elo gain for K4 over K3, CCRL so far shows 28 elo, so probably my original estimate of 20 was spot on. I guess we'll need two more versions to catch Houdini at blitz based on this, but the next version should do it at 40/40 minutes.
40 in 40 without resign = 160 minutes per game (move average without resign = 86 moves, with resign = 67 moves). Good statistics are only possible without resign.
For SWCR with 40 minutes per games I need with four machines 2 years to produced a rating list with many opponents. All of this opponents must be available in an actual version.
For a 40 in 40 rating list it need 2 years if an user have 48 cores and will give 320 EUR for electric current each month.
Biggest problem is to hold such a list actual. Most of the programmers released too many versions, most of the user await it for the most of people with working on a list not possible to test all.
With outer words, noboday can test it.
Possible only 40 in 40 games vs. a small group of engines. But this small group of engines must have a good rating too and all must be actual or a test is not interesting enough.
What the community can do is to compare:
40 in 3 with 40 in 5 with 40 in 10 for an example.
So we can see ... an engine is better with more time with games only.
Furthermore, for a good rating more opponents are more important as many games. Not to see with elo calculation programs. Easy to see with database simulations.
Best
Frank
For an example:
The still running SWCR Komodo 4 test with 1.560 games produced 15 EUR electric current = w32 and x64 = 30 EUR. So you can calculate how expensive it is to test so many versions of one Engine. I think in SWCR around 250 EUR in two years for Komodo only. Each test with around 1.500 games and 40 and 40 produced 60 EUR electric current.
For a good rating list with 40 in 40 it need around two years and you have to pay:
320 x 24 months = 7.680 EUR for electric current
6.000 EUR for hardware
1.000 EUR for hardware wearing
Many time, you have to write many mails to programmers with bug reports and you need a women with understanding because nobody will give you money back but all will have the results.
And of course the programmers will have money too. So the persons which working on such a list have to pay the programs too, but this is a very little problem if I thinking on all the other money.
melajara wrote:Critter 1.4 match seems to be stalled now. Last time I checked, score was 2978 after 2162 games but as I'm writing this, there is no more ad interim results displayed.
From what I observed from several IPON unfolding matches, for whatever reason the provisional score seems to drop after a few hundred games.
It would be ironic that Critter 1.4 score exactly matches Komodo 4 when the profile of play of both programs is very different (from this match, Critter being stronger with the strongest opponents but somewhat inconsistent with weaker ones).
Anyway, we are clearly in the diminishing return phase from the latest version for both programs.
At current rate of progress, we'll need Komodo 7 and Critter 1.6 to bypass Houdini 2/1.5
This demonstrates the engineering ability of Mr Houdart or the luck he had in tuning the parameters of Houdini 1.5
I don't think so. Although IPON only shows a 14 elo gain for K4 over K3, CCRL so far shows 28 elo, so probably my original estimate of 20 was spot on. I guess we'll need two more versions to catch Houdini at blitz based on this, but the next version should do it at 40/40 minutes.
Right Larry, but which Houdini version are you referring to when you say "catch Houdini". You seem to be in this period of time you refer to assuming Robert will be sitting on his hands doing nothing to come out with possibly a much stronger and faster release. Just a thought.
And again ...
I don't think that an engine will have an advantage with more as 30 ELO if you compare results 40 in 10 with 40 in 120. Can be easy see in SWCR Champions-League with 40 moves in 150 minutes, 40 moves in 150 minutes or with CEGT 40 in 120.
I think it's right that Komodo will get a little advantage if the program hold the positional playing strength. If not, and Komodo will get more tactic its possible that Komodo lost this little advantage to the others. This is clear if we study results of all the available rating list.
With other words:
A nice fairy tale in computer chess over so many years.
Have a look on IPON and SWCR. In SWCR I played with the double time. Compare the results and you can see ... only Junior is clearly stronger with more time. But here I am sure this advantage Junior have will be lost with more time as 40 in 10 because Junior is bad in endgames and more advantage with Junior's strong middlegame isn't possible.
So many fairy-tales in computer chess. To opening books, to ponder, to the time control and all other things, sources of programs ... own ideas, copy ideas and so one.
Frank Quisinsky wrote:And again ...
I don't think that an engine will have an advantage with more as 30 ELO if you compare results 40 in 10 with 40 in 120. Can be easy see in SWCR Champions-League with 40 moves in 150 minutes, 40 moves in 150 minutes or with CEGT 40 in 120.
I think it's right that Komodo will get a little advantage if the program hold the positional playing strength. If not, and Komodo will get more tactic its possible that Komodo lost this little advantage to the others. This is clear if we study results of all the available rating list.
With other words:
A nice fairy tale in computer chess over so many years.
Have a look on IPON and SWCR. In SWCR I played with the double time. Compare the results and you can see ... only Junior is clearly stronger with more time. But here I am sure this advantage Junior have will be lost with more time as 40 in 10 because Junior is bad in endgames and more advantage with Junior's strong middlegame isn't possible.
So many fairy-tales in computer chess. To opening books, to ponder, to the time control and all other things, sources of programs ... own ideas, copy ideas and so one.
Best
Frank
I do not see how you can be sure that Junior is going to lose the advantage with more time.
Only testing can answer it and if a program is bad in endgames it is possible that more time can help it to play the endgame better(not better than other programs but only good enough to prevent part of the mistakes that it does at fast time control).
