Komodo 4 on long time control

[MM]

Larry,
What is your & Don's release cycle plans for Komodo?

Do you plan to follow the once-per-year model or the Rybka 2 model of a release every few months?

Personally, I think for serious chess players the Rybka 2 model was the best of all.

Hi, i don't know about the Rybka 2 model, but i can agree with you. For me, the best is an engine that releases updates or new versions each time there's an important improve (20/30 elo).

Regards

[lkaufman]

I like that analogy! Of course, we have a strong commercial incentive to overtake Houdini, but you are right, it goes beyond that. My main gripe about Houdart is that he has refused to acknowledge the well-established fact that the early Houdini versions were just some Ippolit version with minor modifications.

You are right of course but living in a post ICGA-Rybka world I can imagine he is careful. You won't get a heartwarming welcome if you admit you have taken the sources of someone else as a base. The opposite does not work also as proven. Typical -/- situation, Houdart whatever he does always loses. I think that's not fair and something must change.

Houdart made his initial release (the Ippo clone) BEFORE the ICGA-Rybka issue. Anyway if you release a program that is obviously almost 100% identical to another one, anyone with a disassembler can see this and to deny it just makes the person look dishonest. Others who started with Ippo and modified it freely so stated.

[Adam Hair]

Here is the list of the top programs and I see only little difference between CCRL 40/4 and CCRL 40/40

program are ranked here based on average rating between CCRL 40/40 and CCRL 40/4(the last numebr in every line is the standard rating advantage of the program and usaully the good programs have negative standard rating advantage.

1)Houdini 1.5a 64-bit 4CPU 3332.5(-63)
2)Critter 1.2 64-bit 4CPU 3282.5(-47)
3)Rybka 4.1 64-bit 4CPU 3281(-34)
4)Rybka 4 64-bit 4CPU 3274.5(-33)
5)Stockfish 2.1.1 64-bit 4CPU 3263.5(-65)
6)Houdini 1.5a 64-bit 3260(-8)
7)Stockfish 2.0.1 64-bit 4CPU 3251.5(-21)
8)Rybka 3 64-bit 4CPU 3245.5(-33)
9)Stockfish 1.9.1 64-bit 4CPU 3234.5(-27)
10)Stockfish 1.7.1 64-bit 4CP 3224(-24)
11)Stockfish 1.8 64-bit 4CPU 3223.5(-19)
12)Rybka4 64-bit 2CPU 3216(-2)
13)Critter0.90 64-bit 4CPU 3209.5(-29)
14)Rybka 4.1 64-bit 3208.5(+7)
15)Citter1.2 64-bit 3205.5(-5)
16)Komodo 3 64-bit 3202.5(+13)
17)Rybka 4 64-bit 3198(-2)
18)Rybka 3 64-bit 2CPU 3195(-46)
19)Naum4.2 64-bit 4CPU 3187(-14)
20)Stockfish 2.1.1 64-bit 3184.5(-9)

There are 257 engine versions that are represented 600+ games on both lists. The mean difference between the 40/40 list and the 40/4 list is +3.09 and the standard deviation is 23.3. Here is a graph of the data:



Using this information, I computed the z scores (the second column is the diiference from the average difference; the third column is the z score):

Code: Select all

Houdini 1.5a 64-bit 4CPU	   -66	-2.83
Critter 1.2 64-bit 4CPU	    -50	-2.15
Rybka 4.1 64-bit 4CPU	      -37	-1.59
Rybka 4 64-bit 4CPU	        -36   -1.55
Stockfish 2.1.1 64-bit 4CPU	-68	-2.92
Houdini 1.5a 64-bit	        -11	-0.47
Stockfish 2.1.1 64-bit 4CPU	-68	-2.92
Stockfish 2.0.1 64-bit 4CPU	-24	-1.03
Rybka 3 64-bit 4CPU	        -36	-1.55
Stockfish 1.9.1 64-bit 4CPU	-30	-1.29
Rybka 4 64-bit 2CPU	         -5	-0.21
Stockfish 1.8 64-bit 4CPU	  -22	-0.94
Stockfish 1.7.1 64-bit 4CPU	-27	-1.16
Rybka 4.1 64-bit	             4	 0.17
Komodo 3 64-bit	             10	 0.43
Critter 1.2 64-bit	          -8	-0.34
Rybka 4 64-bit	              -5	-0.21
Critter 0.90 64-bit 4CPU	   -32	-1.37
Stockfish 2.1.1 64-bit	     -12	-0.52
Naum 4.2 64-bit 4CPU	       -17	-0.73
Rybka 3 64-bit 2CPU	        -47	-2.02
Stockfish 1.7.1 64-bit 2CPU	 11	 0.47
Komodo 2.03 64-bit	           1	 0.04
Critter 1.01 64-bit	         -3	-0.13
Komodo 2.01 64-bit	          -4	-0.17
Naum 4.1 64-bit 4CPU	        -6	-0.26
Critter 0.90 64-bit	         12	 0.52
Critter 0.80 64-bit 4CPU	   -11	-0.47
Stockfish 1.9.1 64-bit	      -1	-0.04
Rybka 3 64-bit	             -28	-1.20
Naum 4 64-bit 4CPU	         -12	-0.52
Stockfish 2.0.1 64-bit	     -38	-1.63
Stockfish 1.6.3 64-bit 4CPU	-45	-1.93
Spike 1.4 Leiden 4CPU	       -3	-0.13
Stockfish 1.8 64-bit	       -31	-1.33
Deep Shredder 12 64-bit4CPU	-18	-0.77

It looks to me that the all of the top engines tend to have a negative difference (40/40 rating - 40/4 rating). I tend to think this is an artifact rising from the differences in the two lists (only one person contributes to both lists, though he does tend to test the top engines) and Bayeselo. Also, the draw rate for these engines at 40/40 is quite higher than at 40/4.

I don't think one could point to this and say that one engine will gain on another engine at even longer time controls, other than the fact that the draw rate will most likely increase will decrease the Elo gap.

[lkaufman]

Hi Don,

... That's why it's important to us that Komodo or for that matter Critter or Stockfish or some other legitimate program ends up on the rating lists on top....

Just as a side note I am less confident that this is such a good thing! Just keep in mind what happened to Houdini 2.0 with Strelka 5.0 just weeks after the release ... It is a sad thought, but I am less confident that being No.1 is that good if another unique code might be exposed some time later ... The consequences are spoiling the whole hobby to me!

Bye
Ingo

Well, since Houdini started as a copy of an open-source program, changes to it are obvious and easily copied. We won't have this problem. If we become number one it will be a big project for someone to decompile Komodo and make a new program in its image. So I regard the Strelka incident as a good thing, it shows that just modifying someone else's code has a downside.

[lkaufman]

Larry,
What is your & Don's release cycle plans for Komodo?

Do you plan to follow the once-per-year model or the Rybka 2 model of a release every few months?

Personally, I think for serious chess players the Rybka 2 model was the best of all.

I think we will release far more often than once a year, maybe something like the Rybka 2 model. My feeling is that we should make a release whenever we have a significant improvement, at least enough that no one will be able to claim that it was not an improvement.

[IWB]

Well, since Houdini started as a copy of an open-source program, changes to it are obvious and easily copied. We won't have this problem. If we become number one it will be a big project for someone to decompile Komodo and make a new program in its image. So I regard the Strelka incident as a good thing, it shows that just modifying someone else's code has a downside.

I doubt the explenation that this was so fast because of the probably known source of Houdini. I am a bit sceptic that this might be a cause of the speed in decompiling it. Maybe it doesn't need 4 weeks, but 4 monthes and then the mess is there again ... nonetheless I hope you are right because this would give back some fun to computerchess for me!

Regards
Ingo

[lkaufman]

[
I can't run a rapid/long match Komodo 64 bit SSE - Houdini 1.5 32bit.
Can someone else run it?

Thank you

Best Regards

Actually, I realized there is no great need to run this, because we can get the relative ratings at different time controls from the rating lists. Here is the data. Whenever both Houdini 2.0 and 1.5 appear on the same rating list in 32 bit, I averaged the ratings to get a bigger sample, since overall results for the two versions show hardly any difference (1 elo averaging the four lists cited). I also average Komodo 64 bit SSE and non-SSE when both appear on the same lists for the same reasons.

Here are the differences, Komodo 3 64 bit minus Houdini 64 bit:

40/4 (CEGT) : -35 (no CCRL data for H32 at this speed)
40/10(SWCR): -21
40/20 (CEGT): -15
40/40 (CCRL): +3

Notice the trend? It seems the point at which Komodo 3 pulls ahead appears to be at or just below 40/40.
Now the average gain for Houdini going from 32 bit to 64 bit on the four lists (comparing like version to like version) is 34.6 elo.

Conclusion: We must gain about 35 elo over Komodo 3 to pass Houdini at 40/40. We aren't quite there yet, but might be very soon.

[lkaufman]

I doubt the explenation that this was so fast because of the probably known source of Houdini. I am a bit sceptic that this might be a cause of the speed in decompiling it. Maybe it doesn't need 4 weeks, but 4 monthes and then the mess is there again ... nonetheless I hope you are right because this would give back some fun to computerchess for me!

Regards
Ingo

The reason I don't doubt this explanation is because Richard Vida reported that he was able to see all the improvements in Houdini 1.5 (many of which he listed on this forum) with very little work just by comparing to Ippo.

[MM]

[
I can't run a rapid/long match Komodo 64 bit SSE - Houdini 1.5 32bit.
Can someone else run it?

Thank you

Best Regards

Actually, I realized there is no great need to run this, because we can get the relative ratings at different time controls from the rating lists. Here is the data. Whenever both Houdini 2.0 and 1.5 appear on the same rating list in 32 bit, I averaged the ratings to get a bigger sample, since overall results for the two versions show hardly any difference (1 elo averaging the four lists cited). I also average Komodo 64 bit SSE and non-SSE when both appear on the same lists for the same reasons.

Here are the differences, Komodo 3 64 bit minus Houdini 64 bit:

40/4 (CEGT) : -35 (no CCRL data for H32 at this speed)
40/10(SWCR): -21
40/20 (CEGT): -15
40/40 (CCRL): +3

Notice the trend? It seems the point at which Komodo 3 pulls ahead appears to be at or just below 40/40.
Now the average gain for Houdini going from 32 bit to 64 bit on the four lists (comparing like version to like version) is 34.6 elo.

Conclusion: We must gain about 35 elo over Komodo 3 to pass Houdini at 40/40. We aren't quite there yet, but might be very soon.

So we come back again to my article: komodo will overtake (sooner or later) Houdini at ''tournament time control'' and/or anyway at long time control. Cannot be sure, but the trend is that.

Regards

[Houdini]

Hi Don,

... That's why it's important to us that Komodo or for that matter Critter or Stockfish or some other legitimate program ends up on the rating lists on top....

Just as a side note I am less confident that this is such a good thing! Just keep in mind what happened to Houdini 2.0 with Strelka 5.0 just weeks after the release ... It is a sad thought, but I am less confident that being No.1 is that good if another unique code might be exposed some time later ... The consequences are spoiling the whole hobby to me!

Bye
Ingo

Strelka 5 is Houdini 1.5 RE, not Houdini 2.
I share your fears, any #1 engine will be RE'd in a matter of months. For Houdini 1.5 it's taken about 6 months .

Robert