FGRL 60 sec + 0.6 sec Rating list - Komodo 11.2

[leavenfish]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls...
Mark

Lets keep it that way IMHO...it's why I chose to subscribe to Komodo. If it were too similar to say Stockfish, I would have no desire to purchase Komodo which I use for analysis.

[Lyudmil Tsvetkov]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

[lkaufman]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

We count some nodes that SF doesn't count, so our nps number is slightly inflated relative to SF. I think it's about 7%.

[mjlef]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

Komodo uses a form of Lazy Eval that helps speedup its node rate. I think the development versions of Stockfish have Lazy Eval, but the version I saw them using when I checked a month or so ago was less aggressive than what we do in Komodo. I cannot say exactly how much "bigger" Komodo's eval is, but the code is much larger, and it has a lot more terms. I am not claiming bigger is better though.

[Lyudmil Tsvetkov]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

Komodo uses a form of Lazy Eval that helps speedup its node rate. I think the development versions of Stockfish have Lazy Eval, but the version I saw them using when I checked a month or so ago was less aggressive than what we do in Komodo. I cannot say exactly how much "bigger" Komodo's eval is, but the code is much larger, and it has a lot more terms. I am not claiming bigger is better though.

larger in terms of code lines is something else, that could only possibly slow down the engine. (probably not, but still not make it much faster)

nps are comparable, +7% or so, as claimed by Larry, lazy eval supposedly kicks in in just a very limited range of conditions, so we can safely say nps for both engines are more or less the same. (and Houdini is comparable too)

so, in case you found some very special way of speeding up the engine by a factor of 2(1.2, 1.4, 1.8, whatever the ratio the size of Komodo eval is larger than SF one), Komodo and SF evals would seem to be very similar in size. (I have no doubt at all Komodo has very distinctive evaluation function as a whole, that is more than evident)

could you please confirm Komodo eval is at least 1.5 times larger than SF one?(I am not curious here, please don't get me wrong, I only guess this would be very interesting for the computer chess comunity)

I am almost convinced Komodo has somewhat more refined eval than SF, at least what would concern feeding in more sensible term values, but, according to my modest estimates, it could not possibly be 1.2/1.3 times larger than SF.

reasons:
- well, obviously, very similar nps
- it would be incredibly difficult for you, no matter the resources at your disposal, to tune an evaluation function that is exponentially larger than SF one; SF barely manage to tune their eval with all the framework resources, so how would you be able to tune twice bigger eval?
- from games of Komodo, I see it does not even scale down some opposite colour bishops, as well as some very simple one piece, 2Ps vs 1P endgames, still showing scores around +100cps; SF, on the other hand, has scaling functions for both

my point is that all of the current top engines(that would also include the top 10) basically do one and the same thing, with no particular emphasis on either search or evaluation, and to follow that approach, one could not possibly have a very extensive evaluation function.

but maybe I am wrong, and you have found a way of dealing with a significantly larger number of parameters, while leaving the mainstream construction of the engine intact.

again, my personal impression is that Komodo should have a bit larger and more refined eval, at least what concerns better tuned values, but not to the point that this would warrant the establishment of an entirely new engine paradigm.

[fastgm]

Done!
http://www.fastgm.de

[mjlef]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

Komodo uses a form of Lazy Eval that helps speedup its node rate. I think the development versions of Stockfish have Lazy Eval, but the version I saw them using when I checked a month or so ago was less aggressive than what we do in Komodo. I cannot say exactly how much "bigger" Komodo's eval is, but the code is much larger, and it has a lot more terms. I am not claiming bigger is better though.

larger in terms of code lines is something else, that could only possibly slow down the engine. (probably not, but still not make it much faster)

nps are comparable, +7% or so, as claimed by Larry, lazy eval supposedly kicks in in just a very limited range of conditions, so we can safely say nps for both engines are more or less the same. (and Houdini is comparable too)

so, in case you found some very special way of speeding up the engine by a factor of 2(1.2, 1.4, 1.8, whatever the ratio the size of Komodo eval is larger than SF one), Komodo and SF evals would seem to be very similar in size. (I have no doubt at all Komodo has very distinctive evaluation function as a whole, that is more than evident)

could you please confirm Komodo eval is at least 1.5 times larger than SF one?(I am not curious here, please don't get me wrong, I only guess this would be very interesting for the computer chess comunity)

I am almost convinced Komodo has somewhat more refined eval than SF, at least what would concern feeding in more sensible term values, but, according to my modest estimates, it could not possibly be 1.2/1.3 times larger than SF.

reasons:
- well, obviously, very similar nps
- it would be incredibly difficult for you, no matter the resources at your disposal, to tune an evaluation function that is exponentially larger than SF one; SF barely manage to tune their eval with all the framework resources, so how would you be able to tune twice bigger eval?
- from games of Komodo, I see it does not even scale down some opposite colour bishops, as well as some very simple one piece, 2Ps vs 1P endgames, still showing scores around +100cps; SF, on the other hand, has scaling functions for both

my point is that all of the current top engines(that would also include the top 10) basically do one and the same thing, with no particular emphasis on either search or evaluation, and to follow that approach, one could not possibly have a very extensive evaluation function.

but maybe I am wrong, and you have found a way of dealing with a significantly larger number of parameters, while leaving the mainstream construction of the engine intact.

again, my personal impression is that Komodo should have a bit larger and more refined eval, at least what concerns better tuned values, but not to the point that this would warrant the establishment of an entirely new engine paradigm.

First nps. I took Komodo 1912.00 (our current development version) and took out our "lazy eval". I ran the start position to depth 25 in both programs, and "lazy eval" Komodo is about 8% faster in nps. I do not know the speed difference for Stockfish's lazy eval, but I recall it only kicked in for some huge material differences. If someone has the before and after they added it, they could compare. Note Komodo's lazy eval is highly position dependent and can have a bigger nps, but I chose the opening since not a lot is happening and no one is way ahead (well, until we solve chess).

Measuring size of the evaluation is a bit more difficult, since some people have a programming style cramming things together, and others love lots of white space. But a visual comparison of Komodo eval code with Stockfish shows they are similarly formatted and have a similar amount of white space. Stockfish does have more comments which boosts its line counts a bit.

A standard method of measure code size is just count the line numbers. Generally, one line of code does a similar amount of work, and this removes the effect of very long variable names. So doing this, first for Stockfish 8:

material.cpp: 225 lines
evaluate.cpp: 927 lines
pawns.cpp: 291 lines
total 1443 lines

If I left out any files that you would consider to still be part of its eval, please add them in.

And now Komodo. Komodo's eval is in two files. The first, eval.cpp, is used to fill in tables used by the second file pop.cpp. Komodo actually compiles two versions of pop, one using the popcount instruction (hardware) and a second using a software version of popcount. This lets a single Komodo version run on more machines since it "switches" the eval as needed. Komodo uses a lot of popcount's in its evaluation which is one reason it has a bigger slowdown when compiles for 32 bits than other programs. Anyway, here are the total number of lines:

pop.cpp:2194 lines
eval.cpp: 2808 lines
total 5002 lines

So my "twice" estimate was off. It is closer to 3.5 times as many lines of code.

I do not understand why you claim "according to my modest estimates, it could not possibly be 1.2/1.3 times larger than SF". Efficient programs are written so the program bypasses as much code as it can. I am not claiming Komodo is super efficient, but it does try to avoid doing work when it is not going to measuring something anyway. Komodo also compensates by using several caches for evaluation components. It avoids doing multiplies when it can by using popcount and indexing into a table to count multiple pieces/pawns that get the same bonus/penalty. I am confident that Komodo does more work in each full eval than Stockfish. I am not claiming it is better, just more.

Note I did not count the appropriate .h files (Komodo just has function prototypes there and no real code. I did not look at the Stockfish .h files).

Komodo does scale down opposite color bishop endgames, but has different rules than other programs I have looked at. I would have to see the positions you refer to to comment on that.

As for two pawns vs 1, Komodo would scale up most endgames like that. Most are wins for the side with the extra pawn. I have not looked at Stockfish to see what it does so perhaps it has some rules for KPP vs KP which understand better which are draws. If you have a specific position in mind, I could take a look.

Mark

[Gusev]

Thank you very much!!

[Lyudmil Tsvetkov]

The changes we made should help elo at all time controls. We do not expect them to be specific to blitz time controls.

We do not know why Komodo does not do as well as some other programs at fast time controls. I think it is due to Komodo's extensive (and larger) evaluation, which slows our node rate down, but seems to help more at longer time controls. But if we can get enough elo gain, we will start looking better at these fast time controls. This is speculation.

I also think Komodo got a little lucky with the K 11.2 results reported here, and a little unlucky with some earlier results. Our reported gains at about 3 min + 1 sec were 24 elo. So I think once enough testing groups get more games we can get a more accurate estimate of elo gain at different time controls.

Mark

is Komodo evaluation really that much larger and more extensive than SF one?

because my impression is, looking at how engines handle different positions, that both engines have comparable evals.

by that much different, I would mean at least 2 times more terms present.

also, I don't see Komodo showing any different nps than SF, this would also hint at very similar extent of evals.

unless, your eval slows you down twice, but you have figured out how to speed up the engine the same amount.

I don't know, that is just, as you say, speculation.

Komodo uses a form of Lazy Eval that helps speedup its node rate. I think the development versions of Stockfish have Lazy Eval, but the version I saw them using when I checked a month or so ago was less aggressive than what we do in Komodo. I cannot say exactly how much "bigger" Komodo's eval is, but the code is much larger, and it has a lot more terms. I am not claiming bigger is better though.

larger in terms of code lines is something else, that could only possibly slow down the engine. (probably not, but still not make it much faster)

nps are comparable, +7% or so, as claimed by Larry, lazy eval supposedly kicks in in just a very limited range of conditions, so we can safely say nps for both engines are more or less the same. (and Houdini is comparable too)

so, in case you found some very special way of speeding up the engine by a factor of 2(1.2, 1.4, 1.8, whatever the ratio the size of Komodo eval is larger than SF one), Komodo and SF evals would seem to be very similar in size. (I have no doubt at all Komodo has very distinctive evaluation function as a whole, that is more than evident)

could you please confirm Komodo eval is at least 1.5 times larger than SF one?(I am not curious here, please don't get me wrong, I only guess this would be very interesting for the computer chess comunity)

I am almost convinced Komodo has somewhat more refined eval than SF, at least what would concern feeding in more sensible term values, but, according to my modest estimates, it could not possibly be 1.2/1.3 times larger than SF.

reasons:
- well, obviously, very similar nps
- it would be incredibly difficult for you, no matter the resources at your disposal, to tune an evaluation function that is exponentially larger than SF one; SF barely manage to tune their eval with all the framework resources, so how would you be able to tune twice bigger eval?
- from games of Komodo, I see it does not even scale down some opposite colour bishops, as well as some very simple one piece, 2Ps vs 1P endgames, still showing scores around +100cps; SF, on the other hand, has scaling functions for both

my point is that all of the current top engines(that would also include the top 10) basically do one and the same thing, with no particular emphasis on either search or evaluation, and to follow that approach, one could not possibly have a very extensive evaluation function.

but maybe I am wrong, and you have found a way of dealing with a significantly larger number of parameters, while leaving the mainstream construction of the engine intact.

again, my personal impression is that Komodo should have a bit larger and more refined eval, at least what concerns better tuned values, but not to the point that this would warrant the establishment of an entirely new engine paradigm.

First nps. I took Komodo 1912.00 (our current development version) and took out our "lazy eval". I ran the start position to depth 25 in both programs, and "lazy eval" Komodo is about 8% faster in nps. I do not know the speed difference for Stockfish's lazy eval, but I recall it only kicked in for some huge material differences. If someone has the before and after they added it, they could compare. Note Komodo's lazy eval is highly position dependent and can have a bigger nps, but I chose the opening since not a lot is happening and no one is way ahead (well, until we solve chess).

Measuring size of the evaluation is a bit more difficult, since some people have a programming style cramming things together, and others love lots of white space. But a visual comparison of Komodo eval code with Stockfish shows they are similarly formatted and have a similar amount of white space. Stockfish does have more comments which boosts its line counts a bit.

A standard method of measure code size is just count the line numbers. Generally, one line of code does a similar amount of work, and this removes the effect of very long variable names. So doing this, first for Stockfish 8:

material.cpp: 225 lines
evaluate.cpp: 927 lines
pawns.cpp: 291 lines
total 1443 lines

If I left out any files that you would consider to still be part of its eval, please add them in.

And now Komodo. Komodo's eval is in two files. The first, eval.cpp, is used to fill in tables used by the second file pop.cpp. Komodo actually compiles two versions of pop, one using the popcount instruction (hardware) and a second using a software version of popcount. This lets a single Komodo version run on more machines since it "switches" the eval as needed. Komodo uses a lot of popcount's in its evaluation which is one reason it has a bigger slowdown when compiles for 32 bits than other programs. Anyway, here are the total number of lines:

pop.cpp:2194 lines
eval.cpp: 2808 lines
total 5002 lines

So my "twice" estimate was off. It is closer to 3.5 times as many lines of code.

I do not understand why you claim "according to my modest estimates, it could not possibly be 1.2/1.3 times larger than SF". Efficient programs are written so the program bypasses as much code as it can. I am not claiming Komodo is super efficient, but it does try to avoid doing work when it is not going to measuring something anyway. Komodo also compensates by using several caches for evaluation components. It avoids doing multiplies when it can by using popcount and indexing into a table to count multiple pieces/pawns that get the same bonus/penalty. I am confident that Komodo does more work in each full eval than Stockfish. I am not claiming it is better, just more.

Note I did not count the appropriate .h files (Komodo just has function prototypes there and no real code. I did not look at the Stockfish .h files).

Komodo does scale down opposite color bishop endgames, but has different rules than other programs I have looked at. I would have to see the positions you refer to to comment on that.

As for two pawns vs 1, Komodo would scale up most endgames like that. Most are wins for the side with the extra pawn. I have not looked at Stockfish to see what it does so perhaps it has some rules for KPP vs KP which understand better which are draws. If you have a specific position in mind, I could take a look.

Mark

many thanks for the complete info, Mark!

engine followers and Komodo fans, please copy this useful information of Mark, and save it somewhere, as this might be the most the Komodo team has ever revealed about the inner workings of the engine.

first, about SF code, I guess you have left out the following files, SF developers might correct me:

- bitbase.cpp 181 lines
- endgame.cpp 827 lines
- psqt.cpp 127 lines
- types.h 450 lines

that makes 1505 additional lines in total.

so, 1505 + 1443 would make 2948 lines in total.

5002/2948=~1.6

so, higher than 1.3, but still lower than twice bigger.

if all white space and style are more or less the same, as you claim, then Komodo eval would just be larger, but not a significant amount to warrant a change of paradigm.

SF header files mostly contain declarations, too.

talking about PP vs P, I had in mind the general endgame, when configurations with more pieces are also taken into account, for example KRPP vs KRP, etc.

and Komodo, as far as I can see, does not seem to scale those, at least it reports close to a full pawn advantage with no passers present.

concerning opposite colour bishops, Larry himself claimed in the past Komodo does not scale them, as this would not work in the engine, but my personal impression while looking at the games is similar.

so, thanks again for the valuable info.

in conclusion, I don't know what to say: if our numbers are correct, then Komodo has larger eval, but not a shift in paradigm.

what do other people think, more familiar with SF code?

[Lyudmil Tsvetkov]

you mentioned also selective Komodo evaluation, it is true different terms will only kick in specific positions, but you still have to tune all of them within a single framework, and I have difficulties to imagine how one can tune couple of times more terms, but a factor of 1.6 is still manageable.

again, I can not explain to myself the practically almost identical nps...

anyone having a clue?