18 days from SF4 release and about ~30+ ELO gain!

[Ajedrecista]

Hello:

There are rating lists that show more than 56 elo for stockfish4 relative to stockfish3

CEGT 40/4 rating list shows 66 elo improvement so I think that the main reason is not that the 56 elo is too much but the fact that the IPON use slower time control.

http://www.husvankempen.de/nunn/40_4_Ra ... liste.html

18 Stockfish 4.0 x64 1CPU 3032 13 13 1700 66.6% 2911 41.2%

53 Stockfish 3.0 x64 1CPU 2966 12 12 2000 61.0% 2889 43.6%

Note also that the best stockfish in the CEGT 40/4 rating list is a different stockfish and 30 elo may not enough for stockfish developement version with 1 cpu to catch the first place there.

7 Stockfish 2.2.2 x64 4CPU 3081 13 13 1600 68.4% 2947 40.0%

I was not sure if CEGT reported larger gains, thanks for confirm it.

------------------------

The regression results inside the Stockfish testing framework are not directly comparable to any rating list due to the large draw rate.

Sure. Just as a side note I want to remember that the Elo difference of two engines is a function of wins - loses:

Code: Select all

W: wins/games.
L: loses/games:

(Elo difference) = 400*log{[1 + (W - L)]/[1 - (W - L)]}

Of course W - L is limited by the draw ratio D: in the extreme case of D = 1, W = L = 0.

------------------------

Thanks for your insights, Uri and Adam. They are helpful, as usual.

Regards from Spain.

Ajedrecista.

[Uri Blass]

Note that my idea that the commercial can buy the people who use the idea of course include giving them the source code.

Imagine that Robert Houdart decide to do a deal with 7 developers of stockfish and promise them at 1-3% of the profit from the next version of houdini dependent on their success when their job is simply to look at the source of houdini and suggest improvements when they agree not to share the source code with other people.

I guess that this deal is going to cost him less than the price of using a framework of 200 cores and I guess that he is going to have no problem to find people because I guess that a significant part of the people who work for stockfish for free are not going to be against getting some money from doing something similiar for houdini even if they cannot live from that money and it is probably going to be less than 100$ per month.

[mwyoung]

Robert why don't you start your own frame work like stockfish did I am sure there is people who would sign on. then there will be two great engines competing all the time.

The main advantage of the Stockfish testing framework is actually the number of people contributing ideas/code to be tested. One or two people can only produce a limited number of changes to test in a week. The amount of cpu resources being used is great, but of limited value if there were only a handful of changes to test per week/month.

I do not think that taking on a lot of contributors is feasible for a commercial engine.

I think that the main advantage is practically not the number of people but the number of machines that they can use when commercial programmers usually do not have 200 cores that work 24 hours per day for them.

Inspite of it I am not sure that stockfish is going to be developed faster.

Stockfish has also the disadvantage that it is open source so other can learn from it to improve their programs.

That is true, but if the release schedule stays at its current norm of a release about every 12 month give or take. The commercial programs will still have trouble keeping up.

It looks like we are seeing the dawn of a new paradigm, or Mr. Houdart will be correct. Because what we are talking about is money, and as of now the current marketing strategies and development of better chess engines will have to adapt or die.

Free markets abhor a vacuum, if there is money to be made in chess engine, someone will take up the challenge of the new paradigm with new innovations.

For example could Chessbase and its programmers with their resources do better, most likely yes. If there is enough return on the investment.

Or could Stockfish try and become a commercial program...

[Adam Hair]

I think that the main advantage is practically not the number of people but the number of machines that they can use when commercial programmers usually do not have 200 cores that work 24 hours per day for them.

Inspite of it I am not sure that stockfish is going to be developed faster.

Stockfish has also the disadvantage that it is open source so other can learn from it to improve their programs.

I think Adam is right. Do you really think that commercials can't build such framework?!
As for the rest, do you understand what it means being one step ahead?
P.S. I certainly won't miss Houdini.

I think that it is going to cost too much for the commercial to build the framework and use it.

It is clear that it is only a question of money and if the commercial are interested they can probably also buy most of the people who give ideas for stockfish and I think that it is going to cost them less money then the money that they need to use 200 cores for 24 hours per day.

Older 8 core Xeon servers are cheap and plentiful in the US on ebay. A commercial author based in the US could replicate the framework for under $5,000. I do not know if it can be done as cheaply in Europe, but the total cost is not prohibitive for a small business investment.

I really believe that the production of legitimate ideas is the limiting factor.

[Dr.Wael Deeb]

I think that the main advantage is practically not the number of people but the number of machines that they can use when commercial programmers usually do not have 200 cores that work 24 hours per day for them.

Inspite of it I am not sure that stockfish is going to be developed faster.

Stockfish has also the disadvantage that it is open source so other can learn from it to improve their programs.

I think Adam is right. Do you really think that commercials can't build such framework?!
As for the rest, do you understand what it means being one step ahead?
P.S. I certainly won't miss Houdini.

Great minds think alike

[Dr.Wael Deeb]

I think that the main advantage is practically not the number of people but the number of machines that they can use when commercial programmers usually do not have 200 cores that work 24 hours per day for them.

Inspite of it I am not sure that stockfish is going to be developed faster.

Stockfish has also the disadvantage that it is open source so other can learn from it to improve their programs.

I think Adam is right. Do you really think that commercials can't build such framework?!
As for the rest, do you understand what it means being one step ahead?
P.S. I certainly won't miss Houdini.

I think that it is going to cost too much for the commercial to build the framework and use it.

It is clear that it is only a question of money and if the commercial are interested they can probably also buy most of the people who give ideas for stockfish and I think that it is going to cost them less money then the money that they need to use 200 cores for 24 hours per day.

Older 8 core Xeon servers are cheap and plentiful in the US on ebay. A commercial author based in the US could replicate the framework for under $5,000. I do not know if it can be done as cheaply in Europe, but the total cost is not prohibitive for a small business investment.

I really believe that the production of legitimate ideas is the limiting factor.

My thoughts too....

The hardware issue shoudn't be a problem.......
Dr.D

[kranium]

I believe 'ideas' are fairly easy to come by, for ex:
bonus for rook behind passed pawn
penalty for undefended piece
etc.

The difficult part is the testing...
i.e. the truly beneficial ideas are found and verified thru extensive testing...
and the more processing power you have, the more ideas can be tested
(as result: more good ones can/may be identified and implemented faster)
this is where the Stockfish distributed testing network shines...(the sky is the limit and it's free!)

R. Houdart:
"New ideas or code changes are first validated by playing against the current development version of Houdini. If the outcome is promising, the new version plays a tournament against 7 or 9 different opponents. In both stages typically 10,000 to 50,000 games are played, depending on the results. I use 2 servers (16-core and 32-core) to play about 100,000 games per day, with each game taking about 20 to 30 seconds."
(see Martin Thoresen's interview: http://www.tcec-chess.net/viewtopic.php?f=17&t=154)

I think the systems mentioned above are beyond reach of most developers
(the cheapest modern 32 core IBM system I could find is $44,648.00)

A group of older systems is certainly better than nothing, but has the disadvantage of being very slow compared to today's modern systems
i.e. the same # of tests would take much much longer to complete

I believe Houdini's author may have a huge advantage in this regard...
he credits a wealthy Abu Dhabi businessman (with whom I'm personally acquainted) whose customers include IBM on his web page:
"The Houdini 3 development was greatly aided by the kind support of Mr. Ahmed Mansoor"

I commented on Ahmed's 48 core Ivy Bridge E2697, and asked him about the relationship between he and R. Houdart here:
http://talkchess.com/forum/viewtopic.ph ... &start=120
but could not get an answer

My 2 cents-
Norm

[mcostalba]

Indeed, the pace of Stockfish improvement is amazing, the development framework constructed by Gary is awesome.
Clearly no individual or two-person team can keep up with this in the long run, so this could mean the end of commercial chess engines as we currently know them. Maybe in 2 years time only Stockfish and derivates will continue to be developed.

Robert

Improvement cannot be foreseen in advance. It could be that next month we got 0 ELO, it happens and you know it.

But there is another side effect of open development that could be more threatening for commercial engines, a threat that was not foreseen in advance and that even I didn't realize it would be a problem. This is the obsolescence of release process: just few days after Stockfish 4 is out, almost all power users have dismissed it in favor of last nightly build (I just come here now from Playchess where there is even not one SF 4 but are all nightly builds): this is something commercial engines have no defense against, simply they cannot do this. As long as the ELO gap is big it is ok, but when the open developed engine reaches the level of commercials, a new compile each day can really badly affect the commercial release because it greatly speeds up its obsolescent.

I have to say that this was not foreseen and I am sorry for this, it is not out target (daily binaries are even built outside of SF team) but it is something that, considered the open nature of the development, it is almost impossible to avoid.

[Vinvin]

I think that the main advantage is practically not the number of people but the number of machines that they can use when commercial programmers usually do not have 200 cores that work 24 hours per day for them.

Inspite of it I am not sure that stockfish is going to be developed faster.

Stockfish has also the disadvantage that it is open source so other can learn from it to improve their programs.

I think Adam is right. Do you really think that commercials can't build such framework?!
As for the rest, do you understand what it means being one step ahead?
P.S. I certainly won't miss Houdini.

One step beyond
http://www.youtube.com/watch?v=N-uyWAe0NhQ

[syzygy]

Note that my idea that the commercial can buy the people who use the idea of course include giving them the source code.

It does not seem realistic to me to expect a commercial chess engine to generate a sufficient amount of money to pay for the time of all these people.

I'm afraid what you need is a government organisation like the NSA