bob wrote:Tord Romstad wrote:bob wrote:I use my cluster test results exclusively. No book issues or anything else, just plain and simple "engine vs engine." Already found one serious timing bug that will influence rating lists that use "repeating" time controls that I never use or test with (for example, 40/60 repeating, 40 moves in 60 minutes, then repeat. Gross error in time usage in that.
I prefer fischer-clock games to avoid time scrambles also.
So do I.
This is a danger we are all facing: We are always optimizing for our own favorite testing conditions, and there is always a risk that the strength will suffer when somebody else tests the program under different conditions.
For people with limited testing resources, like us in the Stockfish team, the public rating lists are invaluable. I understand why it is not as interesting to you with your ability to get thousands of games very quickly on your cluster, but I still think it would be useful for you to have a look at the public rating lists once in a while, in order to to see how Crafty performs under different conditions and discover bugs like the one described above.
I simply don't pay much attention. If we were to reach a point where we have no ideas to test, I might jump into trying to look at how others are testing and see if there is anything that needs addressing. I did not know we had the repeating time-control bug until Frank pointed it out in a post here a week or two ago.
And if you had paid more attention to the public rating lists, there are good chances that you would have discovered this bug much earlier. That was my point. By ignoring the public rating list, you miss the chance to easily discover bugs that don't appear in the particular testing conditions you use.
Another point is that most users these days are not aware of how awesome Crafty is, because it's so vastly underrated on the public lists. I know you don't care much about what the average computer chess enthusiast thinks about Crafty, but I and your other fans do care. We want Crafty to get the recognition it deserves.
Your partner, MC, commented that he could not understand why G2.x is ahead of Crafty in some lists since it has been out for a year or two. Here's a newsflash for him: I don't spend time going thru other programs. Again, if and when I run out of ideas, I may well start to look around. The TT approach to a _very_ weak singular extension idea is not particularly new. There are a couple of other "almost-singular-extension" ideas as well. And then there is the real-mccoy as defined by Hsu. You might notice that the TT-singular idea is not in Crafty, while in the past there have been other "sorta-singular-extension" ideas a couple of times. What's the moral? I don't spend any time digging through other programs to try and copy ideas. At least not until my "to-do" list has been emptied, and there is a long list of things remaining, including re-testing the old SE stuff I did several years ago, which might work better in light of the rather ridiculous depths we see today with LMR amd forward pruning working together so well.
I'm not sure what any of this has to do with what I wrote, but we haven't claimed anywhere in the thread that our singular extension implementation is new, unique to Stockfish, nor our own, original idea. Joona just listed it as one of several factors that make Stockfish strong. It wasn't an example of how we are somehow better chess programmers than you. We are not, and we never claimed to be.
I've said it previously in the thread, but because people evidently didn't listen, I'll say it again: Please don't let this thread degenerate into a Crafty vs Stockfish flamewar! Reading the CCC is depressing enough in the first place, but a thread where
the good guys start fighting about nothing without any intervention of the trolls is just too much. We're not rivals; we're on the same side!
And by the way, I don't spend any time digging through other programs to try and copy ideas either. It's been a very long time since I last had a look at the source code of another program.
Let's see if MC hangs around doing this for 20 or 30 or 40 years. If so, then he can talk. I do what I enjoy, and no more. And that has kept me involved in this for now 42 years come October. For me, it is more important to enjoy this as opposed to worrying about how high up or down a list I am. Too many have burned out and fell by the wayside following that pursuit.
I certainly won't hang around for 20 years or more. Like you, I don't worry about how high up or down a list I am, but the lists serve as a useful way to measure progress. It doesn't matter much whether Stockfish is number 1 or number 20 on the list, but it matters that version X+1 of Stockfish is stronger than version X.