A very simple test I have done before is to just take 2 programs, test them at 3,4,5,6,7,8,9 and 10 ply (or some similar range) and then plot the ratings based on TIME, not depth. I have done this before and I can already tell you what you will see, 2 lines where the line of the modern program climbs much more rapidly. That's really what I am talking about.Laskos wrote:I got what you meant, but I am not sure that would be a fair comparison. If one pits a strong program at 1 second with a weak program at 2 minutes (to equal the strength), then at 10 seconds and 20 minutes (10x time control), to compare the difference, and we see that the stronger program improves more, that could mean many things, hash filling or optimization for certain size of the tree, etc.Don wrote:My comment applies to older programs that are a several hundred ELO weaker and have large branching factors compared to modern programs. I don't believe there is a substantial difference when comparing program that are just 3 or 4 years older or that are hundreds of ELO weaker. In such a case I would agree with you if you are not starting with equalized programs.Laskos wrote:Almost so, by comparing difference X between 2 engine at say 40/4 with difference Y between same engines at say 40/40. It's easy to do this having rating lists and a bit of time for a representative sample (errors are not so small in all these lists).Don wrote:Just so you understand what I'm saying, I assume that both program are starting from the same point on the ELO scale. Then you double them both and see which will benefit the most. Is that what you understood?Laskos wrote:That would mean that the difference in strength between modern programs and old ones would increase with the time control. I have to check the rating lists (no time now), but I doubt this statement.Don wrote: I'm almost positive that modern programs get a lot more ELO increase for doubling that the old program got.
Kai
Kai
A way to test such a thing is to start with an ancient version of Crafty for instance, and test it against Critter or Stockfish or Komodo. Add some time to Crafty and substract some time from the modern program to the extent that get a roughly even score against each other. Then increase the time for both programs by a constant factor and my hypothesis is that the modern program will be noticeably stronger.
If you compare what the old program does when doubled with what a modern program does when doubled it won't come out the same because the modern program may be starting out at 3000 and the old program may by starting out at 1800 and of course it's well known that doubling the time has a much larger impact on a program that is significantly weaker.
Don
I could be wrong, but intuitively doubling the time close to the optimum of the engine is the same for weak or strong engines as Elo gain goes. The ply was more important as Elo gain for weaker, older engines having high branching factor. I would rather say that modern engines gain less from each new ply as compared to old (weak) engines. Besides that, the Elo gain with ply is diminishing with depth in the case of the modern engines.
One has to test to see what really happens.
Kai
It's not fair testing a modern program going from 18 to 19 ply against an old program going from 3 or 4 ply because ANY program will gain a huge amount going from 3 to 4 compared to going from 18 to 19.
If you don't think the test is fair we can run the test where in the sweet spot range of the OLD program instead of the modern program and I think you will still see the phenomenon I am describing. You won't be able to make the point that the modern program was optimized for really high depths and the older one wasn't.
Don