bob wrote:The right way to measure SMP performance is to take a large set of positions, and search with 1, 2, 4, ..., N processors, and measure the time required to reach a specific depth for each position.
Hi Bob,
thanks for the answer. I took 10 positions to test it that was - so you think this is a too great error margin? How large must it be for around 10% error margin?
regards
What are you trying to measure? SMP speedup? I usually run 2-300 positions, and for a good test I use about 1200 or so.
If you are trying to see how SMP helps a program in terms of winning more games, I believe you need thousands... You can play the same starting position many times, as I use a set of 40 starting positions for my testing. But you need to run them many times to get any sort of accurate reading.
Werner wrote:Hi,
I took 10 positions from WM-Test and had following results:
1->2CPU: 142% and
2->4CPU: 146% average
What does this mean?
For example, if it takes 2 minutes to solve N positions, and it take 6 minutes to solve the other N, than a 2 minute search with 2x the processors is going to have no effect on the results.
The right way to measure SMP performance is to take a large set of positions, and search with 1, 2, 4, ..., N processors, and measure the time required to reach a specific depth for each position.
A _much_ less accurate way is to play a one cpu program against a bunch of opponents, then keep everything the same but use two cpus for that program and see if it performs better. It is much less accurate because the variance/randomness in a basic game of chess is extremely high. And when you factor in the randomness produced by a parallel search, it goes even higher. You need thousands of games (at a bare minimum) to get a reasonable estimate on improvement. Most can't pull that off.
You assume that depth means the same with 1 processor and more processors.
It is not obvious.
For example it may be possible that the program does less late move reductions with more processors.
Tony Thomas wrote:Any chances of a public release?? Homer like the previous Zappa (prior to Zappa Mexico II) struggles in blitz. It could be because you did not care about being good in fast time controls or that Homer's algorithms are ultra optimized for tournament time controls. Anthony was able to get his engine on par with the other major opponents after he spend few days on optimizing, I hope you will do the same.
Well, ok, there will be a new Homer version.
I promise it will be a > 2700 ELO engine. I never gone for elo hunting up to homer 2.x. Now its time to be a hunter. Checkout my homepage and enjoy....
Werner wrote:Hi,
I took 10 positions from WM-Test and had following results:
1->2CPU: 142% and
2->4CPU: 146% average
What does this mean?
For example, if it takes 2 minutes to solve N positions, and it take 6 minutes to solve the other N, than a 2 minute search with 2x the processors is going to have no effect on the results.
The right way to measure SMP performance is to take a large set of positions, and search with 1, 2, 4, ..., N processors, and measure the time required to reach a specific depth for each position.
A _much_ less accurate way is to play a one cpu program against a bunch of opponents, then keep everything the same but use two cpus for that program and see if it performs better. It is much less accurate because the variance/randomness in a basic game of chess is extremely high. And when you factor in the randomness produced by a parallel search, it goes even higher. You need thousands of games (at a bare minimum) to get a reasonable estimate on improvement. Most can't pull that off.
You assume that depth means the same with 1 processor and more processors.
It is not obvious.
For example it may be possible that the program does less late move reductions with more processors.
Uri
I don't see why. But if you search the _same_ depth, why would there be any difference with respect to number of processors used? That gives an accurate SMP speedup number. Which is a different number than what you get if you try the same program against the same opponents, but you vary the number of processors. now, rather than measuring exact speedup, you are measuring the effect of searching faster with respect to Elo improvement. Which is also an interesting number, but is independent of the parallel speedup as it is generally defined.
Daniel Mehrmann wrote:
Well, ok, there will be a new Homer version.
I promise it will be a > 2700 ELO engine. I never gone for elo hunting up to homer 2.x. Now its time to be a hunter. Checkout my homepage and enjoy....
Hi Daniel,
hurry up, there is a great danger coming up in autumn
I'll make sure the Hustle 6 engine is still on the way, but the release of the ultimate engine has to be postponed until this autumn due to my lacking of time.
Werner wrote:Hi,
I took 10 positions from WM-Test and had following results:
1->2CPU: 142% and
2->4CPU: 146% average
What does this mean?
For example, if it takes 2 minutes to solve N positions, and it take 6 minutes to solve the other N, than a 2 minute search with 2x the processors is going to have no effect on the results.
The right way to measure SMP performance is to take a large set of positions, and search with 1, 2, 4, ..., N processors, and measure the time required to reach a specific depth for each position.
A _much_ less accurate way is to play a one cpu program against a bunch of opponents, then keep everything the same but use two cpus for that program and see if it performs better. It is much less accurate because the variance/randomness in a basic game of chess is extremely high. And when you factor in the randomness produced by a parallel search, it goes even higher. You need thousands of games (at a bare minimum) to get a reasonable estimate on improvement. Most can't pull that off.
You assume that depth means the same with 1 processor and more processors.
It is not obvious.
For example it may be possible that the program does less late move reductions with more processors.
Uri
I don't see why. But if you search the _same_ depth, why would there be any difference with respect to number of processors used? That gives an accurate SMP speedup number. Which is a different number than what you get if you try the same program against the same opponents, but you vary the number of processors. now, rather than measuring exact speedup, you are measuring the effect of searching faster with respect to Elo improvement. Which is also an interesting number, but is independent of the parallel speedup as it is generally defined.
My poiint is that parallel search may change the tree so the same depth may not be equivalent.
if your late move reduction has a rule not to reduce the first 3 moves then with parallel search you may get different moves as the first 3 moves.
Tony Thomas wrote:Any chances of a public release?? Homer like the previous Zappa (prior to Zappa Mexico II) struggles in blitz. It could be because you did not care about being good in fast time controls or that Homer's algorithms are ultra optimized for tournament time controls. Anthony was able to get his engine on par with the other major opponents after he spend few days on optimizing, I hope you will do the same.
Well, ok, there will be a new Homer version.
I promise it will be a > 2700 ELO engine. I never gone for elo hunting up to homer 2.x. Now its time to be a hunter. Checkout my homepage and enjoy....
Tony Thomas wrote:Any chances of a public release?? Homer like the previous Zappa (prior to Zappa Mexico II) struggles in blitz. It could be because you did not care about being good in fast time controls or that Homer's algorithms are ultra optimized for tournament time controls. Anthony was able to get his engine on par with the other major opponents after he spend few days on optimizing, I hope you will do the same.
Well, ok, there will be a new Homer version.
I promise it will be a > 2700 ELO engine. I never gone for elo hunting up to homer 2.x. Now its time to be a hunter. Checkout my homepage and enjoy....
Then I too will be forced to become an ELO hunter.
Then Homer will find the new Romi, even less friendly.
Just kidding! hehe
If you are on a sidewalk and the covid goes beep beep
Just step aside or you might have a bit of heat
Covid covid runs through the town all day
Can the people ever change their ways
Sherwin the covid's after you
Sherwin if it catches you you're through
Tony Thomas wrote:Any chances of a public release?? Homer like the previous Zappa (prior to Zappa Mexico II) struggles in blitz. It could be because you did not care about being good in fast time controls or that Homer's algorithms are ultra optimized for tournament time controls. Anthony was able to get his engine on par with the other major opponents after he spend few days on optimizing, I hope you will do the same.
Well, ok, there will be a new Homer version.
I promise it will be a > 2700 ELO engine. I never gone for elo hunting up to homer 2.x. Now its time to be a hunter. Checkout my homepage and enjoy....
Then I too will be forced to become an ELO hunter.
Then Homer will find the new Romi, even less friendly.
Just kidding! hehe
Only if you can manage the source of two engines at the same time. Why not continue programming on the current Romi? You can also release a goat powered Romi as well.
No Daniel, don't do it. Don't abandon the cause! 
