Comparative nodes per second

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
lkaufman
Posts: 3339
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Comparative nodes per second

Post by lkaufman » Tue Apr 10, 2012 7:49 pm

Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

diep
Posts: 1780
Joined: Thu Mar 09, 2006 10:54 pm
Location: The Netherlands
Contact:

Re: Comparative nodes per second

Post by diep » Tue Apr 10, 2012 8:16 pm

lkaufman wrote:Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?
Larry your assumptions are a tad off.

In Diep i'm not counting illegal positions after a check as it has incremental attacktables. So i kick that illegal move out already. Most engines aren't doing this, some are. Also i'm prematurely detecting repetitions. If i drop that condition the NPS already increases factor 2 at many occasions of Diep.

So Diep's nps is in majority of positions about a factor 2 lower than when i would count it in the same manner like others, just because it detects things very early - of course i can do that as i already blow a lot of system time to datastructure there - so early detection in the speedy programs would be a very dumb idea to think about. Yet a seemingly 'small' counting decision there can result easily in factor 2 nps difference.

Then further the engines you quote are doing different types of forward pruning. This really can hammer the nps down by factors. Stockfish is doing an agressive form of futility for example - i'm not sure Ivanhoe is doing that (didn't check ivanhoe). But this can really matter factors in nps. When i say factors i don't mean 20% or 30%. I really mean factor 2 or factor 3.

You just totally cannot compare nps-es in an easy manner.

Junior on other hand is *adding* the futile pruned moves to its nps. So that's why its nps seems high.

However if you really would neglect the forward pruning they do last few plies, most engines are not much above 1000 cycles a node. So that would mean for a StockFish it has the potential to reach at a 3.6Ghz i7 with sixcores somehwere around a 6 * 3.6 mln = roughly 20 mln nps.

Vincent

mcostalba
Posts: 2684
Joined: Sat Jun 14, 2008 7:17 pm

Re: Comparative nodes per second

Post by mcostalba » Tue Apr 10, 2012 8:41 pm

lkaufman wrote:I don't believe that such details would amount to more than a few percent in the given instance.
It's time to became a believer :-)

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

lkaufman
Posts: 3339
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Comparative nodes per second

Post by lkaufman » Tue Apr 10, 2012 9:02 pm

mcostalba wrote:
lkaufman wrote:I don't believe that such details would amount to more than a few percent in the given instance.
It's time to became a believer :-)

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.
We count the node only when we make the move. If we can prune a move without making it, we don't count the node that would result from making the move. Are you saying that Ivanhoe counts nodes that would result from pruned moves that are never made? If so that would indeed inflate the node count. When Stockfish prunes moves by movecount or by futility are the resultant nodes counted or not?

I know that SF is a very fast program. That's why I want to clarify whether this 5 to 3 speed ratio is entirely due to counting issues or only partly so.

tpetzke
Posts: 686
Joined: Thu Mar 03, 2011 3:57 pm
Location: Germany
Contact:

Re: Comparative nodes per second

Post by tpetzke » Tue Apr 10, 2012 9:30 pm

If you count in do_move then you count moves not nodes, of course they are somehow related because every move results in a new node but you don't count terminal nodes (where you only call eval, get a standpat or a hash hit with a cutoff). This should make a big difference.

Thomas...

bob
Posts: 20362
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Comparative nodes per second

Post by bob » Tue Apr 10, 2012 9:31 pm

lkaufman wrote:Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?
Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...

lkaufman
Posts: 3339
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Comparative nodes per second

Post by lkaufman » Tue Apr 10, 2012 9:35 pm

bob wrote:
lkaufman wrote:Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?
Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...
Since Crafty doesn't do the massive move-count based pruning that is used in all the top engines, I don't believe the node-counting in Crafty would be very relevant to node-counting in these engines, where the counting or not of these pruned moves is so critical.

User avatar
rvida
Posts: 481
Joined: Thu Apr 16, 2009 10:00 am
Location: Slovakia, EU

Re: Comparative nodes per second

Post by rvida » Tue Apr 10, 2012 10:15 pm

mcostalba wrote:
lkaufman wrote:I don't believe that such details would amount to more than a few percent in the given instance.
It's time to became a believer :-)

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.
Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.

lkaufman
Posts: 3339
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Comparative nodes per second

Post by lkaufman » Tue Apr 10, 2012 10:18 pm

rvida wrote:
mcostalba wrote:
lkaufman wrote:I don't believe that such details would amount to more than a few percent in the given instance.
It's time to became a believer :-)

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.
Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.
I think this issue accounts for just a few percent. Am I mistaken?

BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: Comparative nodes per second

Post by BubbaTough » Tue Apr 10, 2012 10:54 pm

I suspect much more time is spent in Stockfish eval than Ivanhoe. If I remember right, stockfish does not do lazy eval for example. It may even call eval every node (that eval is not cached) as well. The Ivan's try very hard to go fast by minimizing time spent in eval I think.

Almost everyone in this discussion have looked at both programs more than I have, so feel free to correct any mistaken impressions I might have.

-Sam

Post Reply