Comparative nodes per second

[lkaufman]

Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

[diep]

Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

Larry your assumptions are a tad off.

In Diep i'm not counting illegal positions after a check as it has incremental attacktables. So i kick that illegal move out already. Most engines aren't doing this, some are. Also i'm prematurely detecting repetitions. If i drop that condition the NPS already increases factor 2 at many occasions of Diep.

So Diep's nps is in majority of positions about a factor 2 lower than when i would count it in the same manner like others, just because it detects things very early - of course i can do that as i already blow a lot of system time to datastructure there - so early detection in the speedy programs would be a very dumb idea to think about. Yet a seemingly 'small' counting decision there can result easily in factor 2 nps difference.

Then further the engines you quote are doing different types of forward pruning. This really can hammer the nps down by factors. Stockfish is doing an agressive form of futility for example - i'm not sure Ivanhoe is doing that (didn't check ivanhoe). But this can really matter factors in nps. When i say factors i don't mean 20% or 30%. I really mean factor 2 or factor 3.

You just totally cannot compare nps-es in an easy manner.

Junior on other hand is *adding* the futile pruned moves to its nps. So that's why its nps seems high.

However if you really would neglect the forward pruning they do last few plies, most engines are not much above 1000 cycles a node. So that would mean for a StockFish it has the potential to reach at a 3.6Ghz i7 with sixcores somehwere around a 6 * 3.6 mln = roughly 20 mln nps.

Vincent

[mcostalba]

I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

[lkaufman]

I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

We count the node only when we make the move. If we can prune a move without making it, we don't count the node that would result from making the move. Are you saying that Ivanhoe counts nodes that would result from pruned moves that are never made? If so that would indeed inflate the node count. When Stockfish prunes moves by movecount or by futility are the resultant nodes counted or not?

I know that SF is a very fast program. That's why I want to clarify whether this 5 to 3 speed ratio is entirely due to counting issues or only partly so.

[tpetzke]

If you count in do_move then you count moves not nodes, of course they are somehow related because every move results in a new node but you don't count terminal nodes (where you only call eval, get a standpat or a hash hit with a cutoff). This should make a big difference.

Thomas...

[bob]

Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...

[lkaufman]

Comparing the two strongest open-source engines, Stockfish and Ivanhoe (recent versions), I notice that although they are virtually identical in playing strength (except at bullet levels), the reported nodes per second in Ivanhoe is higher in about the ratio of 5 to 3. I am well aware that node counts can differ depending on details of how they are counted (whether to count illegal moves for example), I don't believe that such details would amount to more than a few percent in the given instance. A ratio of 5 to 3 is HUGE, far more than could plausibly be attributed to differences in the complexity of the eval function (in the given instance at least). Obviously, since the programs are of equal strength, this huge disparity must have a high price or must somehow be misleading. Both programs are highly selective, though details differ, so I doubt this is the answer.

So, two questions:

1. How does Ivanhoe achieve such a huge lead in NPS over Stockfish?
2. What is Ivanhoe giving up to achieve this speed? It can't just be simpler eval, as the speed gap is too large. But Ivanhoe would be totally crushed by Stockfish on an equal-nodes basis. Why?

Have you compared to Crafty? I know exactly how I count nodes, which would give a starting point that doesn't require a lot of effort to figure out...

Since Crafty doesn't do the massive move-count based pruning that is used in all the top engines, I don't believe the node-counting in Crafty would be very relevant to node-counting in these engines, where the counting or not of these pruned moves is so critical.

[rvida]

I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.

[lkaufman]

I don't believe that such details would amount to more than a few percent in the given instance.

It's time to became a believer

SF updates nodes in do_move(), while Ivanhoe (and I guess also Komodo) updates nodes at the beginning of search(). This differs considerably because in case of null move search Ivanhoe counts 2 while SF counts 1. At the end the difference between updating in do_move() or in search() is of about 20-30% in nps if I don't remember wrong.


P.S: I really don't think Ivanhoe is faster than SF in comparable functions, actually I'd would not be surprised of the contrary (although impossible to prove): SF is really super tuned for speed, see for instance its perft performance.

Ivanhoe updates nodes in do_move(). But because the legality check is done only after do_move(), there are some illegal moves included in Ivanhoe's node count.

I think this issue accounts for just a few percent. Am I mistaken?

[BubbaTough]

I suspect much more time is spent in Stockfish eval than Ivanhoe. If I remember right, stockfish does not do lazy eval for example. It may even call eval every node (that eval is not cached) as well. The Ivan's try very hard to go fast by minimizing time spent in eval I think.

Almost everyone in this discussion have looked at both programs more than I have, so feel free to correct any mistaken impressions I might have.

-Sam