Programming issue: engine speed slowing over time

[Chessnut1071]

My objective is finding checkmate in the least number of moves and fastest possible time. My engine works fine clocking about 250,00 - 300,000 legal nodes per second when first started; however, in deep ply solutions, greater than 18-ply, I'm experiencing signification slowdown in speed, as much as 98% at 20-ply. Since some of my solutions can run for hours and even days, speed is important. I'm trying to find the cause of this slowdown and I suspect it's related to the garbage collector and memory in .NET. After searching the internet for solutions it seems that 100s of developers have the same issue and all for different reasons depending on their specific application. I'm using an asynchronous task off the main thread - the command button which starts the engine. Can anyone suggest a workable solution to this issue? faster is better!

[lithander]

Does the memory used by your process increase over time?

It should allocated roughly 10MB + whatever your TT size is and not grow any further over time. If it does continuously grow there's a memory leak that the garbage collector can't fix because the objects are still referenced in some way. Find out how and sever that lifeline.

[Chessnut1071]

Does the memory used by your process increase over time?

It should allocated roughly 10MB + whatever your TT size is and not grow any further over time. If it does continuously grow there's a memory leak that the garbage collector can't fix because the objects are still referenced in some way. Find out how and sever that lifeline.

the memory use stays relatively constant at 61% and CPU runs at 90 to 100%. I don't think it's the garbage collector.

[Chessnut1071]

Does the memory used by your process increase over time?

It should allocated roughly 10MB + whatever your TT size is and not grow any further over time. If it does continuously grow there's a memory leak that the garbage collector can't fix because the objects are still referenced in some way. Find out how and sever that lifeline.

the memory use stays relatively constant at 61% and CPU runs at 90 to 100%. I don't think it's the garbage collector.

It seems like memory is constant with maybe 2% flux; however, speed keeps declining exponentially with cpu rate around 83 - 92%. I wonder if this is an issue with Windows instead.

[hgm]

If it were an issue with Windows it would also happen to other engines. Have you tried that?

What do you mean by 'speed declining exponentially'? You see the nps drop to half the previous value in how many second? If nps decreases exponentially with time, it would mean you can never get beyond a certain total number of nodes. What is that number in your case? How does it relate to your hash-table size? What replacement algorithm do you use for hashing?

One design flaw that would cause this behavior is attemping lossless hashing (i.e. never overwrite something, but rehash until you hit an empty slot). Then you can never have more unique positions in your tree than the size of your hash table allows.

[flok]

Could it be that your cpu is overheating and that it then clocks down?

[Chessnut1071]

If it were an issue with Windows it would also happen to other engines. Have you tried that?

What do you mean by 'speed declining exponentially'? You see the nps drop to half the previous value in how many second? If nps decreases exponentially with time, it would mean you can never get beyond a certain total number of nodes. What is that number in your case? How does it relate to your hash-table size? What replacement algorithm do you use for hashing?

One design flaw that would cause this behavior is attemping lossless hashing (i.e. never overwrite something, but rehash until you hit an empty slot). Then you can never have more unique positions in your tree than the size of your hash table allows.

After documenting the time and engine calls it seems that the engine speed is approximately constant, 190,00 - 230,00 calls per second, until it gets to approximately 74,000,000 engine calls. Then, the speed drops off a cliff to about 1% of the starting speed. I reduced the size of my hash and the engine is only using 48% of the memory, but, it's using over 90% of the cpu. Reducing the size of the hash had the effect of increasing the number of engine calls, probably from increased collisions. Just wondering if it's an issue with memory being swapped to disk after 74,000,000 calls, which would account for the extremely slow speed.

[Chessnut1071]

Could it be that your cpu is overheating and that it then clocks down?

not likely with an HP Envy

[Ras]

Just wondering if it's an issue with memory being swapped to disk after 74,000,000 calls, which would account for the extremely slow speed.

Whatever an "engine call" is supposed to be - if you keep making up nonsense terms, don't be surprised if nobody understands what you mean. Anyway, your OS should have a way to tell you how much swap space is in use.

[Chessnut1071]

If it were an issue with Windows it would also happen to other engines. Have you tried that?

What do you mean by 'speed declining exponentially'? You see the nps drop to half the previous value in how many second? If nps decreases exponentially with time, it would mean you can never get beyond a certain total number of nodes. What is that number in your case? How does it relate to your hash-table size? What replacement algorithm do you use for hashing?

One design flaw that would cause this behavior is attemping lossless hashing (i.e. never overwrite something, but rehash until you hit an empty slot). Then you can never have more unique positions in your tree than the size of your hash table allows.

Something there is about deep ply searches that is eluding me. The search speed doesn't seem to be affected on searches under 16-ply. The search speed is fairly steady around 220,000 legal engine calls, nodes, per second, and it reaches well over 1,000,000,000 nodes without issue. After 18-ply, the engine runs for about 6 minutes, or 70,000,000 nodes before the speed is reduced by 98.5%. I remember in older software the stack was limited to 12 and had to be reset to enlarge it. I'm using 24 for() loops for 24-ply and they probably require 24 stack elements for the loops and at least another six for the method calls from the evaluation. If you're familiar with C# Net 6.0 is there some limit on the stack length I'm missing. I don't think this an issue with C++. I'm thinking about changing the for loops to goto indexs to get rid of the stack requirements from the for loops. Are you using for loops on your deep ply searches?