How do I compare Engines?

[dangi12012]

Given two engines:
https://1drv.ms/u/s!AoDIyNlDG2QTg8sNLsR ... Q?e=onQOO4

Could somebody calculate the difference and report back?
How is this done generally with the fastest result and minimal standard deviation?

[dangi12012]

Update: No need for exact ELO difference.

Engine A is 10% slower than Engine B.
Meaning that crossing a pinvoke boundary in C# inccurs a higher cost than having optimized inline C# code - for single slider lookups.
That verifies that mixing languages only makes sense when offloading more than a single function call and one language is significantly slower than the other (which it still true for C#).

Even tho most of the time is spent in recusing through EvaluateTT a performance difference of 10% is significant and will yield a measurable elo difference.

[AndrewGrant]

Big sample size of one, running your program through another program.

I would suggest looking for a better way to compare the speeds.

[lithander]

Even tho most of the time is spent in recusing through EvaluateTT a performance difference of 10% is significant and will yield a measurable elo difference.

Oh, you use Leorik! Then I have a few tips: There's a project called Leorik.Test and if you run that it will spend the same time budget (e.g. 100ms) on 300 different positions. Afterwards it will print the average NPS. I use that as a benchmark for changes that should not affect anything but the speed. You can run it from Visual Studio and the results are consistent enough but it's usually a few thousand NPS faster when you make a build.

Only when you change the search/eval of a real engine-vs-engine tournament would be necessary. I use cutechess-cli for that.

If you want to profile the move generation code in isolation I recommend the Leorik.Perft project. It's just move generation and nothing else and instead of the 6M nps I get in the engine it's more like 60M nps there. The performance implications of using PInvoke for sliders should be much more visible there!