TalkChess.com

In this case there is just a single opponent - it's version A vs version B. The "restart" issue you mentioned is interesting. This is my Java engine, and I do _not_restart it between games due to the "warm up overhead" of starting a JVM. My first thought was that there must be some sort of resource contention. Based on other responses I think the most likely possibility at this point is that I underestimated the importance of using the same set of starting positions. (I really hope that is the case.) It will take me several days to confirm.

Might not be related at all, but in the early days of Leela chess I got very different results with and without restarting, so now I only test with restart, FWIW.

If you don't want to restart the JVM, at least make sure and transposition and related tables are cleared before each game.