non overlapping error bars

[jswaff]

In this case there is just a single opponent - it's version A vs version B. The "restart" issue you mentioned is interesting. This is my Java engine, and I do _not_restart it between games due to the "warm up overhead" of starting a JVM. My first thought was that there must be some sort of resource contention. Based on other responses I think the most likely possibility at this point is that I underestimated the importance of using the same set of starting positions. (I really hope that is the case.) It will take me several days to confirm.

[brianr]

Might not be related at all, but in the early days of Leela chess I got very different results with and without restarting, so now I only test with restart, FWIW.

If you don't want to restart the JVM, at least make sure and transposition and related tables are cleared before each game.