Page 2 of 2

Re: non overlapping error bars

Posted: Thu Aug 06, 2020 11:59 am
by jswaff
In this case there is just a single opponent - it's version A vs version B. The "restart" issue you mentioned is interesting. This is my Java engine, and I do _not_restart it between games due to the "warm up overhead" of starting a JVM. My first thought was that there must be some sort of resource contention. Based on other responses I think the most likely possibility at this point is that I underestimated the importance of using the same set of starting positions. (I really hope that is the case.) It will take me several days to confirm.

Re: non overlapping error bars

Posted: Thu Aug 06, 2020 12:22 pm
by brianr
Might not be related at all, but in the early days of Leela chess I got very different results with and without restarting, so now I only test with restart, FWIW.

If you don't want to restart the JVM, at least make sure and transposition and related tables are cleared before each game.