Fixed nodes self tuning

[Fabio Gobbato]

I would like to try self tuning playing fixed nodes matches in order to have only deterministic results.

Has anyone already tried this kind of tuning and with what results?

[bob]

I would like to try self tuning playing fixed nodes matches in order to have only deterministic results.

Has anyone already tried this kind of tuning and with what results?

You have to be careful because it distorts the results. If you add something that is computationally more expensive than you expect, this test will show the change to be good, since time doesn't count. But in a real game, your slower program will lose depth and lose more games than the test led you to expect.

BTW why the concern about deterministic results? It doesn't reduce the number of games you need to play to have reasonable confidence in the results..

[Fabio Gobbato]

The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.

[mvk]

The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.

In general, changing to determinism merely converts noise into repeatable noise (in other words: bias), and will not help reduce the number of required measurements. It will help if you want to reproduce the run for some reason, usually debugging.

[bob]

The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.

The games themselves are noise. 5000 games at fixed nodes is no more accurate than 5000 games at fixed time limit. Just because you get the same games every time, each game is still an independent trial and the error term is proportional to the number of games played, not the repeatability of the games...

[Desperado]

Of course there are reasons which make sense to use deterministic behaviour. Just some for the record:

* if you use different hardware for your tests and you want to merge the results.

* you can play "extremely" fast test games.

* Tuning purposes may be a reason, so you can verify results or if you simply want to reproduce the same behaviour.

* Debugging of course.

Well, it is a matter of creativity at the end. Finally you should never use only one approach, because every testing frame has some weakness.

imho, it is a good advice to close a test under real conditions, but you do not necessarily need real conditions as indicator.

Just some thoughts...

[bob]

Of course there are reasons which make sense to use deterministic behaviour. Just some for the record:

* if you use different hardware for your tests and you want to merge the results.

* you can play "extremely" fast test games.

* Tuning purposes may be a reason, so you can verify results or if you simply want to reproduce the same behaviour.

* Debugging of course.

Well, it is a matter of creativity at the end. Finally you should never use only one approach, because every testing frame has some weakness.

imho, it is a good advice to close a test under real conditions, but you do not necessarily need real conditions as indicator.

Just some thoughts...

Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant.

Most of the other reasons don't apply. You can use different hardware platforms today and adjust the time control. Or even better, just ignore the differences in speed since chess engines don't play the same time control in every game anyway. You can certainly play extremely fast games without using fixed nodes or depth. I do that all the time. In any case, anything you change that affects performance doesn't match up well with fixed depth or fixed nodes. Because changes to performance aren't measured at all.

[TomKerrigan]

...
Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant. ...

If you are prototyping evaluation function code then fixed-node tests are useful. They can tell you if it's worthwhile to optimize the performance of the change.

[bob]

...
Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant. ...

If you are prototyping evaluation function code then fixed-node tests are useful. They can tell you if it's worthwhile to optimize the performance of the change.

Sure, so long as you go in with eyes open knowing this is ignoring any computational cost. But I find testing evaluation changes is just as effective using normal timed games, as I don't see any advantage to fixed depth/node games at all other than for debugging.

All of my eval and search testing is now done on a cluster playing nothing but real games. Then we don't have to worry about this. About the only place I would think about fixed depth or fixed nodes would be if I have an idea that is not going to be easy to write efficiently. I might try a sloppy/quick implementation just to see if it is worthwhile. If so, then I have to tackle the efficiency issues to make it work in real games... So in that case, it can be useful. But that has been EXTREMELY rare for me.