Fixed nodes self tuning

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Fabio Gobbato
Posts: 132
Joined: Fri Apr 11, 2014 8:45 am
Contact:

Fixed nodes self tuning

Post by Fabio Gobbato » Thu Jul 09, 2015 8:48 pm

I would like to try self tuning playing fixed nodes matches in order to have only deterministic results.

Has anyone already tried this kind of tuning and with what results?

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Fixed nodes self tuning

Post by bob » Thu Jul 09, 2015 11:59 pm

Fabio Gobbato wrote:I would like to try self tuning playing fixed nodes matches in order to have only deterministic results.

Has anyone already tried this kind of tuning and with what results?
You have to be careful because it distorts the results. If you add something that is computationally more expensive than you expect, this test will show the change to be good, since time doesn't count. But in a real game, your slower program will lose depth and lose more games than the test led you to expect.

BTW why the concern about deterministic results? It doesn't reduce the number of games you need to play to have reasonable confidence in the results..

User avatar
Fabio Gobbato
Posts: 132
Joined: Fri Apr 11, 2014 8:45 am
Contact:

Re: Fixed nodes self tuning

Post by Fabio Gobbato » Fri Jul 10, 2015 7:37 am

The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.

mvk
Posts: 589
Joined: Tue Jun 04, 2013 8:15 pm

Re: Fixed nodes self tuning

Post by mvk » Fri Jul 10, 2015 3:40 pm

Fabio Gobbato wrote:The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.
In general, changing to determinism merely converts noise into repeatable noise (in other words: bias), and will not help reduce the number of required measurements. It will help if you want to reproduce the run for some reason, usually debugging.
[Account deleted]

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Fixed nodes self tuning

Post by bob » Fri Jul 10, 2015 4:49 pm

Fabio Gobbato wrote:The idea was to reduce the number of games because of the less noise, but of course it could lead to a bad tuning.
The games themselves are noise. 5000 games at fixed nodes is no more accurate than 5000 games at fixed time limit. Just because you get the same games every time, each game is still an independent trial and the error term is proportional to the number of games played, not the repeatability of the games...

User avatar
Desperado
Posts: 638
Joined: Mon Dec 15, 2008 10:45 am

Re: Fixed nodes self tuning

Post by Desperado » Fri Jul 10, 2015 5:31 pm

Of course there are reasons which make sense to use deterministic behaviour. Just some for the record:

* if you use different hardware for your tests and you want to merge the results.

* you can play "extremely" fast test games.

* Tuning purposes may be a reason, so you can verify results or if you simply want to reproduce the same behaviour.

* Debugging of course.

Well, it is a matter of creativity at the end. Finally you should never use only one approach, because every testing frame has some weakness.

imho, it is a good advice to close a test under real conditions, but you do not necessarily need real conditions as indicator.

Just some thoughts... :)

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Fixed nodes self tuning

Post by bob » Fri Jul 10, 2015 10:02 pm

Desperado wrote:Of course there are reasons which make sense to use deterministic behaviour. Just some for the record:

* if you use different hardware for your tests and you want to merge the results.

* you can play "extremely" fast test games.

* Tuning purposes may be a reason, so you can verify results or if you simply want to reproduce the same behaviour.

* Debugging of course.

Well, it is a matter of creativity at the end. Finally you should never use only one approach, because every testing frame has some weakness.

imho, it is a good advice to close a test under real conditions, but you do not necessarily need real conditions as indicator.

Just some thoughts... :)
Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant.

Most of the other reasons don't apply. You can use different hardware platforms today and adjust the time control. Or even better, just ignore the differences in speed since chess engines don't play the same time control in every game anyway. You can certainly play extremely fast games without using fixed nodes or depth. I do that all the time. In any case, anything you change that affects performance doesn't match up well with fixed depth or fixed nodes. Because changes to performance aren't measured at all.

TomKerrigan
Posts: 64
Joined: Fri Jul 03, 2015 7:22 pm

Re: Fixed nodes self tuning

Post by TomKerrigan » Sat Jul 11, 2015 1:41 am

bob wrote: ...
Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant. ...
If you are prototyping evaluation function code then fixed-node tests are useful. They can tell you if it's worthwhile to optimize the performance of the change.

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Fixed nodes self tuning

Post by bob » Sat Jul 11, 2015 3:14 am

TomKerrigan wrote:
bob wrote: ...
Fixed depth and fixed nodes both have their uses. But modifying/changing the eval and then measuring the change is NOT one of them, because it makes computational cost irrelevant, when anything done in the evaluation is anything but irrelevant. ...
If you are prototyping evaluation function code then fixed-node tests are useful. They can tell you if it's worthwhile to optimize the performance of the change.
Sure, so long as you go in with eyes open knowing this is ignoring any computational cost. But I find testing evaluation changes is just as effective using normal timed games, as I don't see any advantage to fixed depth/node games at all other than for debugging.

All of my eval and search testing is now done on a cluster playing nothing but real games. Then we don't have to worry about this. About the only place I would think about fixed depth or fixed nodes would be if I have an idea that is not going to be easy to write efficiently. I might try a sloppy/quick implementation just to see if it is worthwhile. If so, then I have to tackle the efficiency issues to make it work in real games... So in that case, it can be useful. But that has been EXTREMELY rare for me.

Post Reply