Spare CPU cycles anyone?

[mcostalba]

I start to understund that a big obstacle to chess programming developing is testing.

That’s what I have learned so far:

-Chess programming is tough.

- A big part of the difficult is that the link between a modification and result in terms of ELO gain/loose is not obvious to say the least.

- It is easier to test if a modification is an improvment then the absolute ELO gain. For developing I would think the first condition suffice. Also because absolute ELO gain/loose of a new version of an engine is perhaps always better to leave to indepenent third parties then to engine author for many reasons.

- For serious developing any change should be tested at least for regressions.

- We have far more ideas then testing time (read nights with your PC running while you sleep)


So what I would propose, at least for discussion, is the following:

To create a testing pool of people devoted to their preferred engine that test daily (nightly) the current and the challanger version of the engine.

Versions are provided by the developers and tester have of course access to development trees.

The gain of this are:

- Faster developing of your preferred engine because new ideas are tested and discarded/accepted with accelerated time cycle.

- Documentation history: do not reinvent the wheel but keep track and publish modifications applied with rationale and testing result. Only this point could greatly help chess programming community because detailed information of this kind is precious and normally requires a lot of time to gather for a single developer.

- High testing results quality due to indipendent and documented testing.

Comments?

Thanks
Marco

[Michael Sherwin]

Except maybe, for a lucky few authors a dedicated (to your engine) pool of testers is impossible to find. Testers, in general, will get bored with your engine if that is mainly what they test. And they will get upset if you send them too many betas that are not better. It is also a great strain on the programmer to try to meet the expectations of their beta testers. If a beta tester does not get back to you within a few days with some data then the programmer must either 'set on his hands waiting' or proceed with development often making any data that is eventually recieved back totally worthless. Beta testers usually in the end will slow a programmer down. And beta testers will almost never play enough games for any one beta to be rated accuratly anyway.

I will say that I was lucky that one person (Tony Thomas) was willing to test most of the RomiChess betas that I sent to him, in 360 game gauntlets. That was helpful in that I recieved some confirmation back that my betas were getting better. But, even in that testing relationship, I worked very hard to ensure that the new betas had a good chance of being better. In other words I played more test games per beta sent than Tony did.

Chess programming is tough!