community test result web page?

[BubbaTough]

Hi all,

For me, one of the most valuable things about this site and chatting with other programmers on ICC is sharing the results of ideas we have tried in our respective programs. Currently, this sharing process if extremely adhoc. In fact, I have probably learned just as much chatting online with authors of programs such as NOW or WEID as I have on this site, and none of the results we have shared in these exchanges is documented anywhere for others to take advantage of. I am also extremely confident that over the years many more ideas I am unaware of have been past through this site without drawing my attention.

If there were someone interested in hosting a website that listed techniques, and the results various programs, private or public, have had testing them, I think it would be very valuable...and I know that while I am not quite willing to dedicate the time to run such a site myself, I would be very happy to contribute to it. I would expect managing such a website would have some challenges to it, since many techniques will be tried in multiple different ways and tested in multiple different ways, and any table of technique/program/test results probably needs to be supplemented with all of the details that the authors supplying results can provide. Maintaining an order for this chaotic quagmire would have be somewhat time consuming I am sure.

Anyway, I doubt much will come of this suggestion, but if anyone does have the time/interest to take this ball and run with it let me know and I will be happy to contribute (and take advantage of their efforts).

Example techniques (to give a flavor of what I am talking about):

1. LMR stuff (excluding killer moves? use history? pawn moves? passed-pawn moves? reducing later moves further? how much to reduce? reducing "bad captures"? excluding "threats to opponent king area"? reduce tactical threats?)

2. extensions (check, fractional extensions, passed pawn to the 6th/7th, singular extensions, threat extensions, reducing extensions based on how many previous extensions, maximum extension limits)

3. null move (what depth? conditional depths? re-search method? when to apply?)

4. eval (mobility vs. mobility - opponent pawn captures, "trapped piece" definitions, protected passed pawn in king/pawn endgames, cut off king in king/rook endings, opening specific heuristics (don't trap in your rook, don't block your d/e pawn, don't block your c pawn in queen pawn openings)

5. Quiescence (all captures or just "good captures"? Check? escape check? maximum depth?

This list could have twice as many subjects (such as Time management, SEE, Move generation, Move Ordering, Razoring/Futility Pruning) and twice as many things listed in each subject, and still they would have all come up in my informal chats, all been things I have tested myself, and most been tested by most of the original program authors I have talked to. And since I only spend about 1 weekend a month generating code for LearningLemming, for many authors it is probably even more chaotic (granted, a weekend of coding for me entails about 40 hours, so its a bit more than it would be for the sleep addicted).

I would never expect a site such as this to be a replacement for original program authors doing their own testings of their own techniques and ideas, since the interactions of techniques and implementation details makes resuts delightfully distinct for each program, but I would expect the combined collection of concepts and results to provide both inspiration and a feel for prioritization for authors of all levels.

-Sam

[Edsel Apostol]

Hi Sam,

Your idea is cool. I could contribute the results of my own testing. I also could share some of my ideas and insights based on my experience in programming Twisted Logic.

Let's only hope that someone would come to participate and help put up a web page where we are going to put all of these. I could do it in my geocities account but I just don't have enough free time.

Edsel Apostol

[hgm]

Never start a new website if there are already plenty of sites that could do the job.

What is wrong with posting these test results in an existing forum (e.g. the Winboard Forum)?

[mjlef]

I would be glad to contribute to such an effort. I think the key is organization, which a simple chat forum is not so well suited for. Maybe we could setup a wiki, create entries for things like Null Move Pruning, then anyone who wants to ocntribute can easily add to whatever is there. I am not quite sure how to setup a wiki, but I hear it is not hard. I signed up for a small web hosting service that might be able to supply some space. I will check. I might also be able to host this on the zillions-of-games.com domain, as a sort of private wiki. Does anyone else have a suitable space or skills setting up a wiki?

Mark

[mjlef]

OK, I found some free wikis sites that host and everything. I setup a starter here:

http://chessprogramming.wikispaces.com/

and a sample page here:

http://chessprogramming.wikispaces.com/ ... ve+Pruning

where I put in an idea I tested last night.

[opraus]

Hi,

Some 'Ideas' do no good in combination with others, and some do better; Therefore, it is hard to say 'universally' that something is good/bad.

But, if we set up a database where every participant enters the 'specs' of their program and the corresponding improvement gained, anyone can compare their 'specs' to others with similar ones to see what works best.

Ie, every engine entry would have a 'check' next to each 'idea' implemented, and a comment/score for it's value.

This way you could query the DB for engines like your own, and see what else improved them, without fear of mixing/matching incompatible 'ideas'.

A spreadsheet in Google Docs might be a start??

Each contributor would add rows for any 'spec'/idea he has implemented, and the rest would have to edit/update with their engines' spec.

Eventually, there would be many rows of specs, with many columns of participants, and each cell would contain anything from na, bad, neutral, good, great, untested ....

We could just share a Google Spreadsheet

[opraus]

Quickie example:

http://spreadsheets.google.com/ccc?key= ... HcPw&hl=en

[mjlef]

I have actually done some similar spreadsheets about a few public programs. It would have to contain a lot of conditions though, since some ideas only work with very specific conditions.

[mageofmaple]

OK, I found some free wikis sites that host and everything. I setup a starter here:

http://chessprogramming.wikispaces.com/

and a sample page here:

http://chessprogramming.wikispaces.com/ ... ve+Pruning

where I put in an idea I tested last night.

Very nice. I like a nice, simple solution. I would be happy to contribute.

[Zach Wegner]

I think this is a good idea. It could be expanded to be a complete source of theory. For instance, in the null move page, we could quote the old post of Christophe Theron about the Null Move Observation. We could link his name to a page about him, his engine, and other ideas he might have talked about. If many people contribute to this, we could have a quite nice little site. And as it pertains to the original post, we could have a results section on each page, where different authors can describe their implementations and the outcome.