A paper about parameter tuning

[BubbaTough]

It seems to me that you cannot make any progress at all, unless you have both people with new ideas and people who test things.

Of course, it is possible for one person to do both tasks admirably well, given the proper resources.

That is so true Dan. Its no coincidence that the teams that produce the most consistent improvements in their engines always put a lot of attention and effort into their testing methodologies, and very often have dedicated testers. I hear a lot of authors of much weaker engines describe many interesting, creative, and promising ideas...and am convinced some of them are no less potent as programmers and concept people than the authors of much higher rated programs. But their inferior testing holds them back.

Effective testing given limited resources is not trivial (as is clear from the many debates on the subject) and is critical to success. I am convinced that one of the major things Vas did to improve computer chess in general was not just giving us an excellent program to test against, but to reveal a fair amount about his testing methodology.

-Sam

[Dann Corbit]

It seems to me that you cannot make any progress at all, unless you have both people with new ideas and people who test things.

Of course, it is possible for one person to do both tasks admirably well, given the proper resources.

That is so true Dan. Its no coincidence that the teams that produce the most consistent improvements in their engines always put a lot of attention and effort into their testing methodologies, and very often have dedicated testers. I hear a lot of authors of much weaker engines describe many interesting, creative, and promising ideas...and am convinced some of them are no less potent as programmers and concept people than the authors of much higher rated programs. But their inferior testing holds them back.

I think also that the weaker engines face a double-edged sword. Testers are more interested in testing the strongest engines, so that the stronger engines get the bulk of the free testing provided by testing groups. Fortunately, people like Graham spend a great deal of energy testing a broad spectrum of engines so that the startup engine developers are not left completely in the cold.

Effective testing given limited resources is not trivial (as is clear from the many debates on the subject) and is critical to success. I am convinced that one of the major things Vas did to improve computer chess in general was not just giving us an excellent program to test against, but to reveal a fair amount about his testing methodology.

-Sam

[Zach Wegner]

It seems to me that you cannot make any progress at all, unless you have both people with new ideas and people who test things.

Of course, it is possible for one person to do both tasks admirably well, given the proper resources.

That is so true Dan. Its no coincidence that the teams that produce the most consistent improvements in their engines always put a lot of attention and effort into their testing methodologies, and very often have dedicated testers. I hear a lot of authors of much weaker engines describe many interesting, creative, and promising ideas...and am convinced some of them are no less potent as programmers and concept people than the authors of much higher rated programs. But their inferior testing holds them back.

I think also that the weaker engines face a double-edged sword. Testers are more interested in testing the strongest engines, so that the stronger engines get the bulk of the free testing provided by testing groups. Fortunately, people like Graham spend a great deal of energy testing a broad spectrum of engines so that the startup engine developers are not left completely in the cold.

Effective testing given limited resources is not trivial (as is clear from the many debates on the subject) and is critical to success. I am convinced that one of the major things Vas did to improve computer chess in general was not just giving us an excellent program to test against, but to reveal a fair amount about his testing methodology.

-Sam

Of course I agree that testing is extremely important. I think the various rating lists, though, are basically meaningless for development. They're interesting for determining the relative strengths of many public engines, and having an "official" rating for engines. But when you're developing, you need _lots_ of results for _lots_ of different versions quickly. This simply isn't possible with a public rating group--no engine has more than a few thousand games, and no engine has more than a handful of versions.

This doesn't mean rating lists are useless, and no disrespect to the testers intended. But if you're relying on them when you're trying to improve an engine, you need to rethink your development model.

[michiguel]

It seems to me that you cannot make any progress at all, unless you have both people with new ideas and people who test things.

Of course, it is possible for one person to do both tasks admirably well, given the proper resources.

That is so true Dan. Its no coincidence that the teams that produce the most consistent improvements in their engines always put a lot of attention and effort into their testing methodologies, and very often have dedicated testers. I hear a lot of authors of much weaker engines describe many interesting, creative, and promising ideas...and am convinced some of them are no less potent as programmers and concept people than the authors of much higher rated programs. But their inferior testing holds them back.

I think also that the weaker engines face a double-edged sword. Testers are more interested in testing the strongest engines, so that the stronger engines get the bulk of the free testing provided by testing groups. Fortunately, people like Graham spend a great deal of energy testing a broad spectrum of engines so that the startup engine developers are not left completely in the cold.

Effective testing given limited resources is not trivial (as is clear from the many debates on the subject) and is critical to success. I am convinced that one of the major things Vas did to improve computer chess in general was not just giving us an excellent program to test against, but to reveal a fair amount about his testing methodology.

-Sam

Of course I agree that testing is extremely important. I think the various rating lists, though, are basically meaningless for development. They're interesting for determining the relative strengths of many public engines, and having an "official" rating for engines. But when you're developing, you need _lots_ of results for _lots_ of different versions quickly. This simply isn't possible with a public rating group--no engine has more than a few thousand games, and no engine has more than a handful of versions.

This doesn't mean rating lists are useless, and no disrespect to the testers intended. But if you're relying on them when you're trying to improve an engine, you need to rethink your development model.

They find all sort of bugs and the rating they get is at lower paces, something that is virtually impossible for us to do. It sort of confirm (or not) what was done privately.

Miguel

[mcostalba]

They find all sort of bugs and the rating they get is at lower paces, something that is virtually impossible for us to do. It sort of confirm (or not) what was done privately.

I agree, public lists are no useful for development because you need to test every code chunk you throw in and this is something _you_ have to do.

Public lists are useful as a reality check to verify by independent testers and at longer time controls that you are advancing in the right direction and there are no regressions.

[diep]

It seems to me that you cannot make any progress at all, unless you have both people with new ideas and people who test things.

Of course, it is possible for one person to do both tasks admirably well, given the proper resources.

That is so true Dan. Its no coincidence that the teams that produce the most consistent improvements in their engines always put a lot of attention and effort into their testing methodologies, and very often have dedicated testers. I hear a lot of authors of much weaker engines describe many interesting, creative, and promising ideas...and am convinced some of them are no less potent as programmers and concept people than the authors of much higher rated programs. But their inferior testing holds them back.

Effective testing given limited resources is not trivial (as is clear from the many debates on the subject) and is critical to success. I am convinced that one of the major things Vas did to improve computer chess in general was not just giving us an excellent program to test against, but to reveal a fair amount about his testing methodology.

-Sam

Well Rybka has great similarities to the panther tank from world war 2.

http://militaryhistory.about.com/od/veh ... anther.htm


Well they had Bob's entire cluster or something from end 2004 far into 2005 to tune rybka 1.0 and zappa world champs 2005. Nothing is new in that testing methology. The question is how Anthony KNEW what other programmers had been doing there. How did he *get* that knowledge.

Because i didn't tell him that. He dropped a few lines in my direction.

How did he KNOW that?

Same is true for Pradu Kannan, he also suddenly was talking similar stuff like Sune Fischer, both seem to be buddy mates somehow. Where did they get the knowledge?

There seems to be some sort of 'big tank filled with information' where some among us get their information from and it is unclear how they get access to that big tank.

If you pay a few software engineers to get to work EXISTING PROVEN concepts, that's quite different from putting a lot of your time into discovering new methods and then getting those to work.

A big number of released engines now seem to have the same programming staff, yet they share they only have proven concepts. Some years ago if you would have done that, then the original engine author/manager would have started 100 courtcases. That's not happening now. Why?

If you carry out proven concepts from different programmers in a very well manner, then that kicks butt of course.

Combining those already is very complicated indeed, but you get what you pay for.

I missed Bob giving online anywhere even 1 line of comment on the master thesis from Anthony Cozzie, which is about parameter tuning, using the Fabien Letouzey concept.

Of course if you careful analyze later then what went ok there, it's easy to improve and get in other fields (neural networks huh) things moving as well.

Myself i have some ideas that can lead to a terminator chip, using self learning concepts and in fact a rather simple mechanism that can create this. It's not based upon neural networks nor testing to death like crafty seems to be doing, but it's again a big wide open research field.

Please distinguish between what others have been doing already and new concepts. There is a big difference between PAID industrial engineering and open research.

You get what you pay for, but who is paying all this?

All GM's i know are paid for example and not even much. Just a couple of thousands a month usually (which for a very big expert in his field is not much using objective standards let's face it).

If you hire Cozzie for version 1.0, it's very unclear who builds 1.x to 2.1
then. For the Neural Network parameter tuning of material another dude who probabla didn't do anything else and whose face i see right now in front of me, so i'm guessing i'm not far off.

The testing will be another staff and the current versions incorporate the mainline checking a la shredder a tad and singular extensions type GCP/Diepeveen/Moreland, the forward pruning from David Omid Tabibi and exactly described by Sune Fischer. And so on and so on.

Oh i forget the experimental unisys hardware of 16 sockets with new nehalem type cores. Expected sales price is 2 million, but release date is not clear yet. So it's very secret yet.

Possibly physical located in Tel Aviv somewhere near the 5 INTEL factories there.

We're speaking about a big budget.

Having all that knowledge is a lot simpler hiring all those guys, as they can brilliantly execute it.

BUT WHO PAID FOR IT?

Vincent

p.s. it is also interesting to note that during world champs 2005 that Shay Bushinsky already seemed to know what was going on, referring to the type of questions he asked when Cozzie and a lot of other programmers were present there in the cafe.

How did he know?

[Rein Halbersma]

[snip]

Please distinguish between what others have been doing already and new concepts. There is a big difference between PAID industrial engineering and open research.

You get what you pay for, but who is paying all this?

All GM's i know are paid for example and not even much. Just a couple of thousands a month usually (which for a very big expert in his field is not much using objective standards let's face it).

If you hire Cozzie for version 1.0, it's very unclear who builds 1.x to 2.1
then. For the Neural Network parameter tuning of material another dude who probabla didn't do anything else and whose face i see right now in front of me, so i'm guessing i'm not far off.

The testing will be another staff and the current versions incorporate the mainline checking a la shredder a tad and singular extensions type GCP/Diepeveen/Moreland, the forward pruning from David Omid Tabibi and exactly described by Sune Fischer. And so on and so on.

Oh i forget the experimental unisys hardware of 16 sockets with new nehalem type cores. Expected sales price is 2 million, but release date is not clear yet. So it's very secret yet.

Possibly physical located in Tel Aviv somewhere near the 5 INTEL factories there.

We're speaking about a big budget.

Having all that knowledge is a lot simpler hiring all those guys, as they can brilliantly execute it.

BUT WHO PAID FOR IT?

Vincent

p.s. it is also interesting to note that during world champs 2005 that Shay Bushinsky already seemed to know what was going on, referring to the type of questions he asked when Cozzie and a lot of other programmers were present there in the cafe.

How did he know?

Is it April 1 already, or did I miss the joke in the above? Are you suggesting that the entire computer chess community is on the payroll of Homeland Security or the NSA or what?

[diep]

[snip]

Please distinguish between what others have been doing already and new concepts. There is a big difference between PAID industrial engineering and open research.

You get what you pay for, but who is paying all this?

All GM's i know are paid for example and not even much. Just a couple of thousands a month usually (which for a very big expert in his field is not much using objective standards let's face it).

If you hire Cozzie for version 1.0, it's very unclear who builds 1.x to 2.1
then. For the Neural Network parameter tuning of material another dude who probabla didn't do anything else and whose face i see right now in front of me, so i'm guessing i'm not far off.

The testing will be another staff and the current versions incorporate the mainline checking a la shredder a tad and singular extensions type GCP/Diepeveen/Moreland, the forward pruning from David Omid Tabibi and exactly described by Sune Fischer. And so on and so on.

Oh i forget the experimental unisys hardware of 16 sockets with new nehalem type cores. Expected sales price is 2 million, but release date is not clear yet. So it's very secret yet.

Possibly physical located in Tel Aviv somewhere near the 5 INTEL factories there.

We're speaking about a big budget.

Having all that knowledge is a lot simpler hiring all those guys, as they can brilliantly execute it.

BUT WHO PAID FOR IT?

Vincent

p.s. it is also interesting to note that during world champs 2005 that Shay Bushinsky already seemed to know what was going on, referring to the type of questions he asked when Cozzie and a lot of other programmers were present there in the cafe.

How did he know?

Is it April 1 already, or did I miss the joke in the above? Are you suggesting that the entire computer chess community is on the payroll of Homeland Security or the NSA or what?

It's none of my business who works where.

NSA is big and not so interesting.

Just one department in the Netherlands (MIVD) has 2000 people working there and is not really looking for more persons i'd guess (i have no clue, but that's what i was told when i tried to sell a factorisation project in the year 2001/2002 or so to them - which failed). Note that MIVD here is 1 out of the 100 organisations that's there in this nation (google for more info).

NCSA is more interesting of course as they deal with big hardware.

Well you can google actually for every person and how many lectures they gave at NCSA or watched.

You know, there is very few programmers there who know something about algorithms and all those organisations are mighty big. Most programmers are that good that they work at contract basis and therefore are officially consultant. How can you be part of any organisation then?

I would argue that a bunch of psychopaths there grab the opportunity when they see a product they can 'produce' themselves by some software engineering parts of others co-using algorithmic or testing knowledge they would not posses without having the job they actually have. Everything can get denied anyway, isn't it?

Without that hidden information, reality is that, with a few exceptions, majority wouldn't even be able to write tic-tac-toe.

Having that knowledge it's easy to post of course that the only interesting thing is just testing, rather than invent new algorithms for an existing challenge or problem.

Please don't confuse all this with some people who push project A or B and who have cash. There is a mighty amount of chess projects that get well funded (of course). Usually it involves travelling abroad for GM's. As Gm's are professional chessplayers who sit more abroad than inside the nation sometimes, obviously it's impossible for any N*SA organisation to hire them. So they for sure are not part of ANY form of organisation.

CYA,
Vincent

[Aaron Becker]

Hah, I work across the street from the NCSA main office. Who do I need to talk to there to get in on the secret chess programming cabal? It sounds like fun.

[diep]

Hah, I work across the street from the NCSA main office. Who do I need to talk to there to get in on the secret chess programming cabal? It sounds like fun.

You can say whatever you want, but someone is paying big cash to a bunch of programmers past few years to get things improved and also manages to show up with real big LOW LATENCY SHARED MEMORY hardware, so called 'clusters', though none of all those programs can run on any cluster.

Toga, Sjeng, Rybka, Jonny to just name 4 of them.

Also clusters happen to have thosuands of cores, not 40 of course. Building clusters of just 40 cores doesn't break even *ever*. You build a cluster to scale to THOUSANDS of cores.

So if you got a cluster version you can just as well start your proggie also at a 4096 core cluster isn't it?

Only Hydra really works at a cluster and it gets completely outsearched by todays software despite that todays software gets factor 10 less nps.

The huge difference with the past and today is that all this happens now for a few years and it happens very sneaky, and don't tell me that GM's work for free. I have a dozen on my chat and none does. They are professional players who play chess and are very good at that. Very bad usually in formulating grammar and factorizing which thing causes what. I can give dramatic examples there (when i played Korchnoi and some other world top players). Thing is they show up for cash, so do programmers.

Once you have a few people working to do all this, you soon have an explosion also in management and scripting and other testing stuff requirements needed.

When in 2000 i could observe a small project team, they already wasted 300k dollar just to start. Now that was all a lot tinier than this and not much of a hardware was involved. In fact none. Another tiny project i saw swallowed 1 million euro as budgetted. Just the homepage bla bla already was 200k+. So we speak about big cash here.

Vincent