Some thoughts on QS

[hgm]

Also, don't forget that even though a move is reduced, it could be search to the original depth if something interesting is found.

I think this is the key to the success of LMR. Other methods for reducing poor moves in the end always fail, because the search starts abusing them to push trouble over the horizon, so it can fail high even with a poor move. So it is very dangerous to accept a fail high of a reduced search, it just invites horizon effect. With LMR the fail high is only accepted if the full-depth re-search also fails high. So any fail high must be real, you cannot fool yourself by turning a blind eye.

Null-move pruning is a bit dangerous, in this respect. It only works because null moves are so efficient in bringing trouble closer.

[Don]

Also, don't forget that even though a move is reduced, it could be search to the original depth if something interesting is found.

I think this is the key to the success of LMR. Other methods for reducing poor moves in the end always fail, because the search starts abusing them to push trouble over the horizon, so it can fail high even with a poor move. So it is very dangerous to accept a fail high of a reduced search, it just invites horizon effect. With LMR the fail high is only accepted if the full-depth re-search also fails high. So any fail high must be real, you cannot fool yourself by turning a blind eye.

Null-move pruning is a bit dangerous, in this respect. It only works because null moves are so efficient in bringing trouble closer.

There was some dubious experimentation that we did trying to avoid the research for the sake of speed but it proved to be a bad idea and we never made it work.

Note that we are not afraid to try almost anything - something you can be very surprised and I learned long ago not to give up too quickly on any idea and especially not to make too many assumptions about what will work and what will not work.

In fact, there is something in Komodo (which I cannot reveal) that we both hate - but it works and it works very well. We tried many times to replace it with something more logical, something that might have a more logical basis but never succeeded. It was a silly experiment that we may have easily rejected if we were less bored at the time but it's now a part of the program.

We always try to attach meaning to anything we do - there should be a logical reason to do something that we can reason about based on "near axioms" that we have adapted. Some of our "axioms" are not always correct and become rules of thumb - for example one of them is that if we do something which only has the affect of changing the move ordering - if it speeds up the search it HAS to be a good thing. Turns out that is not always 100% correct because move ordering interacts with LMR. But it's still a good rule of thumb.

[Don]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

[diep]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

You're comparing 10 versus 12 ply with an engine that's getting like nearby 30 at slow time controls?

What's this for sort of science?

By the way how can you already win 2 ply with it at these small search depths?

Diep without LMR gets 17 ply, with LMR gets 19 ply. It's like 1 ply at 10 ply search depths.

That's without forward pruning in fact.

Your thing is so optimized to LMR it's not even objective to test it without!

[Uri Blass]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

Did you test longer time control to see that you get higher rating improvement?

Maybe the time advantage that the No LMR version is going to need to get 50% is always going to increase at longer time control(I am not sure if it is the case without testing) but even in this case
I expect the rating difference to go down at some point because when both versions get closer to perfect chess there is a limit to the difference
in rating points.

[Don]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

You're comparing 10 versus 12 ply with an engine that's getting like nearby 30 at slow time controls?

What's this for sort of science?

I don't understand your terminology. This is game in 6 seconds with a 1/10 second increment and yes, we get almost 2 ply average extra depth.

The depth AND the superiority will only increase with depth - make note of that.

By the way how can you already win 2 ply with it at these small search depths?

Diep without LMR gets 17 ply, with LMR gets 19 ply. It's like 1 ply at 10 ply search depths.

That's without forward pruning in fact.

Aggressive LMR is the key. We don't do LMR on last 3 ply but heavy foward pruning - even with LMR turned off.

For LMR we search the first N moves full depth, then we search a few more with full ply reduction, then we add another half ply reduction until we are reducing moves at the end of the list by 3 full ply or something like that (I don't remember the exact number.) If you order your moves well you can get away with that as Komodo and every good program does. It took hundreds of thousands of games to figure out the right formula for Komodo and we probably still don't have it right.

Your thing is so optimized to LMR it's not even objective to test it without!

[Uri Blass]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

You're comparing 10 versus 12 ply with an engine that's getting like nearby 30 at slow time controls?

What's this for sort of science?

I don't understand your terminology. This is game in 6 seconds with a 1/10 second increment and yes, we get almost 2 ply average extra depth.

The depth AND the superiority will only increase with depth - make note of that.

By the way how can you already win 2 ply with it at these small search depths?

Diep without LMR gets 17 ply, with LMR gets 19 ply. It's like 1 ply at 10 ply search depths.

That's without forward pruning in fact.

Aggressive LMR is the key. We don't do LMR on last 3 ply but heavy foward pruning - even with LMR turned off.

For LMR we search the first N moves full depth, then we search a few more with full ply reduction, then we add another half ply reduction until we are reducing moves at the end of the list by 3 full ply or something like that (I don't remember the exact number.) If you order your moves well you can get away with that as Komodo and every good program does. It took hundreds of thousands of games to figure out the right formula for Komodo and we probably still don't have it right.

Your thing is so optimized to LMR it's not even objective to test it without!

You can know only that the depth difference is going to increase with more time but depth means nothing about playing strength.

You did not prove that LMR gives you more rating points at longer time control than 6 seconds per game+0.1 seconds per move.

I suggest that you test also at 12+0.2 and at 24+0.4

I will not be surprised if LMR simply does not help when the remaining depth is very big and it is also possible that the LMR that you do when the remaining depth is small is productive when the LMR that you do at big depths is counter productive.

I do not claim to be sure that it is the case but only that it is possible and
you cannot know without testing.

It is clearly possible to reduce too much and if you do too many reductions you lose playing strength relative to correct reductions.

Note that I think that it is logical to reduce more when the remaining
depth is small because even if a move that seems to be bad is really good you can be almost sure that search is not going to find it
even if you do not reduce.

When the remaining depth is big then there is a bigger probability that the search is going to find something without reduction when it is not going to find it with reduction so it seems to me that it is better to be less aggresive in reductions.

[diep]

I'm not disagreeing; you cannot have weak spots, but if i see how SF can get 30-40 ply pretty easy, whereas each additional ply hardly gives it elo, then i wonder whether there is any objective compare possible nowadays with search depths; especially if we realize that if you refer to the tactics, that Diep at around 20 ply solves tactical about every position i have, whereas it hardly extends tactics, and SF not seldom needs 39+ plies for deep tactics.

It's ok not to get much for a ply if your branching factor is less than 2. That's how modern programs work, they give up a lot in things they might miss, but they get a lot for it in depth and other things they might see but never could before. Most programs of today will not solve problems in the same depth as yesterday, but will solve with much less nodes so it's a very good trade-off.

There is also the issue of whether it really makes sense to be so consistent in depth like old programs did - that's nothing like how humans do it so I think heavy pruning, reductions and aggressive futility is a big win - a program SHOULD look much deeper in some lines than others and only in the past few years has then been the case.

Well the whole point is that what you call 'taking a deeper look' only happens in the mainline. That's what effectively happens. Getting a fail high hardly happens.

It takes 15 plies extra or so to find tactics at those depths, just the mainline gets seen very deep. Diep doesn't suffer too much from this problem - obviously its problem is getting any depth at all

Just seeing mainlines deep - it's a way to play chess, but it makes the game boring.

Here is the update I promised after 700 games - enough to make the point.

I'm even surprised by this as it is more than I expected but apparently at these fast time controls (Fischer 6s + 0.1) LMR is worth a whopping 150+ ELO. This will only increase with longer time controls - low branching factor overwhelms safe conservative slow search.

Code: Select all

Rank Name             Elo      +      -    games   score   oppo.   draws 
   1 komodo5-Lmr    3180.5   29.4   29.4     700   68.6%  3000.0   41.9% 
   2 komodo5-noLmr  3000.0   29.4   29.4     700   31.4%  3180.5   41.9% 


      TIME       RATIO    log(r)     NODES    log(r)  ave DEPTH    GAMES   PLAYER
 ---------  ----------  --------  --------  --------  ---------  -------   -------------
    0.1959       0.964    -0.037     0.288     0.005    12.4400      700   komodo5-Lmr
    0.2033       1.000     0.000     0.287     0.000    10.5190      700   komodo5-noLmr

You're comparing 10 versus 12 ply with an engine that's getting like nearby 30 at slow time controls?

What's this for sort of science?

I don't understand your terminology. This is game in 6 seconds with a 1/10 second increment and yes, we get almost 2 ply average extra depth.

The depth AND the superiority will only increase with depth - make note of that.

By the way how can you already win 2 ply with it at these small search depths?

Diep without LMR gets 17 ply, with LMR gets 19 ply. It's like 1 ply at 10 ply search depths.

That's without forward pruning in fact.

Aggressive LMR is the key. We don't do LMR on last 3 ply but heavy foward pruning - even with LMR turned off.

For LMR we search the first N moves full depth, then we search a few more with full ply reduction, then we add another half ply reduction until we are reducing moves at the end of the list by 3 full ply or something like that (I don't remember the exact number.) If you order your moves well you can get away with that as Komodo and every good program does. It took hundreds of thousands of games to figure out the right formula for Komodo and we probably still don't have it right.

Your thing is so optimized to LMR it's not even objective to test it without!

You can know only that the depth difference is going to increase with more time but depth means nothing about playing strength.

You did not prove that LMR gives you more rating points at longer time control than 6 seconds per game+0.1 seconds per move.

I suggest that you test also at 12+0.2 and at 24+0.4

I will not be surprised if LMR simply does not help when the remaining depth is very big and it is also possible that the LMR that you do when the remaining depth is small is productive when the LMR that you do at big depths is counter productive.

I do not claim to be sure that it is the case but only that it is possible and
you cannot know without testing.

It is clearly possible to reduce too much and if you do too many reductions you lose playing strength relative to correct reductions.

Note that I think that it is logical to reduce more when the remaining
depth is small because even if a move that seems to be bad is really good you can be almost sure that search is not going to find it
even if you do not reduce.

When the remaining depth is big then there is a bigger probability that the search is going to find something without reduction when it is not going to find it with reduction so it seems to me that it is better to be less aggresive in reductions.

We seldom agree, but in this case we do.

Rybka3 is on CCRL 3134 and Houdini which is like 7 plies deeper searching yet having the same eval, it's 3208.

That's 70 elopoints for 7 plies.

However in case of Komodo he's winning 2 ply already at a depth of 10, whereas he's heavy forward pruning last few plies.

It means Komodo without LMR is hopelessly inefficient, so that changes the equasion as i assume Don didn't cut'n paste the evaluation of rybka.

The point is that Don's search is total inefficient *without* LMR.

This basically means don gets more out of LMR than rybka - which could be very true as rybka of course forward prunes a lot last few plies.

Also i advice to Don to quote Johan de Koning to not do incesttesting as incesttesting never is good idea.

Vincent

[Don]

We seldom agree, but in this case we do.

Rybka3 is on CCRL 3134 and Houdini which is like 7 plies deeper searching yet having the same eval, it's 3208.

That's 70 elopoints for 7 plies.

However in case of Komodo he's winning 2 ply already at a depth of 10, whereas he's heavy forward pruning last few plies.

It means Komodo without LMR is hopelessly inefficient, so that changes the equasion as i assume Don didn't cut'n paste the evaluation of rybka.

The point is that Don's search is total inefficient *without* LMR.

Komodo without LMR is still stronger than most programs. But what does that matter? This would be like me saying Diep is totally dependent on alpha/beta pruning or else it's hopelessly inefficient. That would be a true but meaningless statement because what matters is how you put everything together.

This basically means don gets more out of LMR than rybka - which could be very true as rybka of course forward prunes a lot last few plies.

Also i advice to Don to quote Johan de Koning to not do incesttesting as incesttesting never is good idea.

Vincent

I have no idea what you are talking about - but we do all our testing against other programs, not komodo vs komodo. Is that what you are talking about?

Don

[diep]

We seldom agree, but in this case we do.

Rybka3 is on CCRL 3134 and Houdini which is like 7 plies deeper searching yet having the same eval, it's 3208.

That's 70 elopoints for 7 plies.

However in case of Komodo he's winning 2 ply already at a depth of 10, whereas he's heavy forward pruning last few plies.

It means Komodo without LMR is hopelessly inefficient, so that changes the equasion as i assume Don didn't cut'n paste the evaluation of rybka.

The point is that Don's search is total inefficient *without* LMR.

Komodo without LMR is still stronger than most programs. But what does that matter? This would be like me saying Diep is totally dependent on alpha/beta pruning or else it's hopelessly inefficient. That would be a true but meaningless statement because what matters is how you put everything together.

This basically means don gets more out of LMR than rybka - which could be very true as rybka of course forward prunes a lot last few plies.

Also i advice to Don to quote Johan de Koning to not do incesttesting as incesttesting never is good idea.

Vincent

I have no idea what you are talking about - but we do all our testing against other programs, not komodo vs komodo. Is that what you are talking about?

Don

The most stupid way of forward pruning, if i enable it in Diep, it's 100 elo stronger for Diep in blitz. It's 300 elo stronger in diep-diep. It's 60 elo weaker if i play against other programs.

If i enable multicut. It's a lot of elo stronger at fast time controls single core and especially in diep - diep, and at slower time controls it's a LOT weaker against other programs. Most importantly i noticed that if i enable it, it searches a ply deeper. That matters most 10 to 12 ply.

When diep already gets above 14 ply search depth as a minimum, then multicut no longer gives elo.

Now you claim super-bullet time controls, and something that gives you 2 ply at 10 ply search depth, this for an engine that's gettingeasily 20-25 ply,
and you test komodo versus komodo.

That's not science.

Any claim that anything works for today game playing programs based upon superbullet time controls we cannot take serious. It's only an indication you need to seriously test it at slower time controls at most.

Yet with Diep there is many algorithms that when enabled seem to work at smaller search depths and very few that i can use at tournament time controls at multiple cores.

In your case your thing is single core as well. So you'll have plenty of cores left to do slower time control tests than the superbullet and you'll have plenty of opponents.

If you don't know what i'm speaking about, then i don't know what to think of you, except that such guys belong in the garbage can.

Using crap methods of testing and presenting evidence belongs in the 80s and 90s of computerchess. Not in 21th century.

Vincent

p.s. and it's about time that government also grows up and uses those methods in medicines - right now introducing psychological medicines you just need 200 persons where your new synthetical cocain drugs (such as ADHD medicine for children) needs to be shown 'working' with 95% sureness for just 24 hours or so - instead of 99.99% sureness for long term.

Maybe you can design a new medicine? As that's where the biggest crap scientists work.