Disabling Null Move Pruning in Stockfish

[bob]

What about test with 2 threads at 60 or 120 sec ?

Null-move is pretty consistent across all depths. If a program uses LMR, completely disabling null-move will cost something around 40-50 Elo. Using Uri's idea is going to cost. I'm running a cluster test to figure out exactly how much it will cost.

[zullil]

Since null-move is pretty much a standard idea, I can easily do a test on my cluster with Crafty. I can disable null-move at several depths and plot the decay of Elo easily enough.

I'll report the results back later.

Will be interesting to see. Thanks for devoting the time and resources to this.

[phenri]

Since null-move is pretty much a standard idea, I can easily do a test on my cluster with Crafty. I can disable null-move at several depths and plot the decay of Elo easily enough.

I'll report the results back later.

Will be interesting to see. Thanks for devoting the time and resources to this.

+1
Thanks

[Uri Blass]

Since null-move is pretty much a standard idea, I can easily do a test on my cluster with Crafty. I can disable null-move at several depths and plot the decay of Elo easily enough.

I'll report the results back later.

Will be interesting to see. Thanks for devoting the time and resources to this.

+1
Thanks

+2

Note only that tests can prove that my theory is right but cannot refute my theory because it is always possible that Crafty simply does not get depth that is big enough to earn elo from disabling null move pruning
at the time control that Bob is going to use and at fast time control the difference should be very close to 0(the only demage for fast time control that crafty does not get 20 plies is that it is slightly slower to check if depth is smaller than 20 in every iteration but this demage is less than 1 elo).

[syzygy]

I know that people are going to blame me that I waste resources if I start testing at 10 minutes per game so I do not do it.

Why don't you just test all those crazy ideas on your own hardware? Then nobody will stop you or complain about it. Use of the testing framework that is provided by other people for free should be made only responsibly imho.

[Uri Blass]

I know that people are going to blame me that I waste resources if I start testing at 10 minutes per game so I do not do it.

Why don't you just test all those crazy ideas on your own hardware? Then nobody will stop you or complain about it. Use of the testing framework that is provided by other people for free should be made only responsibly imho.

I do not understand what is your problem with me.

I replied to the question if I tested my ideas in the stockfish framework and explained why I did not even try to do it.

Note that I gave some productive patchs for stockfish(mainly in improving the time management) so I do not understand the hostile treatement against me when I only explain why I did not test what I want to test.

There are other people(without mentioning names) who submitted clearly many illogical patchs but I did not see that they get the hostile treatment that I got from you and some other people.

Note also that the queue is almost empty so it is not that my patchs blocked productive patchs espacially when I put some of my patchs about finding the max ply that stockfish needs with low priority and I believe that it is possible that the results may help to find better patchs later.

Edit:You know that I cannot test my ideas about long time control in my hardware because I cannot get enough games in a reasonable time
and calling my ideas crazy ideas is insulting

Thinking about it maybe I also was wrong to call the reductions of stockfish in null move pruning that prevent stockfish to solve some zugzwang positions crazy reductions but at least I did not do it as a direct reply to the person who suggested the reductions and I had no plan to insult him and it simply seems to me that null move reductions that prevent stockfish to find the best move even at depth 1000 are not logical for long time control even if they are good in testing for 1 minute per game.

[syzygy]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

[Greg Strong]

If absolutely nobody but you can see the point in a particular test (such as those where you want to measure the effect of too low values of MAX_PLY... I don't see anyone protesting Marco's exclamation that those are "madness"), then maybe you should just consider the possibility that the others are right and that you might be wrong. And if you then still cannot see the problem, maybe just accept anyway that it is better to run such tests privately and not irritate everybody else.

I have nothing against your good ideas, but madness is madness.

I think the idea has merit and see the point in this test. I assume Dr. Hyatt does also, since he's devoting his cluster resources to trying it. Others have also expressed interest. Uri's not utilizing the stockfish test system resources for this idea so what's your problem? Your claim of 'madness' is also based on no data.

[bob]

Since null-move is pretty much a standard idea, I can easily do a test on my cluster with Crafty. I can disable null-move at several depths and plot the decay of Elo easily enough.

I'll report the results back later.

Will be interesting to see. Thanks for devoting the time and resources to this.

Here's the key. My first run for the "depth limit" was 20, 15, 10 and 6. 20 is irrelevant for the most part because this was a quick test at 20sec+0.1sec. Not going to see 20 ply searches very often until the endgame where null-move is restricted anyway.

In any case, what I found was that 20 and 15 made no difference whatever, as expected, since the depth rarely reaches that. 10 started to show a loss, dropping about 3 Elo (error bar = +/- 3 here, so take that with a grain of salt. the 6 value dropped about 20 Elo however, which is certainly serious.

30K games at this time control take under an hour. I could try 1+1 or longer, but those stretch out to several hours per run...

My take on this is it is not worth the trouble. Null-move has good and bad. Good outweighs the bad significantly. Limiting it based on depth clearly hurts performance, just from a logical thinking process. I think it is a case of "I'll take the warts along with the good parts". Eliminate a wart and you eliminate part of the advantage in other positions.

[bob]

Since null-move is pretty much a standard idea, I can easily do a test on my cluster with Crafty. I can disable null-move at several depths and plot the decay of Elo easily enough.

I'll report the results back later.

Will be interesting to see. Thanks for devoting the time and resources to this.

+1
Thanks

+2

Note only that tests can prove that my theory is right but cannot refute my theory because it is always possible that Crafty simply does not get depth that is big enough to earn elo from disabling null move pruning
at the time control that Bob is going to use and at fast time control the difference should be very close to 0(the only demage for fast time control that crafty does not get 20 plies is that it is slightly slower to check if depth is smaller than 20 in every iteration but this demage is less than 1 elo).

I can test about any time-control we choose. I can basically play about 540 games at a time. So make a suggestion for something that is actually doable. I think I can run a 5min+5sec match in 24 hours or so.

So, tell me a good time control, and a depth limit or two to try and I can run a test 3 times, once with normal null-move, once with each of the two depth limits you'd like to see.