I'm trying to implement null move pruning and I'm not sure how to test it. I want to determine if my simple implementation is actually working, or if it's pruning too much.
eg, null move off I get f4f5 d6f6 f2g1 (score is 56)
and with null move on the best moves are e4e5 d6d5 f2g1 (score is 48)
Is it normal to see entirely different lines with scores that are worse? Or should null move on return the same score and best moves more often than not?
Null Move Help
Moderators: hgm, Rebel, chrisw
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
-
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
Re: Null Move Help
Depends on your testing conditions. When using time control, correctly doing nullmove will let your engine search deeper, and so it will eventually choose a different best move. So you might want to test with fixed depth and compare whether nullmove keeps your final PV intact, which should be the goal since the main idea of nullmove pruning is to reduce the tree size. In this case, not matching PVs can be caused bystevemulligan wrote:I'm trying to implement null move pruning and I'm not sure how to test it. I want to determine if my simple implementation is actually working, or if it's pruning too much.
eg, null move off I get f4f5 d6f6 f2g1 (score is 56)
and with null move on the best moves are e4e5 d6d5 f2g1 (score is 48)
Is it normal to see entirely different lines with scores that are worse? Or should null move on return the same score and best moves more often than not?
a) a bug,
b) a zugzwang situation that you don't handle, or
c) some hash table effect.
If you use nullmove also for other purposes, e.g. to detect threats, then it could have a regular impact on the PV.
Sven
-
- Posts: 155
- Joined: Mon Feb 15, 2010 9:33 am
- Location: New Zealand
Re: Null Move Help
NMP, like most modifications to an engine, changes the search tree and so it is entirely normal to get a different score and PV.
NMP should give you a big improvement in time-to-depth (or nodes at given depth). In my engine NMP reduces those metrics to a half at low depths and a quarter in deep searches.
In self-play NMP should be worth roughly 100 Elo, easily confirmed in a fast match with only a few hundred games.
Robert P.
NMP should give you a big improvement in time-to-depth (or nodes at given depth). In my engine NMP reduces those metrics to a half at low depths and a quarter in deep searches.
In self-play NMP should be worth roughly 100 Elo, easily confirmed in a fast match with only a few hundred games.
Robert P.
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
Re: Null Move Help
I setup cutechess-cli for the first time. What are good time control settings for self-play testing? I read somewhere I need like 1000 rounds to get an accurate ELO, is that true for self play as well?
This is what I'm using now (40 moves in 60 seconds):
cutechess-cli -engine conf=Pwned1 -engine conf=Pwned2 -each tc=40/60 -rounds 1000 -pgnout selfplay.pgn
engines.json
(nullmove is the engine command to enable null move pruning/ordering)
This is what I'm using now (40 moves in 60 seconds):
cutechess-cli -engine conf=Pwned1 -engine conf=Pwned2 -each tc=40/60 -rounds 1000 -pgnout selfplay.pgn
engines.json
Code: Select all
[
{
"name": "Pwned1",
"command": "pwned_console.exe",
"protocol": "xboard",
"workingDirectory": "C:\\Source\\pwned\\pwned_console\\bin\\Debug",
"initStrings": ["nullmove"]
},
{
"name": "Pwned2",
"command": "pwned_console.exe",
"protocol": "xboard",
"workingDirectory": "C:\\Source\\pwned\\pwned_console\\bin\\Debug",
"initStrings": []
}
]
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: Null Move Help
Depends what you're testing and how fast your engine is. I use 6"+0.1", which is very fast, although I wouldn't recommend it for an engine that isn't fast and mature enoughstevemulligan wrote:I setup cutechess-cli for the first time. What are good time control settings for self-play testing? I read somewhere I need like 1000 rounds to get an accurate ELO, is that true for self play as well?
-
- Posts: 117
- Joined: Wed Jul 20, 2011 2:54 pm
- Location: Ottawa, Canada
Re: Null Move Help
Sorry I'm not that familiar with terms yet. What does 6"+0.1" look like in xboard syntax?
-
- Posts: 1296
- Joined: Sun Mar 12, 2006 6:46 pm
- Location: Kelowna
- Full name: Tony Mokonen
Re: Null Move Help
6 second base, with a tenth of a second (100 millisecond) increment.
level 0 0:6 0.1
is what Arena gives me if I set a 6 second base, and 100 millisecond increment.
level 0 0:6 0.1
is what Arena gives me if I set a 6 second base, and 100 millisecond increment.
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: Null Move Help
using a GUI like Arena for such testing is a very bad idea. You can be sure there will be a ton of time losses that are 100% attributable to Arena, because it's way to slow to handle such time controls. Besides, you're not taking advantage of multiple cpus. I have a duo core and my program is single threaded so with cutechess-cli I run 2 games in parralel. For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOPtmokonen wrote:6 second base, with a tenth of a second (100 millisecond) increment.
level 0 0:6 0.1
is what Arena gives me if I set a 6 second base, and 100 millisecond increment.
-
- Posts: 211
- Joined: Sun Jan 18, 2009 11:27 pm
- Location: Sweden
- Full name: Patrik Karlsson
Re: Null Move Help
Oh dear, I think Tony will explode this time. He didn't make a point of using Arena. He was just explaining what the time settings meant.lucasart wrote:using a GUI like Arena for such testing is a very bad idea. You can be sure there will be a ton of time losses that are 100% attributable to Arena, because it's way to slow to handle such time controls. Besides, you're not taking advantage of multiple cpus. I have a duo core and my program is single threaded so with cutechess-cli I run 2 games in parralel. For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOPtmokonen wrote:6 second base, with a tenth of a second (100 millisecond) increment.
level 0 0:6 0.1
is what Arena gives me if I set a 6 second base, and 100 millisecond increment.
-
- Posts: 1296
- Joined: Sun Mar 12, 2006 6:46 pm
- Location: Kelowna
- Full name: Tony Mokonen
Re: Null Move Help
Yeah, what he said.... 'twas a copy and paste from an output log, nothing more. Lucas does have a point with using command line tools, though. Haven't tried CLOP yet for myself, but it looks like an interesting concept. Maybe I could use it to try tune material values for variants.elpapa wrote: Oh dear, I think Tony will explode this time. He didn't make a point of using Arena. He was just explaining what the time settings meant.