Null Move Help

[stevemulligan]

I'm trying to implement null move pruning and I'm not sure how to test it. I want to determine if my simple implementation is actually working, or if it's pruning too much.

eg, null move off I get f4f5 d6f6 f2g1 (score is 56)

and with null move on the best moves are e4e5 d6d5 f2g1 (score is 48)

Is it normal to see entirely different lines with scores that are worse? Or should null move on return the same score and best moves more often than not?

[Sven]

I'm trying to implement null move pruning and I'm not sure how to test it. I want to determine if my simple implementation is actually working, or if it's pruning too much.

eg, null move off I get f4f5 d6f6 f2g1 (score is 56)

and with null move on the best moves are e4e5 d6d5 f2g1 (score is 48)

Is it normal to see entirely different lines with scores that are worse? Or should null move on return the same score and best moves more often than not?

Depends on your testing conditions. When using time control, correctly doing nullmove will let your engine search deeper, and so it will eventually choose a different best move. So you might want to test with fixed depth and compare whether nullmove keeps your final PV intact, which should be the goal since the main idea of nullmove pruning is to reduce the tree size. In this case, not matching PVs can be caused by
a) a bug,
b) a zugzwang situation that you don't handle, or
c) some hash table effect.

If you use nullmove also for other purposes, e.g. to detect threats, then it could have a regular impact on the PV.

Sven

[micron]

NMP, like most modifications to an engine, changes the search tree and so it is entirely normal to get a different score and PV.

NMP should give you a big improvement in time-to-depth (or nodes at given depth). In my engine NMP reduces those metrics to a half at low depths and a quarter in deep searches.

In self-play NMP should be worth roughly 100 Elo, easily confirmed in a fast match with only a few hundred games.

Robert P.

[stevemulligan]

I setup cutechess-cli for the first time. What are good time control settings for self-play testing? I read somewhere I need like 1000 rounds to get an accurate ELO, is that true for self play as well?

This is what I'm using now (40 moves in 60 seconds):

cutechess-cli -engine conf=Pwned1 -engine conf=Pwned2 -each tc=40/60 -rounds 1000 -pgnout selfplay.pgn

engines.json

Code: Select all

[
    {
    "name": "Pwned1",
	"command": "pwned_console.exe",
	"protocol": "xboard",
	"workingDirectory": "C:\\Source\\pwned\\pwned_console\\bin\\Debug",
	"initStrings": ["nullmove"]
    },
	{
    "name": "Pwned2",
	"command": "pwned_console.exe",
	"protocol": "xboard",
	"workingDirectory": "C:\\Source\\pwned\\pwned_console\\bin\\Debug",
	"initStrings": []
    }
]

(nullmove is the engine command to enable null move pruning/ordering)

[lucasart]

I setup cutechess-cli for the first time. What are good time control settings for self-play testing? I read somewhere I need like 1000 rounds to get an accurate ELO, is that true for self play as well?

Depends what you're testing and how fast your engine is. I use 6"+0.1", which is very fast, although I wouldn't recommend it for an engine that isn't fast and mature enough

[stevemulligan]

Sorry I'm not that familiar with terms yet. What does 6"+0.1" look like in xboard syntax?

[tmokonen]

6 second base, with a tenth of a second (100 millisecond) increment.

level 0 0:6 0.1

is what Arena gives me if I set a 6 second base, and 100 millisecond increment.

[lucasart]

6 second base, with a tenth of a second (100 millisecond) increment.

level 0 0:6 0.1

is what Arena gives me if I set a 6 second base, and 100 millisecond increment.

using a GUI like Arena for such testing is a very bad idea. You can be sure there will be a ton of time losses that are 100% attributable to Arena, because it's way to slow to handle such time controls. Besides, you're not taking advantage of multiple cpus. I have a duo core and my program is single threaded so with cutechess-cli I run 2 games in parralel. For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOP

[elpapa]

6 second base, with a tenth of a second (100 millisecond) increment.

level 0 0:6 0.1

is what Arena gives me if I set a 6 second base, and 100 millisecond increment.

using a GUI like Arena for such testing is a very bad idea. You can be sure there will be a ton of time losses that are 100% attributable to Arena, because it's way to slow to handle such time controls. Besides, you're not taking advantage of multiple cpus. I have a duo core and my program is single threaded so with cutechess-cli I run 2 games in parralel. For fast and parralel testing, cutechess-cli is beats the crap out of any GUI (precisely because it is not a *G*UI), and it also works easily with shell or python scripting. A great example is the combinaison of cutechess-cli and CLOP

Oh dear, I think Tony will explode this time. He didn't make a point of using Arena. He was just explaining what the time settings meant.

[tmokonen]

Oh dear, I think Tony will explode this time. He didn't make a point of using Arena. He was just explaining what the time settings meant.

Yeah, what he said.... 'twas a copy and paste from an output log, nothing more. Lucas does have a point with using command line tools, though. Haven't tried CLOP yet for myself, but it looks like an interesting concept. Maybe I could use it to try tune material values for variants.