I came to the conclusion that NICE is 95% programmer tool, hence I post it here.
Version 1.2 is a significant improvement.
1. NICE now can make predictions with with a precision of 1 elo. The idea to invest 20 minutes before you put an engine change into cute-chess looks attractive to me when NICE can report an elo gain or regression with a precision of 1 elo. It's to hope NICE turns out to be a good prophet
2. Reporting errors - Some engines (in fact a bit more than some) have the habit to surpress the mainlines of the early plies, some hide the first x plies, some hide the main lines the first 100ms and there are even those who use 1 full second. It's probably inspired by a speed gain of 1-2 millieseconds. When there is no main line NICE will report the number of cases found in the error column of the html and the result will be unreliable.
3. The time-bonus feature has been dropped. Someone said, it only adds noise. After a good look I had to conclude it is even worse, without it the results are more reliable.
http://rebel13.nl/misc/nice.html
NICE 1.2
Moderators: hgm, Rebel, chrisw
-
- Posts: 2559
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: NICE 1.2
thanks Ed, that's really Nice
is there a way to utilize multiple threads (actually processes)? I only see 1 core utilized (1 engine instance)
also, how accurately can Nice detect say search improvements?
is there a way to utilize multiple threads (actually processes)? I only see 1 core utilized (1 engine instance)
also, how accurately can Nice detect say search improvements?
Martin Sedlak
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: NICE 1.2
Edit the batch file you plan to use:
set THREADS=1
I don't know. Well, yet... as most of the time I tested eval changes.also, how accurately can Nice detect say search improvements?
I suppose there is no real difference between the two, tests at 100ms, 250ms, 500ms and 1000ms show significant elo increments. I think the golden rule is to use a higher time control testing search changes.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 2559
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: NICE 1.2
Then I misunderstood you. I tried Cheng 4.39, created test.bat
Code: Select all
del debug*.*
set MT=100
set HASH=128
set THREADS=1
set PROTOCOL=uci
set EPD=epd\sf1.epd
set OPTIONS=MultiPV=1
set MRL=x1
set EXE=engines\Cheng4_4.39.exe
set NAME=Cheng4_4.39
mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT% --log --eoption %OPTIONS%
csv %MRL% %EPD% %OPTIONS% %NAME% %MT% %CCRL%
mrl %MRL% %MRL% %EPD% %MT%
exit
2. Select "Split Run"
3. Select "test.bat"
4. Speed up by threads -> I typed 8
5. Exit program
6. Double click "@start.bat"
The "sf1.epd" is split into 8 parts of 5,000 position and runs in 8 threads.
My task manager shows 8 x Cheng.
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 2559
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: NICE 1.2
Oh, my bad I only ran the original batch file directly, didn't run nice.exe to generate the split batch, sorry
Martin Sedlak
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: NICE 1.2
50 push-ups or one hour in the corner
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1357
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
Re: NICE 1.2
Well I'm stumped. Either I can't make it work with Myrddin (which is an xboard engine) or NICE doesn't support xboard. Has anybody had success using NICE with any xboard engine?
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: NICE 1.2
That is a question for Ferdy.
Meanwhile I did a quick run with Myrddin using the 1500 STS positions using the wb2uci adapter.
http://rebel13.nl/mea/Myrddin.html
Here is how.
The wb2uci.eng file
Code: Select all
[ENGINE]
Name = Myrddin_0.87
Author = John Merlino
Filename = wb2uci.exe
[OPTIONS]
Program = Myrddin_0.87.exe
InitString = BookOff/n
LevelExtend = none
Analyze = true
UseUndo = false
Logfile = false
LevelType = 2
SimulateHint = true
TerminateHard = false
; In WIN XP or Win NT TerminateHard = true
Protocol = 1
Visible = Ponder,Extras
Code: Select all
del debug*.*
set MT=1000
set HASH=128
set THREADS=1
set PROTOCOL=uci
set EPD=epd\sts.epd
set OPTIONS=MultiPV=1
set MRL=Myrddin
set EXE=engines\wb2uci.exe
set NAME=Myrddin_0.87
mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT% --log --eoption %OPTIONS%
csv %MRL% %EPD% %OPTIONS% %NAME% %MT% %CCRL%
mrl %MRL% %MRL% %EPD% %MT%
exit
90% of coding is debugging, the other 10% is writing bugs.
-
- Posts: 1357
- Joined: Wed Mar 08, 2006 10:15 pm
- Location: San Francisco, California
Re: NICE 1.2
It works - AND I found a (possible) bug in Myrddin! It does not accept "st" command values less than 1, because the protocol says it is used to "set the exact number of seconds per move". I guess, when I implemented that more than 10 years ago, I never considered decimal values. But the text of the protocol (and the example, which only uses the number 30) should probably be updated.
However, trying to do a Split Run via Myrddin.bat in NICE fails with "Error in line 8".
Thanks very much!
However, trying to do a Split Run via Myrddin.bat in NICE fails with "Error in line 8".
Thanks very much!