Saw the NNUE similarity test come up saying it doesn't work. I wouldn't give up on it so fast based on the small test I ran. Attaching the pngs.
EDIT: didn't work, and img tags aren't working either, but here are links.
https://pasteboard.co/JS2KIPb.png
https://paste.pics/BU2F9
Simex including NNUE
Moderator: Ras
-
gaard
- Posts: 464
- Joined: Mon Jun 07, 2010 3:13 am
- Location: Holland, MI
- Full name: Martin W
Re: Simex including NNUE
What is the format for the dendrogram input format? If I give you a data file can you produce one for me?jjoshua2 wrote: ↑Thu Mar 11, 2021 1:51 am Saw the NNUE similarity test come up saying it doesn't work. I wouldn't give up on it so fast based on the small test I ran. Attaching the pngs.
EDIT: didn't work, and img tags aren't working either, but here are links.
https://pasteboard.co/JS2KIPb.png
https://paste.pics/BU2F9
-
Ferdy
- Posts: 4853
- Joined: Sun Aug 10, 2008 3:15 pm
- Location: Philippines
Re: Simex including NNUE
See the dendrogram.csv in this repo for the input format.gaard wrote: ↑Fri Mar 12, 2021 12:18 amWhat is the format for the dendrogram input format? If I give you a data file can you produce one for me?jjoshua2 wrote: ↑Thu Mar 11, 2021 1:51 am Saw the NNUE similarity test come up saying it doesn't work. I wouldn't give up on it so fast based on the small test I ran. Attaching the pngs.
EDIT: didn't work, and img tags aren't working either, but here are links.
https://pasteboard.co/JS2KIPb.png
https://paste.pics/BU2F9
-
gaard
- Posts: 464
- Joined: Mon Jun 07, 2010 3:13 am
- Location: Holland, MI
- Full name: Martin W
Re: Simex including NNUE
Ferdy wrote: ↑Fri Mar 12, 2021 1:57 amSee the dendrogram.csv in this repo for the input format.gaard wrote: ↑Fri Mar 12, 2021 12:18 amWhat is the format for the dendrogram input format? If I give you a data file can you produce one for me?jjoshua2 wrote: ↑Thu Mar 11, 2021 1:51 am Saw the NNUE similarity test come up saying it doesn't work. I wouldn't give up on it so fast based on the small test I ran. Attaching the pngs.
EDIT: didn't work, and img tags aren't working either, but here are links.
https://pasteboard.co/JS2KIPb.png
https://paste.pics/BU2F9
Code: Select all
+---------------------fruit2.1
+-7
+--8 +--------------------loo0.32f
! !
+-13 +-----------------------gamfruit
! !
+-20 +---------------------------tog1.3x4
! !
! +---------------------------fru2.3.1
!
! +---------------------ryb0beta
! +-----4
! +----11 +----------------------strka1.8
! ! !
! ! +-------------------------ryb.3.2a
! !
! ! +-------------------------------glang2.2
! ! !
! ! ! +---------------------------------gla1.2.1
! ! ! +--24
! ! ! ! +-----------------------------------viper0.1
25-26 ! !
! ! ! ! +----------------------------------booot6.4
! ! ! ! +-15
! ! ! ! ! +---------------------------------komo14.1
! ! ! ! !
! ! ! ! ! +------------------------------draon1.0
! ! ! ! ! +---2
! ! ! ! ! ! ! +---------------------------fattz2.0
! +-29 ! +-16 +---3 +-1
! ! ! ! ! ! ! +----------------------------stoish13
! ! ! ! ! +-9 !
! ! ! ! ! ! ! +-----------------------------nemo6.00
! ! ! ! ! +-10 !
! ! ! +-17 ! ! ! +---------------------------------rub2.0.1
! ! ! ! ! +-14 !
! ! ! ! ! ! +--------------------------------ige2.9.0
! ! ! ! ! !
! ! ! +-18 ! +----------------------------------eth12.75
! ! ! ! ! !
! ! ! ! ! +-------------------------------laser1.7
! ! ! ! !
! ! ! +-19 ! +-------------------------------fire7.1
! +-36 ! ! +--12
! ! ! ! +---------------------------------fire8
! ! +-22 !
! ! ! ! +----------------------------------shrssic5
! ! ! !
! ! +-23 +----------------------------------senai2.0
! ! ! !
! ! ! ! +--------------------crir1.6a
! ! ! ! +---------5
! ! +-27 +-21 +----------------------houi1.5a
! ! ! ! !
! ! ! ! +--------------------------------pro1.9.0
! ! +-28 !
! ! ! ! +-----------------------------------spark1.0
! ! ! !
! ! ! +------------------------------------spike1.4
! ! +-39
! ! ! ! +------------------------brit0.2c
! ! ! ! +-----------6
! ! ! ! ! +-------------------------brit0.3a
! ! ! +-35
! ! ! ! +----------------------------------------cher2007
! ! ! +-34
! ! ! +--------------------------------------ganf6.01
! +-40
! ! +------------------------------------list5.12
! ! +-31
! ! ! +---------------------------------ruf2.1.0
! ! +-33
! ! ! ! +-------------------------------------sje12.13
! ! +-38 +-32
! ! ! ! +-----------------------------------yace
! ! ! !
! +-41 +--------------------------------------naum2.0
! !
! ! +------------------------------------shrder10
! +-42
! ! +---------------------------------------sos5
! +-37
! ! +--------------------------------zappa1.1
! +-30
! +---------------------------------zapexico
!
+-----------------------------boo.14.0
Fruit 2.1
Loop 10.32
Gambit Fruit 1.0
Toga II 1.3
Rybka 1.0 Beta
Strelka 1.8
Rybka 2.3.2a
Dragon 1.0
Fat Fritz 2.0
Stockfish 13.0
Nemorino 6.00
RubiChess 2.0.1
Igel 2.9.0
Ethereal 12.75
Fire 7.1
Fire 8
Critter 1.6a
Houdini 1.5a
Bright 0.2c
Bright 0.3a
Closest pairs:
Dragon 1.0
Stockfish 13
Fat Fritz 2.0
Nemorino 6.00
Solutions:
Positions:
-
gaard
- Posts: 464
- Joined: Mon Jun 07, 2010 3:13 am
- Location: Holland, MI
- Full name: Martin W
Re: Simex including NNUE
Interestingly, Dragon/Stockfish/Fat Fritz are the closest by very small margins.gaard wrote: ↑Fri Mar 12, 2021 3:50 amFerdy wrote: ↑Fri Mar 12, 2021 1:57 amSee the dendrogram.csv in this repo for the input format.gaard wrote: ↑Fri Mar 12, 2021 12:18 amWhat is the format for the dendrogram input format? If I give you a data file can you produce one for me?jjoshua2 wrote: ↑Thu Mar 11, 2021 1:51 am Saw the NNUE similarity test come up saying it doesn't work. I wouldn't give up on it so fast based on the small test I ran. Attaching the pngs.
EDIT: didn't work, and img tags aren't working either, but here are links.
https://pasteboard.co/JS2KIPb.png
https://paste.pics/BU2F9Using 14 units as the benchmark, families are:Code: Select all
+---------------------fruit2.1 +-7 +--8 +--------------------loo0.32f ! ! +-13 +-----------------------gamfruit ! ! +-20 +---------------------------tog1.3x4 ! ! ! +---------------------------fru2.3.1 ! ! +---------------------ryb0beta ! +-----4 ! +----11 +----------------------strka1.8 ! ! ! ! ! +-------------------------ryb.3.2a ! ! ! ! +-------------------------------glang2.2 ! ! ! ! ! ! +---------------------------------gla1.2.1 ! ! ! +--24 ! ! ! ! +-----------------------------------viper0.1 25-26 ! ! ! ! ! ! +----------------------------------booot6.4 ! ! ! ! +-15 ! ! ! ! ! +---------------------------------komo14.1 ! ! ! ! ! ! ! ! ! ! +------------------------------draon1.0 ! ! ! ! ! +---2 ! ! ! ! ! ! ! +---------------------------fattz2.0 ! +-29 ! +-16 +---3 +-1 ! ! ! ! ! ! ! +----------------------------stoish13 ! ! ! ! ! +-9 ! ! ! ! ! ! ! ! +-----------------------------nemo6.00 ! ! ! ! ! +-10 ! ! ! ! +-17 ! ! ! +---------------------------------rub2.0.1 ! ! ! ! ! +-14 ! ! ! ! ! ! ! +--------------------------------ige2.9.0 ! ! ! ! ! ! ! ! ! +-18 ! +----------------------------------eth12.75 ! ! ! ! ! ! ! ! ! ! ! +-------------------------------laser1.7 ! ! ! ! ! ! ! ! +-19 ! +-------------------------------fire7.1 ! +-36 ! ! +--12 ! ! ! ! +---------------------------------fire8 ! ! +-22 ! ! ! ! ! +----------------------------------shrssic5 ! ! ! ! ! ! +-23 +----------------------------------senai2.0 ! ! ! ! ! ! ! ! +--------------------crir1.6a ! ! ! ! +---------5 ! ! +-27 +-21 +----------------------houi1.5a ! ! ! ! ! ! ! ! ! +--------------------------------pro1.9.0 ! ! +-28 ! ! ! ! ! +-----------------------------------spark1.0 ! ! ! ! ! ! ! +------------------------------------spike1.4 ! ! +-39 ! ! ! ! +------------------------brit0.2c ! ! ! ! +-----------6 ! ! ! ! ! +-------------------------brit0.3a ! ! ! +-35 ! ! ! ! +----------------------------------------cher2007 ! ! ! +-34 ! ! ! +--------------------------------------ganf6.01 ! +-40 ! ! +------------------------------------list5.12 ! ! +-31 ! ! ! +---------------------------------ruf2.1.0 ! ! +-33 ! ! ! ! +-------------------------------------sje12.13 ! ! +-38 +-32 ! ! ! ! +-----------------------------------yace ! ! ! ! ! +-41 +--------------------------------------naum2.0 ! ! ! ! +------------------------------------shrder10 ! +-42 ! ! +---------------------------------------sos5 ! +-37 ! ! +--------------------------------zappa1.1 ! +-30 ! +---------------------------------zapexico ! +-----------------------------boo.14.0
Fruit 2.1
Loop 10.32
Gambit Fruit 1.0
Toga II 1.3
Rybka 1.0 Beta
Strelka 1.8
Rybka 2.3.2a
Dragon 1.0
Fat Fritz 2.0
Stockfish 13.0
Nemorino 6.00
RubiChess 2.0.1
Igel 2.9.0
Ethereal 12.75
Fire 7.1
Fire 8
Critter 1.6a
Houdini 1.5a
Bright 0.2c
Bright 0.3a
Closest pairs:
Dragon 1.0
Stockfish 13
Fat Fritz 2.0
Nemorino 6.00
Solutions:
Positions:
The smallest difference:
Stockfish 13
Fat Fritz 2.0
-
Guenther
- Posts: 4718
- Joined: Wed Oct 01, 2008 6:33 am
- Location: Regensburg, Germany
- Full name: Guenther Simon
Re: Simex including NNUE
Martin, please add the condition(s) for the used sim data, that is essential (time - depth - positions), also the first sheet is not accessible.
Otherwise all this sim data is useless (it is useless anyway for various programs, which show wrong depths, do early pruning or whatever,
also timing must be checked in the log files first for irregularities).
-
gaard
- Posts: 464
- Joined: Mon Jun 07, 2010 3:13 am
- Location: Holland, MI
- Full name: Martin W
Re: Simex including NNUE
Positions:Guenther wrote: ↑Fri Mar 12, 2021 8:46 amMartin, please add the condition(s) for the used sim data, that is essential (time - depth - positions), also the first sheet is not accessible.
Otherwise all this sim data is useless (it is useless anyway for various programs, which show wrong depths, do early pruning or whatever,
also timing must be checked in the log files first for irregularities).
polyglot epd-test -epd pos.epd -min-depth 1 -max-time 1.0 -depth-delta 64
All engines single threaded with opening books off. All else default. No tablebases. i7 9750H.
PolyGlot output:
-
Guenther
- Posts: 4718
- Joined: Wed Oct 01, 2008 6:33 am
- Location: Regensburg, Germany
- Full name: Guenther Simon
Re: Simex including NNUE
Thanks for the data!gaard wrote: ↑Fri Mar 12, 2021 10:53 pmPositions:Guenther wrote: ↑Fri Mar 12, 2021 8:46 amMartin, please add the condition(s) for the used sim data, that is essential (time - depth - positions), also the first sheet is not accessible.
Otherwise all this sim data is useless (it is useless anyway for various programs, which show wrong depths, do early pruning or whatever,
also timing must be checked in the log files first for irregularities).
polyglot epd-test -epd pos.epd -min-depth 1 -max-time 1.0 -depth-delta 64
All engines single threaded with opening books off. All else default. No tablebases. i7 9750H.
PolyGlot output:
It seems you took a different approach than the simtest and you allowed up to 1000ms in case no solution was found for your epd.
It is easy to extract the data from the polyglot solution files, but I still don't know how, or with what you processed the data then?
If I process the data from the polyglot files to the usual move arrays used in simex2 and run it through simex2 it seems I get
exactly inverted results - only checked one pair though?
(also simex2 gives only 2 digits precision, while you have 3 digits in your infile.csv)
For this I used SF13 and Sjeng 12.13, because I was surprised by the high number shown between them from your file.
(which I assumed to be the moves correlation as usual)
It says 62.806 in your table, while I get 37.20 for them? So do the numbers mean the opposite as normally used with simex2?
I guess you know that there are several drawbacks with certain engines, e.g. usually Booot 6.4 only reports after around depth 12,
no matter if the move was found or not. Rybka only reports from depth 3 and 4 on, also some programs give irregular output
for some moves w/o any move(ofc a very tiny subset of all positions, so statistically neglectable).
Code: Select all
sjeng_12.13,48.750,58.278,52.972,53.611,53.944,53.750,62.444,58.361,62.056,55.528,58.028,49.611,51.028,50.528,54.000,52.250,48.500,55.278,57.528,57.556,54.861,49.472,50.528,51.194,60.000,55.111,58.972,46.944,52.556,52.167,54.167,51.500,55.194,-----,52.361,54.361,52.306,62.806,51.139,50.611,52.222,48.028,48.639,51.417
sos_5,49.000,55.833,51.694,52.333,54.389,52.167,59.389,55.361,59.917,53.444,55.667,49.000,49.833,49.000,53.667,51.083,49.278,53.722,55.278,54.722,54.167,53.028,50.139,52.250,57.333,51.222,57.278,49.389,51.028,51.583,53.028,50.000,54.361,52.361,-----,52.389,53.222,60.139,50.389,51.361,53.528,50.028,49.278,48.833
spark_1.0,49.167,50.500,49.194,48.667,51.000,45.750,56.694,51.556,55.361,51.611,51.444,47.361,46.306,47.639,51.028,50.306,48.583,46.111,53.111,51.333,48.556,51.639,46.722,51.472,53.500,47.167,54.139,49.972,48.194,45.583,48.500,49.639,48.750,54.361,52.389,-----,47.528,55.778,49.250,47.528,52.917,52.056,50.139,47.389
spike_1.4,50.361,51.694,51.444,50.417,51.917,47.139,57.500,52.639,55.778,51.222,52.056,49.917,48.472,50.944,52.389,51.194,47.000,48.194,52.806,50.639,49.528,50.667,49.250,50.528,53.944,48.583,53.333,50.083,50.194,49.139,48.500,50.389,48.583,52.306,53.222,47.528,-----,57.583,50.750,50.667,52.000,51.722,51.167,49.611
stockfish_13,60.306,51.806,60.167,59.694,60.278,52.472,39.556,47.528,37.056,53.722,51.361,60.333,59.167,59.639,60.722,59.389,58.722,52.444,46.778,48.167,49.889,59.806,59.500,58.972,42.056,53.333,45.806,58.917,57.000,57.750,53.139,58.194,52.750,62.806,60.139,55.778,57.583,-----,59.194,59.389,62.944,61.250,60.083,57.750
strelka_1.8,41.500,54.111,47.667,48.611,53.639,46.694,59.556,55.028,58.972,53.806,55.833,39.972,41.361,40.833,50.472,50.278,43.778,47.139,54.917,53.833,51.361,51.944,39.306,50.222,56.694,49.972,55.278,48.139,29.028,36.583,50.222,49.000,53.611,51.139,50.389,49.250,50.750,59.194,-----,42.444,50.833,50.306,45.444,47.611
toga_ii_1.3x4,37.944,54.528,46.222,48.417,52.611,45.944,59.417,55.778,59.333,52.778,55.611,34.806,38.472,36.222,49.028,47.083,41.722,46.639,54.694,53.500,52.833,50.639,33.389,50.361,55.889,47.889,55.667,49.056,42.306,40.444,50.722,47.139,52.861,50.611,51.361,47.528,50.667,59.389,42.444,-----,48.167,49.250,45.944,46.389
viper_0.1,46.889,58.333,48.250,50.806,52.750,52.861,63.000,58.722,62.167,57.278,59.139,46.722,47.556,49.111,52.556,45.278,48.306,53.917,57.611,58.139,55.278,51.250,47.444,52.500,60.167,54.361,59.028,48.972,51.833,50.639,55.861,52.528,56.278,52.222,53.528,52.917,52.000,62.944,50.833,48.167,-----,49.556,49.806,51.778
yace,47.889,56.750,50.500,51.778,54.806,51.528,61.472,56.722,60.278,55.972,57.222,47.917,49.306,50.000,51.083,49.417,47.806,53.500,55.389,56.333,53.889,47.722,49.111,49.111,58.056,53.917,57.722,45.917,52.500,51.722,52.389,48.028,54.611,48.028,50.028,52.056,51.722,61.250,50.306,49.250,49.556,-----,48.972,50.000
zappa_1.1,43.889,55.417,47.972,48.722,51.528,49.889,60.306,55.306,60.028,55.444,56.389,43.444,44.889,43.444,48.889,47.556,46.250,51.361,55.556,55.472,53.194,50.444,44.250,50.028,56.778,50.389,56.694,46.278,48.500,46.972,51.694,48.472,54.056,48.639,49.278,50.139,51.167,60.083,45.444,45.944,49.806,48.972,-----,44.111
zappa_mexico,46.944,52.528,50.028,49.306,51.833,47.111,57.111,53.222,57.778,51.306,53.250,45.778,45.611,45.833,49.528,48.361,47.056,48.528,52.778,51.889,49.250,50.861,46.056,51.556,54.361,47.250,54.611,47.861,48.194,46.000,49.389,47.500,49.694,51.417,48.833,47.389,49.611,57.750,47.611,46.389,51.778,50.000,44.111,-----Code: Select all
epd\pos.epd,3600,1000
Sjeng 1213,-----,37.20
Stockfish 13,37.20,------
gaard
- Posts: 464
- Joined: Mon Jun 07, 2010 3:13 am
- Location: Holland, MI
- Full name: Martin W
Re: Simex including NNUE
I noticed also that some engines, Komodo 14.1, for example, consistently go over the one second mark.
I've found the apps here very helpful, and neighbor.exe is what I used to make the dendrogram: https://evolution.genetics.washington.e ... -new1.html
It takes a distance matrix which is why you'd need to subtract the distance from 100 to get similarity (%), for use in other applications.
I've found the apps here very helpful, and neighbor.exe is what I used to make the dendrogram: https://evolution.genetics.washington.e ... -new1.html
It takes a distance matrix which is why you'd need to subtract the distance from 100 to get similarity (%), for use in other applications.