xr_a_y wrote: ↑Thu Mar 11, 2021 12:09 pm
What is called "Minic" here is a net if I understand well. But is it "Napping Nexus" or "Nascent Nutrient" ?
The first one is based on SF data, the second on Minic (the engine) ones.
Ed can answer, because the names listed are the filenames he used to save the data. I think, not entirely sure, he took NNs and ran them within SF12 and SF13, ie he’s trying to compare nets not search. There seem to be three mimic connected nets, which he called Minic, nascent and nexus. Does that figure?
ok I missed the "nexus" and "nascent" in here. So my guess is that the net called "Minic" here is something else. Maybe the good SV net I was using at first.
Ah, yes, thanks, that’s how I thought it was, but good to confirm.
Can you detail a little on how nexus and nascent were trained, net characteristics and so on? I try to build up some anecdotal data points on net character against our RMS and pearsonr results.
Just to say few words about Orion 0.7 (the site states that "Orion 0.7 - From the information on the website the nnue origin of version 0.7 is unclear, the RMS implies a strong correlation with SF12.") : the site should mention "Orion 0.7 NNUE" and not "Orion 0.7".
In fact, "Orion 0.7" embeds a traditional hand-crafted evaluation function, the latter being entirely replaced by the NNUE network of SF12 (hence the high similarity observed) during my experiments to implement my own NNUE inference code. The resulting "experimental" version of Orion was then named "Orion 0.7 NNUE".
Kind regards from France
Thanks! When Ed sends me more data (including Orion results) which I don’t have, I’ll add them to the list. Presumably v0.7 will give high pearsonr and v0.8 low?
xr_a_y wrote: ↑Thu Mar 11, 2021 12:09 pm
What is called "Minic" here is a net if I understand well. But is it "Napping Nexus" or "Nascent Nutrient" ?
The first one is based on SF data, the second on Minic (the engine) ones.
Ed can answer, because the names listed are the filenames he used to save the data. I think, not entirely sure, he took NNs and ran them within SF12 and SF13, ie he’s trying to compare nets not search. There seem to be three mimic connected nets, which he called Minic, nascent and nexus. Does that figure?
ok I missed the "nexus" and "nascent" in here. So my guess is that the net called "Minic" here is something else. Maybe the good SV net I was using at first.
Ah, yes, thanks, that’s how I thought it was, but good to confirm.
Can you detail a little on how nexus and nascent were trained, net characteristics and so on? I try to build up some anecdotal data points on net character against our RMS and pearsonr results.
napping nexus (09/2020) : a Stockfish data based net build in many steps from 2 billions depth 10 data. Data are from Stockfish genfen. Learner used is the Nodchip repository one. Validation data are 1M depth 16 and 20 Stockfish data. With this net, MinicNNUE is around 100Elo weaker than with SV net.
nascent nutrient (10/2020) : a Minic data based net build from 400M random positions evaluated by Minic at depth 12. Data were generated using the Minic genfen feature and Minic random mover. The learner merged inside Minic is used in order to ensure Minic evaluation is used. Validation data are 1M depth 16 Minic data extracted from Minic2.48 games in pgn format. No RL with game outcome for this net. With this net, MinicNNUE is around 100Elo weaker than with napping nexus.
connor_mcmonigle wrote: ↑Thu Mar 11, 2021 1:19 am
Interesting results. If you're just directly applying RMSE to the raw low depth evaluations, you're likely picking up mostly on the scale of the evaluation rather than the uniqueness of the evaluations (which could explain the high similarity of the FF2 net assuming ChessBase's claims about its origins are true). A number of engines scale the output of the network to best match the evaluation scale expected by their search (static null move pruning conditions, etc.). To resolve this, I'd suggest first standardizing (subtract mean and divide by variance) the evaluation distributions to Normal(0, 1) before the taking RMSE. This metric should better correspond to what we mean by a unique evaluation function.
Yup, quite so. When we ran the results, I also programmed in the pearsonr, which I think basically does a scale and translate (slope and intercept) on the data and then measures the deviations. It wasn’t presented on Ed’s result page because last time I tried presenting pearsonr data it just got grunted at, but actually it’s probably a rather fine similarity measure for CC evals.
For example, two nets where one uses (in effect) 13359 for material and the other 2, 6, 6, 10, 18 are basically identical, but RMS won’t tell you. Pearsonr will.
I have only a subset of Ed's data, but here's an inverse sorted list of engine pairs with pearsonr.
0.94, 0.95 and above looks to be the value where quite strong net similarity shows up.
Pearson r, RMS, engine pair
======================
0.838 108.6 SF12-Igel-280 v SF12-nexus
0.854 105.8 SF12-Igel-270 v SF12-nexus
0.856 104.6 SF12-Igel-280 v SF12-nn-516f5b95189a
0.858 80.5 SF12-nn-803c91ad5c v _SF13
0.858 108.6 SF12-Igel-280 v SF12
0.859 104.7 SF12-Igel-280 v SF12-Minic
0.86 79.5 SF12-nn-803c91ad5c v SF13
0.864 93.7 SF12-nn-803c91ad5c v SF12
0.865 111.1 SF12-Igel-280 v _SF12
0.868 95.5 SF12-nn-803c91ad5c v _SF12
0.868 100.1 SF12-Igel-280 v SF12-nn-dd0c4c630f7e
0.87 101.2 SF12-Igel-280 v SF12-nn-0c6fc5ef48e1
0.871 89.1 SF12-Minic v SF12-nn-803c91ad5c
0.872 105.9 SF12-Igel-270 v SF12
0.873 101.9 SF12-Igel-270 v SF12-Minic
0.874 87.6 SF12-Igel-280 v SF13
0.875 88.5 SF12-Igel-280 v _SF13
0.875 108.8 SF12-Igel-270 v _SF12
0.875 88.1 SF12-nn-516f5b95189a v SF12-nn-803c91ad5c
0.875 101.3 SF12-Igel-270 v SF12-nn-516f5b95189a
0.876 60.7 SF12-nn-803c91ad5c v _FF2
0.881 93.0 SF12-Minic v SF12-nascent
0.883 96.0 SF12-nascent v SF12
0.884 97.3 SF12-Igel-270 v SF12-nn-dd0c4c630f7e
0.885 79.0 SF12-nascent v _SF13
0.885 84.6 SF12-nn-0c6fc5ef48e1 v SF12-nn-803c91ad5c
0.885 83.4 SF12-nn-803c91ad5c v SF12-nn-dd0c4c630f7e
0.886 78.0 SF12-nascent v SF13
0.886 98.3 SF12-Igel-270 v SF12-nn-0c6fc5ef48e1
0.886 85.0 SF12-Igel-270 v SF13
0.886 98.4 SF12-nascent v _SF12
0.887 86.0 SF12-Igel-270 v _SF13
0.89 91.0 SF12-nascent v SF12-nn-516f5b95189a
0.891 85.5 SF12-nexus v SF12-nn-803c91ad5c
0.892 76.2 SF12-nexus v _SF13
0.892 76.5 SF12-nexus v SF13
0.894 91.7 SF12-nascent v SF12-nexus
0.895 62.7 SF12-Igel-280 v _FF2
0.895 88.7 SF12-nascent v SF12-nn-0c6fc5ef48e1
0.895 53.6 SF12-Igel-280 v SF12-nn-803c91ad5c
0.898 78.9 SF12-nexus v _FF2
0.9 51.9 SF12-Igel-270 v SF12-nn-803c91ad5c
0.901 56.1 SF12-nascent v _FF2
0.901 86.3 SF12-nascent v SF12-nn-dd0c4c630f7e
0.902 60.6 SF12-Igel-270 v _FF2
0.905 75.4 SF12-nexus v _SF12
0.905 74.6 SF12-nexus v SF12
0.915 46.1 SF12-nascent v SF12-nn-803c91ad5c
0.916 69.3 SF12-Minic v SF12-nexus
0.922 66.1 SF12-nexus v SF12-nn-dd0c4c630f7e
0.923 65.8 SF12-nexus v SF12-nn-0c6fc5ef48e1
0.923 39.3 SF12-Igel-280 v SF12-nascent
0.925 38.6 SF12-Igel-270 v SF12-nascent
0.925 65.7 SF12-nexus v SF12-nn-516f5b95189a
0.928 69.6 SF12-Minic v _FF2
0.929 72.2 SF12 v _FF2
0.934 73.4 _FF2 v _SF12
0.934 60.6 SF12-Minic v SF13
0.934 67.4 SF12-nn-516f5b95189a v _FF2
0.935 59.6 SF12-Minic v _SF13
0.935 64.8 SF12-nn-dd0c4c630f7e v _FF2
0.938 65.0 SF12-nn-0c6fc5ef48e1 v _FF2
0.939 56.9 SF12-nn-dd0c4c630f7e v SF13
0.94 59.9 SF12 v SF13
0.941 61.0 SF13 v _SF12
0.941 58.8 SF12 v _SF13
0.942 55.5 SF12-nn-dd0c4c630f7e v _SF13
0.943 56.4 SF12-nn-516f5b95189a v SF13
0.944 55.7 SF12-nn-516f5b95189a v _SF13
0.945 58.9 _SF12 v _SF13
0.945 54.7 SF12-nn-0c6fc5ef48e1 v SF13
0.947 55.1 SF12-Minic v SF12-nn-516f5b95189a
0.947 54.8 SF12-Minic v SF12-nn-0c6fc5ef48e1
0.947 51.8 SF13 v _FF2
0.947 54.2 SF12-Minic v SF12-nn-dd0c4c630f7e
0.948 53.2 SF12-nn-0c6fc5ef48e1 v _SF13
0.948 55.0 SF12-nn-516f5b95189a v SF12
0.949 55.3 SF12-nn-516f5b95189a v _SF12
0.951 53.6 SF12-nn-0c6fc5ef48e1 v SF12
0.951 50.9 _FF2 v _SF13
0.952 52.8 SF12-nn-dd0c4c630f7e v SF12
0.954 26.1 SF12-Igel-270 v SF12-Igel-280
0.954 52.7 SF12-nn-dd0c4c630f7e v _SF12
0.954 52.6 SF12-Minic v _SF12
0.955 51.3 SF12-Minic v SF12
0.955 52.2 SF12-nn-0c6fc5ef48e1 v _SF12
0.956 49.7 SF12-nn-516f5b95189a v SF12-nn-dd0c4c630f7e
0.959 47.8 SF12-nn-0c6fc5ef48e1 v SF12-nn-516f5b95189a
0.96 47.0 SF12-nn-0c6fc5ef48e1 v SF12-nn-dd0c4c630f7e
0.975 33.9 SF13 v _SF13
0.977 37.4 SF12 v _SF12
(Just noticed I typed variance instead of sqrt variance in my original post. Whoops)
Ahh. Cool. Thanks for the information. Surprising that with standardization, we see that the FF2 network is of even greater similarity to the master SF network than even different nets from SV's training run... Hmm
Anyhow, I'm a little confused as to how the data is being presented on Ed's website. Are all the nets in Ed's test run inside SF? I don't see how this can be the case given Orion 0.8 is listed which, to the best if my knowledge, uses an incompatible file format. Same for Minic which now is using a somewhat modified version of Seer's network architecture and inference code which is definitely incompatible (weights aren't even quantized and topology differs).
In the case of Orion 0.8 and Seer v1.2.1, as both use a network trained on Stockfish's evaluation, but original training code and inference code, it would be interesting to see how similar their static evals are. I have some new networks intended for Seer v1.3 trained from "zero" using all original code and Seer's search only. I know David has been working on something similar so it would be quite interesting to test these nets as well for comparison and to establish a similarity threshold for network training data originality when using different training code+inference code.
connor_mcmonigle wrote: ↑Thu Mar 11, 2021 4:26 pm
Anyhow, I'm a little confused as to how the data is being presented on Ed's website. Are all the nets in Ed's test run inside SF? I don't see how this can be the case given Orion 0.8 is listed which, to the best if my knowledge, uses an incompatible file format. Same for Minic which now is using a somewhat modified version of Seer's network architecture and inference code which is definitely incompatible (weights aren't even quantized and topology differs).
Yes, I confirm that Orion 0.8 network has a different architecture than SF's one (mainly half number of parameters, and weights encoded as floats instead of integers).
connor_mcmonigle wrote: ↑Thu Mar 11, 2021 4:26 pm
In the case of Orion 0.8 and Seer v1.2.1, as both use a network trained on Stockfish's evaluation, but original training code and inference code, it would be interesting to see how similar their static evals are.
Very nice idea, and maybe a good way to measure the impact/efficiency of architecture or training method ?
connor_mcmonigle wrote: ↑Thu Mar 11, 2021 4:26 pm
I have some new networks intended for Seer v1.3 trained from "zero" using all original code and Seer's search only. I know David has been working on something similar (...).
Yes, I'm currently trying the "zero" approach, but I'm struggling with the second step of the process : I successfully trained a net based on a pure material evaluation function (~2150 elo), but when using this net as the new eval function, I'm failing to train a stronger new net. I restarted the process with more positions (32M instead of 16M), always searched at depth 8, but evaluating this (small) dataset is really long (~2 weeks). So, no major progress expected before a few... months ?!
Remarks:
1. nets are tested as much as possible with SF12 and SF13.
2. Nemorino and Orion 0.8 are the exceptions since they have a different file format.
3. The Orion 0.7 version has the exact SF12 net.
4. the folder ff2 contains the epd's of sf12, 13 and ff2 at 100ms, 250 and 500ms, made by someone else on a different pc.
5. epd's labelled with "sv" are the tested "sergio" nets.
6. epd's labelled with "sf" are Stockfish nets.
90% of coding is debugging, the other 10% is writing bugs.
Or is it Minic HCE? No it can't have a SIM of 72, no way. Must be MinicNNUE+SV net.
So the high similarity of what you call "Minic", that is in fact MinicNNUE+sv net..., is very logical, as this net is probably quite close to the SF12 one.
Really, naming convention here matters, be careful please. I don t think the text of your analysis about Minic in the web page is accurate enough.
Or is it Minic HCE? No it can't have a SIM of 72, no way. Must be MinicNNUE+SV net.
So the high similarity of what you call "Minic", that is in fact MinicNNUE+sv net..., is very logical, as this net is probably quite close to the SF12 one.
Really, naming convention here matters, be careful please. I don t think the text of your analysis about Minic in the web page is accurate enough.
Vivien, apologies, let's fix it. I think you are right.
3. Minic - we learn from its website the author in the beginning used Stockfish NNUE but later moved to an own net, it could explain our finding with the similar RMS to Stockfish.
How would you like me to change the text?
90% of coding is debugging, the other 10% is writing bugs.
To clarify, it will be helpful to call those Minic2.46 to 2.53 MinicNNUE+named net, if a net is being used.
I only really "assume" nnue technology starting from Minic 3.00 where the implementation was switchd to Seer one and architecture is not the same as SF at all. Now in the next Minic 3.05 to come, supported net has even diverged from Seer architecture.
To go back on the very high similarity here, I think we already discuss that back in September. We conclude evaluation, so here net in used, is much more important than search in similarity testing. So using here a SF data based net is clearly leading to high similarity. On the contrary, looking at a Minic data net (but still using the same architecture as the SF one) namely nascent nutrient, we get a smaller similarity.
So whatever the search is, seeing nets that have high similarity with SF probably indicates SF data were used to build the net.
The Igel nets at that time were Night Nurse 0.1 and 0.2b. They were trained from Bad Gyal data.
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
