Minimal network

[Henk]

So far I found that to learn only 4000 training examples I already need about 29 layers with 4 filters per layer in an ordinary convolutional neural network.

To fit more examples I expect more layers and filters. Probably I need to switch to residual network too when number of layers increases.

Overfitting should be prevented. But this time I'm only focussing on training error.


The more layers and filters the slower it gets.

[Henk]

To get a lower validation error ( I mean lower than first epoch) I needed at least about 3000 training examples in my ultra minimal network of two conv layers where each layer has only one filter. For instance 2000 training examples did not give a lower validation error. Maybe if you add noise you need less examples.

Cool information. Isn't it.

Code: Select all

0.304705596535846  0.299141061043657
0.304200726404863  0.296937026701584
0.303885715964357  0.29536382114373
0.303647085111191  0.294051130754107
0.303474092302082  0.292888186599641
0.303335064270266  0.291813712011117
0.303245155439217  0.290795499021344

By the way minimum validation for 3000 examples is about 0.302 for this network.

Of course no progress. I am not obsessed by movement.

Hmm then what about first epoch. How many training examples are needed to get any better validation error when using only one epoch. To be continued

[Henk]

Using only one training example gives a validation error of 0.305. Probably result depends on quality of weight initialization.

Here an error of 0.305 is equivalent to an error of about a rook (450 CP)

[Henk]

Minimum validation error of two layer conv using two filters per layer and 200K training examples appears to be about 0.153 in my network.
That would be equal to 240 centipawn (half rook, or bad bishop). So useless.

Increasing number of training examples did not help because 100K training examples gave 0.154

[Henk]

Validation error four layers one filter each does not go below val 0.2 using 100K training examples

[Henk]

Validation error four layers one filter each does not go below val 0.2 using 100K training examples

Increasing number of layers with only one filter did not help.
So you need at least one layer with two filters or more.

[Daniel Shawul]

So far I found that to learn only 4000 training examples I already need about 29 layers with 4 filters per layer in an ordinary convolutional neural network.

To fit more examples I expect more layers and filters. Probably I need to switch to residual network too when number of layers increases.

Overfitting should be prevented. But this time I'm only focussing on training error.


The more layers and filters the slower it gets.

Isn't the minimal network a single perceptron ? Most hand-written chess evaluation can be represented a perceptron with n
inputs and a single output for the evaluation (0 to 1).

I think you should pick a specific accuracy (say 70%) to define the minimum network size that would achieve it.
The validation set should be sperate from training set so that one could do several epochs to overfit the training set.

[Henk]

Validation error four layers one filter each does not go below val 0.2 using 100K training examples

Increasing number of layers with only one filter did not help.
So you need at least one layer with two filters or more.

3 filters did no go below 0.13
I skip 5 filters so trying now 7 filters. Each layer or trap uses a copy layer(connection) too. Like in residual network. I'm currently using only eight layers so residual network not necessary.

Downside is that network gets slower and slower.

[smatovic]

Downside is that network gets slower and slower.

1 TFLOPS for 50 bucks...
https://www.ebay.com/sch/i.html?_nkw=nvidia+GTX+750

--
Srdja

[Henk]

Downside is that network gets slower and slower.

1 TFLOPS for 50 bucks...
https://www.ebay.com/sch/i.html?_nkw=nvidia+GTX+750

--
Srdja

Yes i know they are not that expensive but I don't know how to install them in my current laptop. Actually I'm scared of opening computers. Maybe afterwards my laptop won't work anymore etc. Maybe if there is some fool proof method to install them. I don't know yet.