The methods with quick convergence use to be prone to overfitting.(I am a kind of maniac about tuning, I spent last year trying different methods. zero elo of improvement, but a very funny year).The larger the training set, the less prone to overfitting and the longer it last to get results.so I'm looking for cheaper alternatives
it is possible to tune with a single run on a very large data set?
Let me give you an idea, as an mental experiment or implement it, if it looks fine to you. (I have already done, it is just as good/bad as any other method).
running votes with inertia.
Code: Select all
enum {
dont_change,increase,decrease
} VOTES;
double votes[MaxParameters][3];
for each position
for each feature parameter
if the eval is good enough (error < reference)
then
votes[i][dont_change]++;
else
if increasing the parameter reduce the error
then
votes[i][increase]++;
else
votes[i][decrease]++;
end if
end if
// criteria to award an increment.
ratio = votes[i][increase] - votes[i][decrease] / (votes[i][increase] + votes[i][decrease] + votes[i][dont_change])
if ratio > Ref_level then
increase parameter;
// to avoid bouncing up and down we do not reset votes to zero (inertia)
votes[i][increase] /= 2;
votes[i][decrease] /= 2;
votes[i][dont_change] /= 2;
else
if ratio < - ref_level then
decrease parameter;
// to avoid bouncing up and down (inertia)
votes[i][increase] /= 2;
votes[i][decrease] /= 2;
votes[i][dont_change] /= 2;
end if
end if
end for features
end for positions
you can stop and save and restore the state and resume the process.
A sparse feature do not award enough votes for an increment and can be dropped of the evaluation.