Rebel wrote: ↑Mon Feb 24, 2020 8:51 am
abulmo2 wrote: ↑Mon Feb 24, 2020 1:12 am
Rebel wrote: ↑Sun Feb 23, 2020 11:19 am
abulmo2 wrote: ↑Sat Feb 22, 2020 11:08 pm
Rebel wrote: ↑Sat Feb 22, 2020 9:56 pm
As lone individuals writing chess engines we don't have the resources to do expensive, time consuming test methodologies, beta test procedures before we release.
In writing Amoeba, as a lone individual, I spent 99.9% of my time testing and 0.1% writing code. I have limited financial resources, but I have an invaluable resource: time.
Can you elaborate what the testing is about?
I am mostly testing for enhancements with SPRT; using my own tournament manager.
My development version is also quite different to the release version, as it contains many testing code: perft, epd test, etc.
I also use a language that is probably better at reducing bugs (the D language), as it uses a garbage collector and help to write things in less line of codes.
Is my program bug free? Unfortunately not. Probably half of the Elo of my program came from bugs removal and I guess there are still many of them to suppress.
Year by year your engine will become more and more robust. Tools like MRi may speedup that process.
This is only true IFF (if and only if) you do not add any NEW features. But you are really thinking far too narrowly here. there are lots of possible types of bugs:
(1) syntax - easy to find since compiler points 'em out.
(2) simple things like array bounds violations. There are programs that will catch MOST (but not all) of these (pointers cause issues).
(3) pointer errors. Up to you for the most part. Pretty easy to do silly things like pass a null pointer to a procedure that expects a pointer to something useful. Hard to detect this. And to have a chance you have to compile the entire source in one chunk. Probably not on a piece of code I worked on for Los Alamos back in the 80's, code that was over 7 million lines long.
(4) logic errors. No software is ever going to catch that until you can provide perfect specifications in addition to perfect code. And now you have yet another thing to debug, the specifications. This is about 99% of my bugs. mal-formed test to recognize an open file, or a backward pawn, or whatever.
(5) parallel threading errors. No software will catch these, ever. Gross ones, maybe. Subtle race conditions? Never. Unless you demand an atomic lock for EVERY shared variable and you require one specific order for memory updates for every group of shared values that get updated in parallel. This leads to forgetting about parallel search since the restrictions would make it useless.
(4) and (5) are the ones that give me concern. Programs that cause a crash are usually straightforward to find and fix, excepting parallel race conditions which can be one's worst nightmare to try to find. Programs that just don't produce an optimal answer are much harder to debug since there is no obvious bug showing up in output.
I can only speak for myself, but when I add code, I usually add errors that I have to find and fix. If not, why did I have errors in the original code. I'm still the same person with the same skills.