What do you focus on to make your engine better?

[zd3nik]

I tend to repetitively fall into the abyss that is LMR. Probably because my engines don't achieve the kind of search depths I would like and LMR (and variants) seem to me to have the most potential for the greatest gains in search depth. But I certainly haven't had any luck figuring out how to unlock those gains without introducing some serious weaknesses.

Today I tried a quick experiment: execute a null move followed up with a quiescence search after every LMR where the remaining depth is > 1. If the qsearch shows a significant gain, reduce or eliminate the late move reduction (e.g. increase the depth back to, or closer to, what it would have been without LMR). This could certainly be optimized, but for a quick test I just did this the simplest way. And it worked great in some test positions. But the net effect was definitely an ELO loss in the quick tournaments I ran. I may try again using more aggressive initial reduction depths, relying on the null-move-qsearch threat detection to back off the reductions where appropriate. But if it's like any other idea I've ever tried with LMR it's probably just a colossal waste of time.

So I want to try and divert my attention to some other aspect of the engine to work on. And I'm wondering what part of the engine would provide the best R.O.I.

Is there any aspect of your chess engine that you spend more time on than others when you're trying to make improvements? Has your work on this aspect of your engine been productive? Any general tips for engine improvement you care to share?

Thanks,
STC

[stegemma]

I think that first of all you should define your goals. What do you want to reach? The strongest program in the world? For me, I'm focusing on speed of the move generator and this let my software to be the weakest in the world . Despite of that, now I'm focused on make my Satana engine stronger than... the previous release of Satana! Another goal is to make it play better than its historical opponent: Neurone, another engine not very strong but with very good heuristics.

After have defined your goal (that should be not so far from the actual ELO) maybe you can even think about the context. It can happens that your engine is stronger at certain time level of play and weaker at lower one; for this you should test at different time controls... maybe your software is not so good at 1 minute per game.

[Daniel Anulliero]

I tend to repetitively fall into the abyss that is LMR. Probably because my engines don't achieve the kind of search depths I would like and LMR (and variants) seem to me to have the most potential for the greatest gains in search depth. But I certainly haven't had any luck figuring out how to unlock those gains without introducing some serious weaknesses.

Today I tried a quick experiment: execute a null move followed up with a quiescence search after every LMR where the remaining depth is > 1. If the qsearch shows a significant gain, reduce or eliminate the late move reduction (e.g. increase the depth back to, or closer to, what it would have been without LMR). This could certainly be optimized, but for a quick test I just did this the simplest way. And it worked great in some test positions. But the net effect was definitely an ELO loss in the quick tournaments I ran. I may try again using more aggressive initial reduction depths, relying on the null-move-qsearch threat detection to back off the reductions where appropriate. But if it's like any other idea I've ever tried with LMR it's probably just a colossal waste of time.

So I want to try and divert my attention to some other aspect of the engine to work on. And I'm wondering what part of the engine would provide the best R.O.I.

Is there any aspect of your chess engine that you spend more time on than others when you're trying to make improvements? Has your work on this aspect of your engine been productive? Any general tips for engine improvement you care to share?

Thanks,
STC

Well , lmr for me is just like for you ( Isa my chess program drew your Clubfoot in hgm monthly tourney ) , a "brain crack" ! I tried tons of diférents réductions system ( by history or not , by moves number, by depth etc...) I'm far from an expert (I programming Chess just for fun) but I understood lmr is dificult to improve without:
-a very good move ordering
-a very good eval very well tuned
-a very good bug free search
Lol not much help for you !
Bests
Dany

[Ferdy]

Focus on finding bugs is one. One symptom that you have a bug is that your engine is very difficult to improve.

[cdani]

Focus on finding bugs is one. One symptom that you have a bug is that your engine is very difficult to improve.

Absolutely critical.

I'm far from an expert (I programming Chess just for fun) but I understood lmr is dificult to improve without:
-a very good move ordering
-a very good eval very well tuned
-a very good bug free search

Sure LMR improves your engine easier if it already has a good base.

Anyway, some time ago I have done Andscacs-Sungorus, an experiment joining a very simple eval with the advanced search capabilities of Andscacs, and it played at maybe 2700, profiting a lot from all the search stuff. Of course it played badly a lot of positions, but keep winning games thanks to the high depths reached. Can something like this be tuned from the start, so keeping simple the eval? Difficult to say, but I suppose it will be more difficult than with a more equilibrated engine that will output more consistent results.

I will say than LMR has some critical requirements. In the list of Daniel I will put first move ordering. Having a decent and fast quiescence function to discard all the bad positions generated by the now reduced search is also in the critical list.

[cdani]

I have done some basic tests to try to verify the relation between LMR and move ordering. I used different versions of Andscacs:

* andscacs081019.exe, is like 10 elo stronger than the last published version
* andscacs081019nolmr.exe, the same without lmr.
* andscacs081019noqmo.exe, the same without quiet move ordering.
* andscacs081019noqmo_nolmr.exe, the same without quiet move ordering and also without lmr.

Results of three tests (4 seconds + 0.02):

Code: Select all

randand81019_81019noqmo_4_02b.pgn
 1 Andscacs 0.81019        147   10   10  1566   86%  -147   18%
 2 Andscacs 0.81019noqmo  -147   10   10  1566   14%   147   18%

randand81019_81019nolmr_4_02b.pgn
 1 Andscacs 0.81019         53    8    8  1542   66%   -53   34%
 2 Andscacs 0.81019nolmr   -53    8    8  1542   34%    53   34%

randand81019noqmo_81019moqmo_nolmr_4_02b.pgn
 1 Andscacs 0.81019noqmo         120    9    9  1566   80%  -120   17%
 2 Andscacs 0.81019noqmo_nolmr  -120    9    9  1566   20%   120   17%

In this file there are the different executables and the resulting pgns:
http://www.andscacs.com/andscacs81019_test.zip

[zd3nik]

I think that first of all you should define your goals. What do you want to reach? The strongest program in the world? For me, I'm focusing on speed of the move generator and this let my software to be the weakest in the world . Despite of that, now I'm focused on make my Satana engine stronger than... the previous release of Satana! Another goal is to make it play better than its historical opponent: Neurone, another engine not very strong but with very good heuristics.

After have defined your goal (that should be not so far from the actual ELO) maybe you can even think about the context. It can happens that your engine is stronger at certain time level of play and weaker at lower one; for this you should test at different time controls... maybe your software is not so good at 1 minute per game.

Excellent advice.

My goal is just to make my engines stronger. And I've been targeting higher search depths to achieve that goal. And to achieve higher search depths I've been targeting LMR.

Given my lack of progress with LMR tweaks, I'm trying to decide where to focus my efforts. So you could say I'm trying deciding what my new goal should be.

[zd3nik]

Focus on finding bugs is one. One symptom that you have a bug is that your engine is very difficult to improve.

Agreed. I discovered this first hand in many of my previous engines. I feel pretty confident that my current engines are pretty bug free because I've adopted techniques that tend to help find and prevent bugs. Mostly with a lot of asserts in key areas of the engine and frequently running debug builds. But also simpler code, even at the expense of speed in many cases.

In any case, I will keep this advice in mind. Bugs do indeed hinder engine strength, so no amount of search optimization now superior technique will show improvement.

[zd3nik]

I have done some basic tests to try to verify the relation between LMR and move ordering. I used different versions of Andscacs:

* andscacs081019.exe, is like 10 elo stronger than the last published version
* andscacs081019nolmr.exe, the same without lmr.
* andscacs081019noqmo.exe, the same without quiet move ordering.
* andscacs081019noqmo_nolmr.exe, the same without quiet move ordering and also without lmr.

Results of three tests (4 seconds + 0.02):

Code: Select all

randand81019_81019noqmo_4_02b.pgn
 1 Andscacs 0.81019        147   10   10  1566   86%  -147   18%
 2 Andscacs 0.81019noqmo  -147   10   10  1566   14%   147   18%

randand81019_81019nolmr_4_02b.pgn
 1 Andscacs 0.81019         53    8    8  1542   66%   -53   34%
 2 Andscacs 0.81019nolmr   -53    8    8  1542   34%    53   34%

randand81019noqmo_81019moqmo_nolmr_4_02b.pgn
 1 Andscacs 0.81019noqmo         120    9    9  1566   80%  -120   17%
 2 Andscacs 0.81019noqmo_nolmr  -120    9    9  1566   20%   120   17%

In this file there are the different executables and the resulting pgns:
http://www.andscacs.com/andscacs81019_test.zip

I must admit I don't know how to interpret these results. I haven't looked at the zip file yet, I'm just referring to what you've posted here. Are the percentages the win/loss ratio? Why are all the other numbers the same (with some just having flipped sign)?

[cdani]

Results of three tests (4 seconds + 0.02):

Code: Select all

randand81019_81019noqmo_4_02b.pgn
 1 Andscacs 0.81019        147   10   10  1566   86%  -147   18%
 2 Andscacs 0.81019noqmo  -147   10   10  1566   14%   147   18%

randand81019_81019nolmr_4_02b.pgn
 1 Andscacs 0.81019         53    8    8  1542   66%   -53   34%
 2 Andscacs 0.81019nolmr   -53    8    8  1542   34%    53   34%

randand81019noqmo_81019moqmo_nolmr_4_02b.pgn
 1 Andscacs 0.81019noqmo         120    9    9  1566   80%  -120   17%
 2 Andscacs 0.81019noqmo_nolmr  -120    9    9  1566   20%   120   17%

I must admit I don't know how to interpret these results. I haven't looked at the zip file yet, I'm just referring to what you've posted here. Are the percentages the win/loss ratio? Why are all the other numbers the same (with some just having flipped sign)?

The results are from Bayeselo. I add here the headers:

Code: Select all

Rank Name                    Elo    +    - games score oppo. draws
   1 Andscacs 0.81019        147   10   10  1566   86%  -147   18%
   2 Andscacs 0.81019noqmo  -147   10   10  1566   14%   147   18%

The most important is 147/-147, that means a difference of 294 elo. +10 -10 is the confidence of those 294, so is from +304 to +284.

My first interpretation is that LMR has a lot more to win when there is no good quiet move ordering.

Another one is that quiet move ordering is more important than LMR.

I added the file just for if someone want to do their own tests.