Are there engines that are not optimized only for bullet?

[Uri Blass]

Based on watching the tests in
http://tests.stockfishchess.org/tests

it seems that Stockfish use more than 99% of their testing time at 60+0.6 or faster.

Stockfish recently did some 240+2.4 time control test and people complained about wasting resources and they simply stopped the test.

see
http://tests.stockfishchess.org/tests/v ... 02b9ffc4f1

I wonder if there is another team that optimize their program for longer time control than bullet.
Note that I understand that you need to use significant part of the testing time for bullet but I think that at least 30% of the testing time should be used for significantly slower time control(let say 240+2.4 time control).

Maybe you can find that some changes that work at 60+0.6 do not work at longer time control or you may find some change that does not change much at 60+0.6 but is productive at 240+2.4 time control.

Stockfish has no chance to find changes that are good at 240+2.4 but are not good at 60+0.6 because they never try changes that do not pass 60+0.6 and even the test that they stopped at 240+0.4 was for something that passed 60+0.6 earlier.

[AndrewGrant]

Worth noting for any readers who will not read through the entire thread you have linked,
The test was stopped because the test went against fishtest rules, NOT because there is some issue with 240+2.4s testing.

[cdani]

Sometimes I test Andscacs at even longer time controls than 240 s, but the result is not reliable, of course. Is more to have a rough idea between clearly different versions. No enough resources to obtain anything meaningful.

[yanquis1972]

Based on watching the tests in
http://tests.stockfishchess.org/tests

it seems that Stockfish use more than 99% of their testing time at 60+0.6 or faster.

Stockfish recently did some 240+2.4 time control test and people complained about wasting resources and they simply stopped the test.

see
http://tests.stockfishchess.org/tests/v ... 02b9ffc4f1

I wonder if there is another team that optimize their program for longer time control than bullet.
Note that I understand that you need to use significant part of the testing time for bullet but I think that at least 30% of the testing time should be used for significantly slower time control(let say 240+2.4 time control).

Maybe you can find that some changes that work at 60+0.6 do not work at longer time control or you may find some change that does not change much at 60+0.6 but is productive at 240+2.4 time control.

Stockfish has no chance to find changes that are good at 240+2.4 but are not good at 60+0.6 because they never try changes that do not pass 60+0.6 and even the test that they stopped at 240+0.4 was for something that passed 60+0.6 earlier.

isn't it the strongest (elo) engine in TCEC? maybe by a good margin? or is komodo close? those are the only two top-top tier engines that are continuously developed. my point being, maybe testing at bullet isn't the same as optimizing for bullet. vas relied on hyperbullet for test, afaik, & i don't believe it was generally thought it suffered at LTC.

[Uri Blass]

Based on watching the tests in
http://tests.stockfishchess.org/tests

it seems that Stockfish use more than 99% of their testing time at 60+0.6 or faster.

Stockfish recently did some 240+2.4 time control test and people complained about wasting resources and they simply stopped the test.

see
http://tests.stockfishchess.org/tests/v ... 02b9ffc4f1

I wonder if there is another team that optimize their program for longer time control than bullet.
Note that I understand that you need to use significant part of the testing time for bullet but I think that at least 30% of the testing time should be used for significantly slower time control(let say 240+2.4 time control).

Maybe you can find that some changes that work at 60+0.6 do not work at longer time control or you may find some change that does not change much at 60+0.6 but is productive at 240+2.4 time control.

Stockfish has no chance to find changes that are good at 240+2.4 but are not good at 60+0.6 because they never try changes that do not pass 60+0.6 and even the test that they stopped at 240+0.4 was for something that passed 60+0.6 earlier.

isn't it the strongest (elo) engine in TCEC? maybe by a good margin? or is komodo close? those are the only two top-top tier engines that are continuously developed. my point being, maybe testing at bullet isn't the same as optimizing for bullet. vas relied on hyperbullet for test, afaik, & i don't believe it was generally thought it suffered at LTC.

The fact that something is the strongest does not mean that the way to test it is correct.
It is possible that stockfish could be stronger at longer time control in case of using part of the time for longer time control games.

Note that I am not against testing at bullet and the problem is that almost all of the testing time is in bullet.
testing in bullet to get an idea if some idea is good or bad is not something bad but I believe that later you should verify the result at longer time control let say 240+2.4(and maybe even test at 240+2.4 some other changes that lead to something near 0 elo change in bullet that today stockfish even does not test).

[AndrewGrant]

You will be very hard pressed to find ideas which fail STC/LTC as per Fishtest, but magically gain elo after 240+2.4s time controls.

I can think of one idea in the last two years which was shown to work at longer time controls, which was removing / adjusting LazyEval.

The only class of ideas under this category are speedups that throw away precision. But those are few and far between.

[syzygy]

I wonder if there is another team that optimize their program for longer time control than bullet.

If you mean if there are teams about as active as the SF team running their tests at much longer time controls than what fishtest uses, then the answer is most likely "no".

Resources are finite, so you have to use them wisely if you want to improve the engine the most in the least time. You can run many more tests at faster time controls than at slower time controls. And there are plenty of indications that testing at the faster time controls suffices for the vast majority of changes.

[Uri Blass]

I wonder if there is another team that optimize their program for longer time control than bullet.

If you mean if there are teams about as active as the SF team running their tests at much longer time controls than what fishtest uses, then the answer is most likely "no".

Resources are finite, so you have to use them wisely if you want to improve the engine the most in the least time. You can run many more tests at faster time controls than at slower time controls. And there are plenty of indications that testing at the faster time controls suffices for the vast majority of changes.

I understand that resources are finite so I do not say not to test at bullet time control but I still think that stockfish can use something like 3% of the games that may take 30% of the time for 240+2.4 time control.

[lkaufman]

Komodo tests at longer time controls (such as 300" + 3") whenever we think a certain idea needs a lot of time, and also sometimes before a release.

[syzygy]

I understand that resources are finite so I do not say not to test at bullet time control but I still think that stockfish can use something like 3% of the games that may take 30% of the time for 240+2.4 time control.

If I understand you right, you propose selectively running some of the tests at 240+2.4. But how are you going to decide for which changes it is worth it to test at longer time controls?

(Of course 240+2.4 is still very short.)