Official Release of Ethereal 11.00

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

AndrewGrant
Posts: 1750
Joined: Tue Apr 19, 2016 6:08 am
Location: U.S.A
Full name: Andrew Grant

Official Release of Ethereal 11.00

Post by AndrewGrant »

Hello all,

I'm here again to announce a new official release of Ethereal, this time version 11.00.

It has been almost two months, 45 patches, and 25 million test games without a release. Time to make things official once again :)

I am confident that this version will safely secure the position as second best open source engine.

Binaries for nopopcnt, popcnt, and pext for Windows, as well as armv7 and armv8 for Android are here

As always, Linux users should have no problem building. Simply use "make nopopcnt/popcnt/pext" and all will work.

I will post again in this thread with test data in the morning, once 11.00 tests are completed.

A quick summary of patches. Those with an * are by Laldon, and those with a + are inspired by others ...

Code: Select all

10.56   : Skip Null Move Pruning when the TT entry is poor
10.57 * : Remove OCB scaling for the Bishop + 2 Knights case
10.58   : Allocate even more time when the score drops between searches
10.59   : Tune BaseKingSafety, and KingSafetyNoQueen constants
10.60   : Reduce Counter Move Pruning for improving nodes
10.61   : Implement "Follow-up History" (SF calls it (ss-2)->contHist)
10.62   : Use Follow-up History for low depth pruning
10.63   : Cleanup Zobrist key implementations
10.64   : Slightly faster Zobrist for non-normal move functions
10.65   : Remove separate Zobrist table for Pawns & Kings
10.66   : Limit Futility Pruning based on History values
10.67   : Tune threat evaluations using NDEPTHS=5 tuning session
10.68   : No longer bound the return value from a TT cutoff
10.69   : Move time management code to a new file, time.c
10.70   : Do ProbCut when in check, use Depth 2 verification search
10.71   : Bonus for Minors in front of our own, or our enemies, pawns
10.72   : Move JChess, EtherealV1, EtherealV2, Ethereal0xFF, to legacy branch
10.73   : Tune PSQT and assorted bonuses for Knights and Bishops
10.74   : Evaluate distance between Passers and each King
10.75 + : Cleanup draw detect and fix a bug, suggested by Ratosh (Pirarucu)
10.76   : Bonus when a Passer has a safe promotion pathway
10.77   : One-time compute of King squares for evaluation function
10.78   : Do more Counter Move / Follow-up Move pruning for non improving nodes
10.79   : Extend ThreatByPawnPush to understand double moves and enpass threats
10.80   : Speedup the move picker, and use SEE(1) for qsearch moves
10.81   : Try at least one quiet move after hitting LMP or FMP conditions
10.82 * : Increase time usage for large score jumps (Hold onto possible wins)
10.83 * : Increase time usage for small score jumps (Hold onto possible wins)
10.84 + : Update chess programming wiki link by Captflint
10.85 + : Fix bug with history hashing for Null Moves, found by Aloril
10.86   : Begrudgingly implement pondering support for Ethereal
10.87   : Simplify time management updates when the PV changes
10.88   : Add NUMA support for Windows as per Stockfish as per Texel
10.89   : Cleanup the Tuner, simplify adding additional terms to the tuner
10.90 + : Tweak Magic Bitboard workings for better cache usage, inspired by Laser
10.91   : Evaluate Pawn storm threats to our King
10.92   : Remove OCB scaling for the Bishop + 2 Rooks case
10.93   : Tune the Pawn Storm threats array
10.94   : NUMA fix when running many copies of Ethereal at once, found by CoffeeOne
10.95   : Use a different SEE margin for quiet vs noisy moves in pruning
10.96   : Don't let a player try two Null moves in a row
10.97 * : Remove OCB scaling for the Bishop + <*> case
10.98 * : Tuning session using NDEPTHS=8
10.99   : Tuning session using NDEPTHS=10
11.00   : Small code cleanups in attacks.c and board.c
As always, a quick list of thank-yous to those who have contributed ideas and efforts to development ...

CoffeeOne has helped confirm correctness of the Numa support, as well as track down a bug or two. Additionally, who provided some computational power to the testing framework.

Aloril tracked down a position where Ethereal blundered wildly. After some work, I the issue issued was the result of Null Move Pruning.

CMCanavessi, for relentlessly bother me about Pondering support. CCCC was the tipping point, but he was the real inspiration. If he wants to play with Ethereal using ponder, then he shall :)

Ratosh pointed out some inefficiencies with the draw detection, which led me to realize I had turned off the fifty move rule quite recently.

Noobpwnftw, This literally could not have been done without him. Ethereal has reached the point where the vast majority of patches are too small to test using my own hardware. His computational resources have opened up an entire new set of possible patches for Ethereal, many of which have been explored.

Ipmanchess, for not letting me give up. Every time I start to think I'm nearly done with development, he shows up with the latest data vs Fire and tells me I am so close, and that just a few more patches and Ethereal will be there.

Graham + friends, CCRL has given all authors something to work towards. Watching Ethereal slowly climb up the rungs, patch by patch, is one of the most surreal things. 25 months ago I posted on CCRL forums asking what Ethereal would need in order to be placed. Now, ... so much has changed.

TCEC, for giving Ethereal a chance to show the world what I've been working on for the last four years. Seeing the steady climb from Div3->DivP this season has been incredible. The amount of interest around Ethereal is at an all time high.

Laldon has helped keep me sane. Hes long been my right hand man in Ethereal development. I've done what I can to let him know how much I appreciate his efforts in the past, but that's no longer enough. Laldon is now listed as an author on the UCI output for Ethereal.
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
User avatar
Graham Banks
Posts: 41415
Joined: Sun Feb 26, 2006 10:52 am
Location: Auckland, NZ

Re: Official Release of Ethereal 11.00

Post by Graham Banks »

Looking forward to testing 11.0. :)
gbanksnz at gmail.com
AndrewGrant
Posts: 1750
Joined: Tue Apr 19, 2016 6:08 am
Location: U.S.A
Full name: Andrew Grant

Re: Official Release of Ethereal 11.00

Post by AndrewGrant »

Am I too late to have you swap for 11.00 in Division 1?

No worries if so, results are already quite good :-)
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
User avatar
Graham Banks
Posts: 41415
Joined: Sun Feb 26, 2006 10:52 am
Location: Auckland, NZ

Re: Official Release of Ethereal 11.00

Post by Graham Banks »

AndrewGrant wrote: Fri Sep 07, 2018 2:34 am Am I too late to have you swap for 11.00 in Division 1?

No worries if so, results are already quite good :-)
Replacing it shortly. :)
gbanksnz at gmail.com
User avatar
CMCanavessi
Posts: 1142
Joined: Thu Dec 28, 2017 4:06 pm
Location: Argentina

Re: Official Release of Ethereal 11.00

Post by CMCanavessi »

AndrewGrant wrote: Fri Sep 07, 2018 2:34 am Am I too late to have you swap for 11.00 in Division 1?

No worries if so, results are already quite good :-)
Awesome job Andrew!! Your efforts are not in vain, I have high hopes for Ethereal in CCCC and TCEC. And next season in CCLS, it's a serious candidate for the superfinal!!
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: Official Release of Ethereal 11.00

Post by MikeB »

You are doing a great job Andrew - keep up the good work! 👊👍
Image
Gabor Szots
Posts: 1362
Joined: Sat Jul 21, 2018 7:43 am
Location: Szentendre, Hungary
Full name: Gabor Szots

Re: Official Release of Ethereal 11.00

Post by Gabor Szots »

Testing starts today. Won't make this weeks update, though.
Gabor Szots
CCRL testing group
AndrewGrant
Posts: 1750
Joined: Tue Apr 19, 2016 6:08 am
Location: U.S.A
Full name: Andrew Grant

Re: Official Release of Ethereal 11.00

Post by AndrewGrant »

Code: Select all

|================|================|========|=======|=======|=======|=======|=========|=======|
|      TEST      |      BASE      |  TIME  |   W   |   L   |   D   |   N   |   ELO   | ERROR |
|================|================|========|=======|=======|=======|=======|=========|=======|
| Ethereal 10.61 | Ethereal 10.55 | 60+.6s |  2698 |  1988 | 10314 | 15000 |   16.46 |  3.10 |
| Ethereal 10.68 | Ethereal 10.55 | 60+.6s |  2904 |  1828 | 10268 | 15000 |   24.97 |  3.11 |
| Ethereal 10.75 | Ethereal 10.55 | 60+.6s |  3278 |  1547 | 10175 | 15000 |   40.27 |  3.13 |
| Ethereal 10.78 | Ethereal 10.55 | 60+.6s |  3554 |  1597 |  9849 | 15000 |   45.59 |  3.23 |
| Ethereal 10.81 | Ethereal 10.55 | 60+.6s |  2581 |   968 |  6777 | 10326 |   54.72 |  3.88 |
| Ethereal 10.95 | Ethereal 10.55 | 60+.6s |  3278 |  1045 |  7406 | 11729 |   66.96 |  3.76 |
| Ethereal 11.00 | Ethereal 10.55 | 60+.6s |  1481 |   436 |  3083 |  5000 |   73.70 |  5.88 |
|================|================|========|=======|=======|=======|=======|=========|=======|
#WeAreAllDraude #JusticeForDraude #RememberDraude #LeptirBigUltra
"Those who can't do, clone instead" - Eduard ( A real life friend, not this forum's Eduard )
BrendanJNorman
Posts: 2526
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Official Release of Ethereal 11.00

Post by BrendanJNorman »

AndrewGrant wrote: Fri Sep 07, 2018 2:12 am Hello all,

I'm here again to announce a new official release of Ethereal, this time version 11.00.

It has been almost two months, 45 patches, and 25 million test games without a release. Time to make things official once again :)

I am confident that this version will safely secure the position as second best open source engine.

Binaries for nopopcnt, popcnt, and pext for Windows, as well as armv7 and armv8 for Android are here

As always, Linux users should have no problem building. Simply use "make nopopcnt/popcnt/pext" and all will work.

I will post again in this thread with test data in the morning, once 11.00 tests are completed.

A quick summary of patches. Those with an * are by Laldon, and those with a + are inspired by others ...

Code: Select all

10.56   : Skip Null Move Pruning when the TT entry is poor
10.57 * : Remove OCB scaling for the Bishop + 2 Knights case
10.58   : Allocate even more time when the score drops between searches
10.59   : Tune BaseKingSafety, and KingSafetyNoQueen constants
10.60   : Reduce Counter Move Pruning for improving nodes
10.61   : Implement "Follow-up History" (SF calls it (ss-2)->contHist)
10.62   : Use Follow-up History for low depth pruning
10.63   : Cleanup Zobrist key implementations
10.64   : Slightly faster Zobrist for non-normal move functions
10.65   : Remove separate Zobrist table for Pawns & Kings
10.66   : Limit Futility Pruning based on History values
10.67   : Tune threat evaluations using NDEPTHS=5 tuning session
10.68   : No longer bound the return value from a TT cutoff
10.69   : Move time management code to a new file, time.c
10.70   : Do ProbCut when in check, use Depth 2 verification search
10.71   : Bonus for Minors in front of our own, or our enemies, pawns
10.72   : Move JChess, EtherealV1, EtherealV2, Ethereal0xFF, to legacy branch
10.73   : Tune PSQT and assorted bonuses for Knights and Bishops
10.74   : Evaluate distance between Passers and each King
10.75 + : Cleanup draw detect and fix a bug, suggested by Ratosh (Pirarucu)
10.76   : Bonus when a Passer has a safe promotion pathway
10.77   : One-time compute of King squares for evaluation function
10.78   : Do more Counter Move / Follow-up Move pruning for non improving nodes
10.79   : Extend ThreatByPawnPush to understand double moves and enpass threats
10.80   : Speedup the move picker, and use SEE(1) for qsearch moves
10.81   : Try at least one quiet move after hitting LMP or FMP conditions
10.82 * : Increase time usage for large score jumps (Hold onto possible wins)
10.83 * : Increase time usage for small score jumps (Hold onto possible wins)
10.84 + : Update chess programming wiki link by Captflint
10.85 + : Fix bug with history hashing for Null Moves, found by Aloril
10.86   : Begrudgingly implement pondering support for Ethereal
10.87   : Simplify time management updates when the PV changes
10.88   : Add NUMA support for Windows as per Stockfish as per Texel
10.89   : Cleanup the Tuner, simplify adding additional terms to the tuner
10.90 + : Tweak Magic Bitboard workings for better cache usage, inspired by Laser
10.91   : Evaluate Pawn storm threats to our King
10.92   : Remove OCB scaling for the Bishop + 2 Rooks case
10.93   : Tune the Pawn Storm threats array
10.94   : NUMA fix when running many copies of Ethereal at once, found by CoffeeOne
10.95   : Use a different SEE margin for quiet vs noisy moves in pruning
10.96   : Don't let a player try two Null moves in a row
10.97 * : Remove OCB scaling for the Bishop + <*> case
10.98 * : Tuning session using NDEPTHS=8
10.99   : Tuning session using NDEPTHS=10
11.00   : Small code cleanups in attacks.c and board.c
As always, a quick list of thank-yous to those who have contributed ideas and efforts to development ...

CoffeeOne has helped confirm correctness of the Numa support, as well as track down a bug or two. Additionally, who provided some computational power to the testing framework.

Aloril tracked down a position where Ethereal blundered wildly. After some work, I the issue issued was the result of Null Move Pruning.

CMCanavessi, for relentlessly bother me about Pondering support. CCCC was the tipping point, but he was the real inspiration. If he wants to play with Ethereal using ponder, then he shall :)

Ratosh pointed out some inefficiencies with the draw detection, which led me to realize I had turned off the fifty move rule quite recently.

Noobpwnftw, This literally could not have been done without him. Ethereal has reached the point where the vast majority of patches are too small to test using my own hardware. His computational resources have opened up an entire new set of possible patches for Ethereal, many of which have been explored.

Ipmanchess, for not letting me give up. Every time I start to think I'm nearly done with development, he shows up with the latest data vs Fire and tells me I am so close, and that just a few more patches and Ethereal will be there.

Graham + friends, CCRL has given all authors something to work towards. Watching Ethereal slowly climb up the rungs, patch by patch, is one of the most surreal things. 25 months ago I posted on CCRL forums asking what Ethereal would need in order to be placed. Now, ... so much has changed.

TCEC, for giving Ethereal a chance to show the world what I've been working on for the last four years. Seeing the steady climb from Div3->DivP this season has been incredible. The amount of interest around Ethereal is at an all time high.

Laldon has helped keep me sane. Hes long been my right hand man in Ethereal development. I've done what I can to let him know how much I appreciate his efforts in the past, but that's no longer enough. Laldon is now listed as an author on the UCI output for Ethereal.
A fantastic engine Andrew.

Thank you.

Its nice to see such a strong engine retain an original playing style.

Ethereal is a nice analysis companion for my recently purchased Komodo 12.1.1.

Best of luck once you enter the top 5! :)
User avatar
Werner
Posts: 2871
Joined: Wed Mar 08, 2006 10:09 pm
Location: Germany
Full name: Werner Schüle

Re: Official Release of Ethereal 11.00

Post by Werner »

Thanks Andrew,
Good job!
Is this more or less the TCEC version? More results for 10.97 on Sunday.
Good luck
Werner