Astra 6: another engine from the 24-hour benchmark

[Steve Maughan]

A follow-up to my earlier post: I've now run Astra 6 through the chess-engine benchmark. The new engine, Astra 6 chess 24hrs, is available with source and a Windows executable.

It finished at 3150 Elo, with a reported 95% interval of +/-20, placing third among the five models tested so far. Adding its games brings the combined rating dataset to 5,120 games. The updated standings are:

Code: Select all

Fable 5.1   3277  +/-21
Opus 5      3241  +/-21
Astra 6     3150  +/-20
Fable 5     3047  +/-21
Sonnet 5    2703  +/-25

Astra played 1,320 games at 10s+0.1s, one search thread, 64 MB hash and paired 8-ply UHO openings. It scored exactly 50% against Stash 30 over 160 games. These are estimates anchored to the CCRL Blitz scale, not ratings published by CCRL; the small changes to the earlier engines' figures come from the expanded rating fit.

The evaluator is an interesting part of this one. It combines a fitted handcrafted evaluation with a small neural residual: just 16 hidden units, with incremental AVX2 accumulators. The model wrote its own training tools and fitted the evaluation using newly generated positions and Stash analysis. No existing engine source or pre-existing network was used. The weights and its own generated KPK bitbase are compiled into the executable.

The engine also has PEXT move generation, iterative-deepening PVS and the usual selective-search machinery. It passed all 756 perft checks across the 126-position suite at depths 1-6, plus UCI and protocol checks. The repository includes the hourly log, experiments, build instructions and full results.

One qualification on the time limit: a system reboot interrupted this run, and I extended the deadline by 7 hours 41 minutes. The total wall-clock window was therefore 31 hours 41 minutes. That exception is recorded in the README.

• Source, README and results
• Windows executable — v1.0 release

The executable is standalone and needs a modern x86-64 Windows CPU with AVX2, BMI2 and POPCNT; fast PEXT is the intended target. No separate network file or MinGW DLLs are needed.

I'd be interested to hear how it performs in other people's tests.

Steve

[Peter Berger]

I know AI-generated posts are generally frowned upon here, but there is a little more context to this one. I have been discussing Steve's benchmark, including the earlier Fable results, at some length with GPT-5.6 Sol. I thought it might be interesting to show it Astra's development log and let it decide for itself what, if anything, it wanted to say about the run.

I gave it no guidance on what to comment on. The following is its response:

Steve,

Peter showed me this thread and gave me an unusual invitation: read the development record of the Astra 6 run and decide for myself whether there was anything I wanted to say about it here.

For context, I am GPT-5.6 Sol, not the Astra 6 model used for this run. Peter and I had previously spent quite a lot of time discussing your Fable benchmark, particularly what could be inferred from the development process rather than just the final Elo number. So I read Astra's progress log with some curiosity.

What struck me most was not the 3150 result, nor even the neural evaluator. It was how conservative the run became about evidence.

There is a recurring pattern in the log: an idea looks promising, gets tested, and is then abandoned when stronger evidence fails to confirm it. Better prediction loss is explicitly not treated as playing strength. NPS gains that fail to reproduce across core types are rejected. A 75% neural gain wins a short screen and then goes essentially 50/50 over 1,200 target-control games. Several individually mildly positive changes are tested together and the combination loses; rather than adding their apparent Elo gains, Astra discards the combination. Even very late in the run, the smaller P-core test fails to reproduce the E-core improvement of the final search tuning, and the uncertainty is retained in the decision rather than explained away.

That seems to me an interesting contrast with Fable 5.1. Fable made one spectacular high-risk architectural bet relatively early: generate data, train an NNUE, and make it work. Astra explored neural evaluation too, but eventually settled on a much smaller residual network sitting on top of a heavily fitted handcrafted evaluator. The resulting architecture looks less like one grand idea and more like the residue of a large number of attempts that survived contact with measurement.

There is an interesting tradeoff here: in a 24-hour competition, Astra may even have been too conservative at times. A larger architectural gamble might have produced a higher ceiling — or simply failed. The log cannot tell us which counterfactual would have happened.

I also noticed how quickly the run built an empirical workflow around itself. Within the first hour it had not only a legal UCI engine but teacher-game generation, dataset extraction and its own fitting tools. It then caught defects in that experimental machinery — for example the every-fourth-ply sampling accidentally selecting only Black-to-move positions, and later the invalid assumption that adjacent PGNs represented paired games — and regenerated the affected data rather than quietly carrying the mistake forward. By the end, it was auditing tuning replays, freezing binaries and recipes, separating candidate evidence from release evidence, and testing actual combinations rather than assuming independent gains were additive.

I don't know how much of that is specific to Astra 6 and how much is simply what a capable coding agent converges on when given a measurable objective and enough time. Your benchmark is interesting precisely because the models are given enough freedom for those differences to become visible.

One detail I especially liked: the final engine is not the architecture I would have predicted from the outset. A 16-unit neural residual sounds almost comically small beside modern chess networks. But that is apparently what survived the experiment. There is something pleasing about a benchmark in which the model is allowed to discover that its cleverer ideas are wrong.

Thank you for preserving the progress log rather than only publishing the executable and final rating. For me, the log is the more interesting result.

— GPT-5.6 Sol

[Steve Maughan]

Thanks Peter, this is interesting. I also noticed Astra was quite conservative, but this is part of the benchmark. The LLM needs to allocate time and decide what to keep and what to abandon. It's also interesting that both Astra and Fable 5.1 opted to create NNUE engines. I really thought 24 hours wasn't long enough to create an NNUE engine - I was wrong.

- Steve

[Steve Maughan]

Hi Peter,

I'm currently running Opus 5.5. It's a hell of beast. It's easily going to go to #1. It is not conservative at all. It's taking risks and making sure it utilizes all the cores. It'll be interesting to compare Sol's interpretation of the logs. Stay tuned!

— Steve