How well can an LLM do hardware engineering?

Leaderboard

Bars show the overall score. Black whiskers show the 95% interval. 77 tasks across combustion, battery, cfd and packaging; one trial per task.

Run it, or talk to us.

The public task set ships as a Harbor dataset — each task is a container, a prompt, a reference solution and a verifier, runnable against any model with a completions endpoint.

Behind the public set is a much larger private task inventory built for post-training: engineering-simulation environments with a programmatic, non-hackable reward, in families parameterised into as many operating points as an RL run needs. If you are building post-training environments, get in touch.

Where the score comes from.

The headline is the equal-weighted mean of these four numbers, so a domain where the ordering differs from the headline is worth seeing directly.

How a task is built and scored.

• The problem A task gives the physics, the operating point, the quantity to report and the tolerance it must land in. Tasks are derived from traces of an expert working the problem with an agent, rather than written to fill a grid.
• Where it comes from The author's own engineering work, and questions users ask our production agent. Operating points are moved away from the values in published tables, so the answer is unlikely to be recalled — it has to be computed.
• The grade A deterministic verifier runs in the container. It deletes the submission's numeric artifacts and re-runs the submission's entry point, so the graded number comes from code that ran during grading.
• The band A band's centre is a reference solution reproduced at the same resolution. Its width is set by how much competent practitioners disagree on the same problem; tighter bands would mostly measure noise.
• The budget Each task carries a budget. A run that exceeds it is graded on what it produced up to that point, and the number of such runs is published beside the mean.
• The board One score per model, 0 to 1, over 77 tasks in 4 engineering domains (Combustion 21, Battery 34, CFD 13, Packaging 9), one representative per physics family, one trial each. The score is the equal-weighted mean of the per-domain means — weighting by task count would let whichever physics has the most variants decide the headline. The mark is the 95% interval; the dot is the estimate. 34 trial(s) ran out of budget; they are counted, not dropped.
• The score Per task, the reported value is inside its band or it is not. Per model, the headline is the equal-weighted mean of the per-domain means, over one task per physics family.

Two reference points bracket each run: a reference solution scores 1.0 by construction, and an empty submission has to score below 0.5.