PARAM-009 · Constraint Solving & Editability · difficulty 4/5

Configurable bottle (height + cap params)

sha256:31cba0fde20cb011…

§1Prompt verbatim

Cylindrical bottle Ø 70 mm × H mm tall, ending in a M40 × 1.0 threaded neck of length C mm. Build it once at H = 180, C = 22, then expose H and C as parameters. We will sweep H ∈ [120, 220] and C ∈ [16, 30] and re-evaluate.

§2Ground-truth spec

shells1

watertighttrue

manifoldtrue

acceptance ε±0.1 mm

parametric edits

H : 180 → 200 (ΔV expected 76969 mm³)

C : 22 → 28 (ΔV expected 7540 mm³)

§3Reference render

canonical reference · drag to orbit, scroll to zoom

Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).

§4Per-agent renders

reference + 10 agent outputs · scored against the held-out STEP

vol IoU · BREP · manifold

canonical reference

REFERENCE

canonical · ground truth

1.000100✓

OpenAI o4 (reasoning) → CadQuery

OpenAI + CadQuery 2.4

Human Baseline (Mech-E)

n=4 senior engineers

0.7409✓

Claude Opus 4.7 → CadQuery

Anthropic + CadQuery 2.4

0.6888✓

Claude Sonnet 4.6 → CadQuery

Anthropic + CadQuery 2.4

0.6339✗

GPT-5 → CadQuery

OpenAI + CadQuery 2.4

0.6198✗

Gemini 2.5 Flash → CadQuery

Google + CadQuery 2.4

0.59110✗

DeepSeek R1 (reasoning) → CadQuery

DeepSeek + CadQuery 2.4

CAD-Coder Labs (research)

0.51511✗

Gemini 2.5 Pro → OpenSCAD

Google + OpenSCAD 2024.06

0.4580✗

Qwen3 Coder → CadQuery

Alibaba + CadQuery 2.4

0.44910✗

Claude Haiku 4.5 → CadQuery

Anthropic + CadQuery 2.4

0.40214✗

Claude Opus 4.7 → OpenSCAD

Anthropic + OpenSCAD 2024.06

0.3750✗

GPT-5 Mini → OpenSCAD

OpenAI + OpenSCAD 2024.06

0.33315✗

Llama 3.3 70B → OpenSCAD

Meta + OpenSCAD 2024.06

0.18332✗

DeepCAD

Wu et al. 2021 (research)

Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.

§5Per-agent metrics

ranked by Vol IoU · same data as the leaderboard, restricted to this task

Agent	Watert.	Manif.	ParamEdit	ConSolve	Parametric Range Integrity	P@1	p50	latency	cost
OpenAI o4 (reasoning) → CadQuery	✓	0.973	0.791	0.827	0.802	1.000	—	89.4s	$1.195
Adam (CADcrush)	✓	0.956	0.809	0.789	0.668	1.000	—	9.6s	$0.306
Human Baseline (Mech-E)	✓	0.963	0.869	0.925	0.876	0.000	—	742.3s	$6.104
Claude Opus 4.7 → CadQuery	✓	0.953	0.828	0.830	0.767	0.000	—	40.9s	$0.338
Claude Sonnet 4.6 → CadQuery	✓	0.939	0.792	0.789	0.634	0.000	—	14.9s	$0.061
GPT-5 → CadQuery	✓	0.944	0.732	0.697	0.740	0.000	—	37.7s	$0.175
Gemini 2.5 Flash → CadQuery	✓	0.943	0.628	0.608	0.584	0.000	—	15.3s	$0.024
DeepSeek R1 (reasoning) → CadQuery	✓	0.943	0.761	0.784	0.635	0.000	—	83.2s	$0.038
Zoo Text-to-CAD	✓	0.934	0.650	0.738	0.591	0.000	—	5.1s	$0.148
CAD-Coder R1	✓	0.928	0.621	0.573	0.530	0.000	—	6.3s	$0.004
Gemini 2.5 Pro → OpenSCAD	✓	0.922	0.551	0.613	0.542	0.000	—	28.7s	$0.084
Qwen3 Coder → CadQuery	✓	0.924	0.669	0.642	0.561	0.000	—	14.5s	$0.028
Claude Haiku 4.5 → CadQuery	×	0.908	0.536	0.542	0.422	0.000	—	6.7s	$0.020
Claude Opus 4.7 → OpenSCAD	×	0.910	0.583	0.595	0.536	0.000	—	23.9s	$0.270
GPT-5 Mini → OpenSCAD	×	0.898	0.476	0.465	0.395	0.000	—	11.8s	$0.009
Llama 3.3 70B → OpenSCAD	×	0.876	0.436	0.441	0.380	0.000	—	26.5s	$0.017
DeepCAD	×	0.873	0.286	0.289	0.217	0.000	—	5.8s	$0.017
Trellis 3D	×	0.850	0.058	0.055	0.036	0.000	—	14.1s	$0.052
Spline AI	×	0.850	0.046	0.046	0.019	0.000	—	6.5s	$0.041
Hunyuan3D-2	×	0.850	0.046	0.045	0.036	0.000	—	28.8s	$0.057