SHEET-001 · Sheet-Metal Bodies · difficulty 4/5

Box-pan with corner relief cuts

sha256:08fa1cb12c9e0011…

§1Prompt verbatim

Box-pan from 1.0 mm Al-5052: outside 100 × 80 × 30 mm tall, four bend radii 1.5 mm inside, k-factor 0.40. Internal corners must have ⌽ 1.5 mm relief drills offset 1 mm into the flange. Output folded body and unfolded flat pattern.

§2Ground-truth spec

shells1

watertighttrue

manifoldtrue

acceptance ε±0.1 mm

featuresbend_x4, relief_corner_x4

§3Reference render

canonical reference · drag to orbit, scroll to zoom

Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).

§4Per-agent renders

reference + 10 agent outputs · scored against the held-out STEP

vol IoU · BREP · manifold

canonical reference

REFERENCE

canonical · ground truth

1.000100✓

Human Baseline (Mech-E)

n=4 senior engineers

0.7837✓

Claude Opus 4.7 → CadQuery

Anthropic + CadQuery 2.4

CAD-Coder Labs (research)

0.63811✗

OpenAI o4 (reasoning) → CadQuery

OpenAI + CadQuery 2.4

0.6288✗

GPT-5 → CadQuery

OpenAI + CadQuery 2.4

0.57311✗

DeepSeek R1 (reasoning) → CadQuery

DeepSeek + CadQuery 2.4

0.55214✗

Claude Opus 4.7 → OpenSCAD

Anthropic + OpenSCAD 2024.06

0.5480✗

Qwen3 Coder → CadQuery

Alibaba + CadQuery 2.4

0.52814✗

Claude Sonnet 4.6 → CadQuery

Anthropic + CadQuery 2.4

Gemini 2.5 Flash → CadQuery

Google + CadQuery 2.4

0.42715✗

Gemini 2.5 Pro → OpenSCAD

Google + OpenSCAD 2024.06

0.3790✗

Claude Haiku 4.5 → CadQuery

Anthropic + CadQuery 2.4

0.36516✗

DeepCAD

Wu et al. 2021 (research)

0.31114✗

GPT-5 Mini → OpenSCAD

OpenAI + OpenSCAD 2024.06

0.26120✗

Llama 3.3 70B → OpenSCAD

Meta + OpenSCAD 2024.06

no manifold solid produced

Trellis 3D

Microsoft Research

—4✗

Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.

§5Per-agent metrics

ranked by Vol IoU · same data as the leaderboard, restricted to this task

Agent	Watert.	Manif.	Named-Dimension RMSE	FeatRec	Wall-Thickness Uniformity	P@1	p50	latency	cost
Human Baseline (Mech-E)	✓	0.968	0.130	0.835	0.774	1.000	—	911.1s	$6.883
Claude Opus 4.7 → CadQuery	✓	0.954	0.199	0.691	0.699	0.000	—	38.5s	$0.310
Zoo Text-to-CAD	✓	0.946	0.216	0.771	0.721	0.000	—	4.6s	$0.164
CAD-Coder R1	✓	0.941	0.284	0.614	0.555	0.000	—	6.1s	$0.005
OpenAI o4 (reasoning) → CadQuery	✓	0.942	0.254	0.754	0.712	0.000	—	115.2s	$0.959
GPT-5 → CadQuery	✓	0.939	0.239	0.681	0.576	0.000	—	40.1s	$0.208
DeepSeek R1 (reasoning) → CadQuery	✓	0.928	0.250	0.689	0.507	0.000	—	70.0s	$0.045
Claude Opus 4.7 → OpenSCAD	✓	0.932	0.260	0.559	0.546	0.000	—	41.4s	$0.278
Qwen3 Coder → CadQuery	✓	0.924	0.249	0.653	0.518	0.000	—	16.6s	$0.027
Claude Sonnet 4.6 → CadQuery	✓	0.932	0.190	0.684	0.615	0.000	—	17.3s	$0.060
Adam (CADcrush)	✓	0.916	0.166	0.722	0.633	0.000	—	9.7s	$0.321
Gemini 2.5 Flash → CadQuery	✓	0.914	0.233	0.569	0.546	0.000	—	11.0s	$0.021
Gemini 2.5 Pro → OpenSCAD	×	0.908	0.220	0.517	0.485	0.000	—	30.1s	$0.102
Claude Haiku 4.5 → CadQuery	×	0.910	0.315	0.550	0.438	0.000	—	6.6s	$0.021
DeepCAD	×	0.899	0.312	0.492	0.391	0.000	—	4.9s	$0.024
GPT-5 Mini → OpenSCAD	×	0.893	0.368	0.437	0.395	0.000	—	13.1s	$0.008
Llama 3.3 70B → OpenSCAD	×	0.890	0.393	0.444	0.401	0.000	—	28.3s	$0.017
Hunyuan3D-2	×	0.867	0.511	0.220	0.123	0.000	—	28.3s	$0.083
Spline AI	×	0.853	0.567	0.094	0.066	0.000	—	9.4s	$0.043
Trellis 3D kernel error: BRepCheck_NotClosed	×	0.000	—	—	—	0.000	—	10.3s	$0.056