DFMCNC-005 · DFM · 3-Axis CNC · difficulty 3/5

4-pocket plate, R 0.5 internal corners

sha256:7d01ef02a3cb0011…

§1Prompt verbatim

Aluminium plate 100 × 100 × 10 mm with four rectangular pockets (35 × 35 × 5 mm deep) on a 2×2 grid, internal corner radius R 0.5 mm. Must be machinable on a 3-axis VMC with a Ø 1 mm endmill at the corners and Ø 6 mm for bulk removal.

§2Ground-truth spec

shells1

watertighttrue

manifoldtrue

acceptance ε±0.05 mm

featurespocket_x4, internal_R0.5

§3Reference render

canonical reference · drag to orbit, scroll to zoom

Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).

§4Per-agent renders

reference + 10 agent outputs · scored against the held-out STEP

vol IoU · BREP · manifold

canonical reference

REFERENCE

canonical · ground truth

1.000100✓

Human Baseline (Mech-E)

n=4 senior engineers

0.7349✓

GPT-5 → CadQuery

OpenAI + CadQuery 2.4

Claude Opus 4.7 → OpenSCAD

Anthropic + OpenSCAD 2024.06

0.5880✗

OpenAI o4 (reasoning) → CadQuery

OpenAI + CadQuery 2.4

0.5879✗

Claude Opus 4.7 → CadQuery

Anthropic + CadQuery 2.4

0.58010✗

Gemini 2.5 Pro → OpenSCAD

Google + OpenSCAD 2024.06

Claude Sonnet 4.6 → CadQuery

Anthropic + CadQuery 2.4

0.54313✗

DeepSeek R1 (reasoning) → CadQuery

DeepSeek + CadQuery 2.4

0.49514✗

Gemini 2.5 Flash → CadQuery

Google + CadQuery 2.4

0.48710✗

Qwen3 Coder → CadQuery

Alibaba + CadQuery 2.4

0.43813✗

Claude Haiku 4.5 → CadQuery

Anthropic + CadQuery 2.4

0.39812✗

CAD-Coder R1

CAD-Coder Labs (research)

0.39711✗

Llama 3.3 70B → OpenSCAD

Meta + OpenSCAD 2024.06

0.36615✗

GPT-5 Mini → OpenSCAD

OpenAI + OpenSCAD 2024.06

0.34217✗

DeepCAD

Wu et al. 2021 (research)

no manifold solid produced

Hunyuan3D-2

Tencent

—5✗

Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.

§5Per-agent metrics

ranked by Vol IoU · same data as the leaderboard, restricted to this task

Agent	Watert.	Manif.	DFM	Min-Wall Compliance	CAM Reachability	p50	latency	cost
Human Baseline (Mech-E)	✓	0.963	91.6	0.884	0.861	—	762.3s	$7.056
GPT-5 → CadQuery	✓	0.952	81.1	0.580	0.677	—	34.2s	$0.204
Zoo Text-to-CAD	✓	0.939	83.4	0.665	0.669	—	7.4s	$0.161
Claude Opus 4.7 → OpenSCAD	✓	0.934	77.8	0.557	0.537	—	24.9s	$0.252
OpenAI o4 (reasoning) → CadQuery	✓	0.933	84.1	0.656	0.715	—	99.2s	$1.097
Claude Opus 4.7 → CadQuery	✓	0.943	81.6	0.627	0.640	—	30.5s	$0.294
Gemini 2.5 Pro → OpenSCAD	✓	0.934	73.0	0.504	0.504	—	28.6s	$0.084
Adam (CADcrush)	✓	0.932	80.8	0.622	0.674	—	11.4s	$0.268
Claude Sonnet 4.6 → CadQuery	✓	0.934	82.5	0.587	0.655	—	16.2s	$0.083
DeepSeek R1 (reasoning) → CadQuery	✓	0.927	74.5	0.521	0.510	—	95.7s	$0.045
Gemini 2.5 Flash → CadQuery	✓	0.922	75.5	0.548	0.556	—	8.7s	$0.018
Qwen3 Coder → CadQuery	✓	0.917	75.0	0.481	0.494	—	17.9s	$0.026
Claude Haiku 4.5 → CadQuery	✓	0.915	66.8	0.446	0.434	—	8.4s	$0.020
CAD-Coder R1	✓	0.914	74.8	0.549	0.533	—	5.5s	$0.006
Llama 3.3 70B → OpenSCAD	×	0.908	66.9	0.397	0.374	—	16.4s	$0.018
GPT-5 Mini → OpenSCAD	×	0.905	71.9	0.416	0.341	—	16.7s	$0.009
DeepCAD	×	0.883	67.3	0.355	0.361	—	4.3s	$0.023
Trellis 3D	×	0.872	50.8	0.213	0.097	—	13.7s	$0.047
Spline AI	×	0.850	48.1	0.092	0.045	—	9.0s	$0.043
Hunyuan3D-2 kernel error: BRepCheck_NotClosed	×	0.000	—	—	—	—	32.6s	$0.072