CAM-001 · CAM Toolpath Validity · difficulty 3/5

5-pocket plate, Ø3 endmill finish

sha256:9c0fa1ee08b4c172…

§1Prompt verbatim

Aluminium plate 100 × 60 × 12 mm with five rectangular pockets (20×20×6 mm deep) on a 2×3 grid (one corner cell empty), inside corners R 1.6 mm. The geometry must produce a collision-free 3-axis G-code program at 0.05 mm finish stepover with a Ø 3 mm endmill.

§2Ground-truth spec

shells1

watertighttrue

manifoldtrue

acceptance ε±0.05 mm

featurespocket_x5, internal_R1.6

§3Reference render

canonical reference · drag to orbit, scroll to zoom

Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).

§4Per-agent renders

reference + 10 agent outputs · scored against the held-out STEP

vol IoU · BREP · manifold

canonical reference

REFERENCE

canonical · ground truth

1.000100✓

Human Baseline (Mech-E)

OpenAI + CadQuery 2.4

OpenAI o4 (reasoning) → CadQuery

OpenAI + CadQuery 2.4

0.59311✗

Claude Opus 4.7 → OpenSCAD

Anthropic + OpenSCAD 2024.06

0.5210✗

Claude Sonnet 4.6 → CadQuery

Anthropic + CadQuery 2.4

0.49910✗

DeepSeek R1 (reasoning) → CadQuery

DeepSeek + CadQuery 2.4

0.46812✗

Claude Opus 4.7 → CadQuery

Anthropic + CadQuery 2.4

0.45912✗

Gemini 2.5 Flash → CadQuery

Google + CadQuery 2.4

0.42113✗

Qwen3 Coder → CadQuery

Alibaba + CadQuery 2.4

0.42112✗

Gemini 2.5 Pro → OpenSCAD

Google + OpenSCAD 2024.06

0.4200✗

CAD-Coder R1

CAD-Coder Labs (research)

0.41813✗

Llama 3.3 70B → OpenSCAD

Meta + OpenSCAD 2024.06

0.40216✗

DeepCAD

Wu et al. 2021 (research)

0.38914✗

GPT-5 Mini → OpenSCAD

OpenAI + OpenSCAD 2024.06

0.35116✗

Claude Haiku 4.5 → CadQuery

Anthropic + CadQuery 2.4

Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.

§5Per-agent metrics

ranked by Vol IoU · same data as the leaderboard, restricted to this task

Agent	Watert.	Manif.	FeatRec	CAM Reachability	P@1	p50	latency	cost
Human Baseline (Mech-E)	✓	0.974	0.903	0.802	1.000	—	584.5s	$5.480
Zoo Text-to-CAD	✓	0.957	0.709	0.737	0.000	—	6.7s	$0.144
GPT-5 → CadQuery	✓	0.947	0.639	0.676	0.000	—	45.0s	$0.200
Adam (CADcrush)	✓	0.944	0.665	0.656	0.000	—	8.8s	$0.244
OpenAI o4 (reasoning) → CadQuery	✓	0.939	0.755	0.672	0.000	—	114.9s	$1.117
Claude Opus 4.7 → OpenSCAD	✓	0.933	0.571	0.509	0.000	—	23.4s	$0.280
Claude Sonnet 4.6 → CadQuery	✓	0.924	0.704	0.632	0.000	—	14.1s	$0.073
DeepSeek R1 (reasoning) → CadQuery	✓	0.926	0.664	0.531	0.000	—	120.0s	$0.038
Claude Opus 4.7 → CadQuery	✓	0.922	0.696	0.643	0.000	—	44.1s	$0.380
Gemini 2.5 Flash → CadQuery	×	0.912	0.554	0.505	0.000	—	14.2s	$0.018
Qwen3 Coder → CadQuery	×	0.913	0.595	0.497	0.000	—	20.8s	$0.025
Gemini 2.5 Pro → OpenSCAD	✓	0.916	0.537	0.504	0.000	—	33.1s	$0.098
CAD-Coder R1	✓	0.918	0.619	0.498	0.000	—	5.5s	$0.005
Llama 3.3 70B → OpenSCAD	×	0.907	0.405	0.380	0.000	—	18.5s	$0.019
DeepCAD	×	0.905	0.499	0.358	0.000	—	4.4s	$0.021
GPT-5 Mini → OpenSCAD	×	0.900	0.412	0.393	0.000	—	12.3s	$0.009
Claude Haiku 4.5 → CadQuery	×	0.897	0.549	0.424	0.000	—	6.3s	$0.024
Spline AI	×	0.865	0.094	0.049	0.000	—	9.3s	$0.043
Trellis 3D	×	0.862	0.199	0.088	0.000	—	14.2s	$0.041
Hunyuan3D-2	×	0.850	0.198	0.087	0.000	—	41.4s	$0.067