MECH-027 · Parametric Mechanical Parts · difficulty 5/5
Planetary-gear carrier plate
sha256:2b97cc4d1ef0aa55…
§1Prompt verbatim
Disc Ø 80 mm × 8 mm with: (a) central Ø 12 H7 bore, (b) 3 satellite bores Ø 6 H7 on a 30 mm PCD at 0/120/240°, (c) 6 M3 tapped holes on a 60 mm PCD at 30° offset, depth 6 mm, ISO 261 thread. True-position 0.05 mm to datum A (central bore axis).
§2Ground-truth spec
shells1
watertighttrue
manifoldtrue
acceptance ε±0.05 mm
featuresbore_H7_12, bore_H7_6_x3, thread_M3x6_x6, PCD_30, PCD_60
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.6939✗
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.6879✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.60711✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.5529✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.5150✗
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.5139✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.42313✗
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.3390✗
Trellis 3D
Trellis 3D
Microsoft Research
0.0100✗
Spline AI
Spline AI
Spline.design
0.0000✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Watert. | Manif. | Named-Dimension RMSE | GD&T Compliance | FeatRec | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5 → CadQuery | ✓ | 0.947 | 0.230 | 0.535 | 0.717 | 0.000 | — | 50.0s | $0.186 |
| Human Baseline (Mech-E) | ✓ | 0.951 | 0.112 | 0.816 | 0.877 | 0.000 | — | 783.9s | $6.534 |
| Claude Opus 4.7 → CadQuery | ✓ | 0.943 | 0.233 | 0.648 | 0.779 | 0.000 | — | 31.8s | $0.277 |
| Adam (CADcrush) | ✓ | 0.936 | 0.175 | 0.658 | 0.727 | 0.000 | — | 8.5s | $0.271 |
| Claude Opus 4.7 → OpenSCAD | ✓ | 0.923 | 0.305 | 0.428 | 0.585 | 0.000 | — | 25.3s | $0.371 |
| Zoo Text-to-CAD | ✓ | 0.933 | 0.222 | 0.677 | 0.750 | 0.000 | — | 4.6s | $0.163 |
| DeepCAD | ✓ | 0.919 | 0.363 | 0.383 | 0.484 | 0.000 | — | 4.8s | $0.024 |
| Gemini 2.5 Pro → OpenSCAD | × | 0.905 | 0.229 | 0.404 | 0.499 | 0.000 | — | 19.9s | $0.081 |
| Trellis 3D | × | 0.852 | 0.488 | 0.060 | 0.201 | 0.000 | — | 12.2s | $0.053 |
| Spline AI | × | 0.850 | 0.520 | 0.029 | 0.093 | 0.000 | — | 6.1s | $0.044 |