REVENG-009 · Reverse Engineering · difficulty 5/5
Three-view ortho → housing with cores
sha256:60189cae3b771acc…
§1Prompt verbatim
Reproduce the 80 × 60 × 40 mm housing from the supplied multi-view drawing including all M4 tapped holes, draft, and ribs. Drawing follows ASME Y14.5-2018 third-angle convention.
§2Ground-truth spec
shells1
watertighttrue
manifoldtrue
acceptance ε±0.1 mm
featuresthread_M4_x6, rib_x4, draft_1deg
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.7378✓
Trellis 3D
Trellis 3D
Microsoft Research
0.6230✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.57913✗
Hunyuan3D-2
Hunyuan3D-2
Tencent
0.47011✗
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.44110✗
OpenAI o4 (reasoning) → CadQuery
OpenAI o4 (reasoning) → CadQuery
OpenAI + CadQuery 2.4
0.42410✗
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.4210✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.4000✗
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.37716✗
DeepSeek R1 (reasoning) → CadQuery
DeepSeek R1 (reasoning) → CadQuery
DeepSeek + CadQuery 2.4
0.35914✗
CAD-Coder R1
CAD-Coder R1
CAD-Coder Labs (research)
0.33918✗
Qwen3 Coder → CadQuery
Qwen3 Coder → CadQuery
Alibaba + CadQuery 2.4
0.29616✗
GPT-5 Mini → OpenSCAD
GPT-5 Mini → OpenSCAD
OpenAI + OpenSCAD 2024.06
0.27421✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.27220✗
Spline AI
Spline AI
Spline.design
0.1910✗
Claude Haiku 4.5 → CadQuery
Claude Haiku 4.5 → CadQuery
Anthropic + CadQuery 2.4
0.18629✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.15431✗
no manifold solid produced
Claude Sonnet 4.6 → CadQuery
Claude Sonnet 4.6 → CadQuery
Anthropic + CadQuery 2.4
—66✗
no manifold solid produced
Gemini 2.5 Flash → CadQuery
Gemini 2.5 Flash → CadQuery
Google + CadQuery 2.4
—59✗
no manifold solid produced
Llama 3.3 70B → OpenSCAD
Llama 3.3 70B → OpenSCAD
Meta + OpenSCAD 2024.06
—14✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Vol IoU | Watert. | Manif. | Named-Dimension RMSE | FeatRec | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|---|
| Human Baseline (Mech-E) | 0.737 | ✓ | 0.957 | 0.068 | 0.838 | 0.000 | — | 905.3s | $6.088 |
| Trellis 3D | 0.623 | ✓ | 0.951 | 0.463 | 0.208 | 0.000 | — | 9.5s | $0.059 |
| Claude Opus 4.7 → CadQuery | 0.579 | ✓ | 0.937 | 0.184 | 0.710 | 0.000 | — | 44.2s | $0.316 |
| Hunyuan3D-2 | 0.470 | ✓ | 0.923 | 0.477 | 0.195 | 0.000 | — | 44.8s | $0.072 |
| GPT-5 → CadQuery | 0.441 | ✓ | 0.922 | 0.287 | 0.632 | 0.000 | — | 41.1s | $0.199 |
| OpenAI o4 (reasoning) → CadQuery | 0.424 | ✓ | 0.919 | 0.173 | 0.720 | 0.000 | — | 89.6s | $1.031 |
| Gemini 2.5 Pro → OpenSCAD | 0.421 | ✓ | 0.915 | 0.231 | 0.539 | 0.000 | — | 21.8s | $0.095 |
| Claude Opus 4.7 → OpenSCAD | 0.400 | × | 0.908 | 0.226 | 0.527 | 0.000 | — | 24.1s | $0.273 |
| Zoo Text-to-CAD | 0.377 | × | 0.910 | 0.182 | 0.769 | 0.000 | — | 6.3s | $0.209 |
| DeepSeek R1 (reasoning) → CadQuery | 0.359 | × | 0.901 | 0.259 | 0.683 | 0.000 | — | 105.9s | $0.033 |
| CAD-Coder R1 | 0.339 | × | 0.902 | 0.318 | 0.636 | 0.000 | — | 7.2s | $0.005 |
| Qwen3 Coder → CadQuery | 0.296 | × | 0.898 | 0.247 | 0.618 | 0.000 | — | 17.4s | $0.032 |
| GPT-5 Mini → OpenSCAD | 0.274 | × | 0.892 | 0.330 | 0.416 | 0.000 | — | 12.9s | $0.011 |
| Adam (CADcrush) | 0.272 | × | 0.888 | 0.226 | 0.704 | 0.000 | — | 8.0s | $0.263 |
| Spline AI | 0.191 | × | 0.881 | 0.544 | 0.089 | 0.000 | — | 6.2s | $0.033 |
| Claude Haiku 4.5 → CadQuery | 0.186 | × | 0.877 | 0.380 | 0.542 | 0.000 | — | 6.8s | $0.020 |
| DeepCAD | 0.154 | × | 0.873 | 0.369 | 0.500 | 0.000 | — | 5.1s | $0.016 |
| Claude Sonnet 4.6 → CadQuery kernel error: BRepCheck_NotClosed | 0.000 | × | 0.000 | — | — | 0.000 | — | 16.8s | $0.063 |
| Gemini 2.5 Flash → CadQuery kernel error: BRepCheck_NotClosed | 0.000 | × | 0.000 | — | — | 0.000 | — | 9.2s | $0.018 |
| Llama 3.3 70B → OpenSCAD kernel error: BRepCheck_NotClosed | 0.000 | × | 0.000 | — | — | 0.000 | — | 18.8s | $0.022 |