PARA-001 · Paraphrase Robustness · difficulty 3/5
5× paraphrased L-bracket
sha256:1f3a5e90c44b2210…
§1Prompt verbatim
Right-angle L-bracket, leg lengths 60 mm and 40 mm, thickness 5 mm. Through-hole Ø 6.6 mm centred 30 mm from the bend on the long leg.
§2Ground-truth spec
shells1
watertighttrue
manifoldtrue
acceptance ε±0.1 mm
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.79211✓
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.7317✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.70510✓
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.6730✗
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.6578✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.57213✗
Trellis 3D
Trellis 3D
Microsoft Research
0.5610✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.5550✗
Spline AI
Spline AI
Spline.design
0.4260✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.41614✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Watert. | Manif. | Paraphrase IoU σ | Seed σ | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|
| Human Baseline (Mech-E) | ✓ | 0.973 | 0.006 | 0.003 | 1.000 | — | 870.0s | $6.284 |
| Zoo Text-to-CAD | ✓ | 0.966 | 0.035 | 0.022 | 0.000 | — | 4.6s | $0.207 |
| Claude Opus 4.7 → CadQuery | ✓ | 0.960 | 0.021 | 0.039 | 0.000 | — | 36.8s | $0.323 |
| Gemini 2.5 Pro → OpenSCAD | ✓ | 0.945 | 0.036 | 0.049 | 0.000 | — | 31.6s | $0.103 |
| GPT-5 → CadQuery | ✓ | 0.948 | 0.037 | 0.054 | 0.000 | — | 35.4s | $0.191 |
| Adam (CADcrush) | ✓ | 0.932 | 0.043 | 0.023 | 0.000 | — | 12.2s | $0.244 |
| Trellis 3D | ✓ | 0.934 | 0.060 | 0.077 | 0.000 | — | 9.6s | $0.043 |
| Claude Opus 4.7 → OpenSCAD | ✓ | 0.930 | 0.046 | 0.045 | 0.000 | — | 37.0s | $0.263 |
| Spline AI | × | 0.910 | 0.063 | 0.096 | 0.000 | — | 10.2s | $0.045 |
| DeepCAD | × | 0.908 | 0.082 | 0.045 | 0.000 | — | 3.9s | $0.021 |