SKETCH-003 · 2-D Sketch Constraints · difficulty 2/5
Tangent-arc transition profile
sha256:12aa30c87ef00921…
§1Prompt verbatim
Closed profile: horizontal segment 50 mm, tangent-arc R 20 mm sweeping 90°, vertical segment 50 mm, tangent-arc R 20 mm sweeping 90° back to start. Apply tangent + equal-length + perpendicular constraints. Sketch must be fully constrained (DOF = 0).
§2Ground-truth spec
acceptance ε±0.001 mm
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.7617✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.7427✓
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.7348✓
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.7180✓
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.7159✓
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.6729✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.5410✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.16331✗
Trellis 3D
Trellis 3D
Microsoft Research
0.0400✗
Spline AI
Spline AI
Spline.design
0.0000✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Bidirectional Chamfer | Watert. | Manif. | ConSolve | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7 → CadQuery | 0.115 | ✓ | 0.967 | 0.753 | 1.000 | — | 29.1s | $0.346 |
| Human Baseline (Mech-E) | 0.122 | ✓ | 0.972 | 0.813 | 0.000 | — | 901.1s | $5.536 |
| Adam (CADcrush) | 0.136 | ✓ | 0.957 | 0.692 | 0.000 | — | 7.0s | $0.246 |
| Gemini 2.5 Pro → OpenSCAD | 0.155 | ✓ | 0.953 | 0.528 | 0.000 | — | 20.1s | $0.087 |
| Zoo Text-to-CAD | 0.130 | ✓ | 0.951 | 0.740 | 0.000 | — | 7.2s | $0.182 |
| GPT-5 → CadQuery | 0.139 | ✓ | 0.948 | 0.745 | 0.000 | — | 48.7s | $0.169 |
| Claude Opus 4.7 → OpenSCAD | 0.157 | ✓ | 0.936 | 0.606 | 0.000 | — | 23.9s | $0.371 |
| DeepCAD | 0.475 | × | 0.876 | 0.312 | 0.000 | — | 4.4s | $0.021 |
| Trellis 3D | 1.610 | × | 0.856 | 0.057 | 0.000 | — | 14.6s | $0.040 |
| Spline AI | 1.607 | × | 0.850 | 0.045 | 0.000 | — | 5.9s | $0.046 |