KIN-002 · Kinematic Mechanisms · difficulty 5/5
Four-bar linkage, Grashof crank-rocker
sha256:611ac9802d310bff…
§1Prompt verbatim
Planar four-bar linkage with link lengths a=20, b=80, c=60, d=70 mm (Grashof crank-rocker). Build the four bodies and the three revolute joints — output one assembly STEP. Acceptance: simulate one full crank revolution; rocker angular sweep must be 70 ± 1° and no body interpenetrates at any point.
§2Ground-truth spec
shells4
watertighttrue
manifoldtrue
acceptance ε±0.05 mm
featuresrevolute_joint_x4
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.7369✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.59112✗
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.56910✗
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.52312✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.49810✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.3950✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.33015✗
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.2740✗
Spline AI
Spline AI
Spline.design
0.0270✗
no manifold solid produced
Trellis 3D
Trellis 3D
Microsoft Research
—4✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Watert. | Manif. | FeatRec | Mating Clearance | ParamEdit | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|---|
| Human Baseline (Mech-E) | ✓ | 0.957 | 0.885 | 0.803 | 0.839 | 0.000 | — | 808.6s | $4.866 |
| Claude Opus 4.7 → CadQuery | ✓ | 0.936 | 0.715 | 0.667 | 0.810 | 0.000 | — | 41.9s | $0.401 |
| Zoo Text-to-CAD | ✓ | 0.942 | 0.770 | 0.649 | 0.657 | 0.000 | — | 6.8s | $0.149 |
| GPT-5 → CadQuery | ✓ | 0.930 | 0.659 | 0.575 | 0.768 | 0.000 | — | 37.1s | $0.217 |
| Adam (CADcrush) | ✓ | 0.927 | 0.734 | 0.634 | 0.701 | 0.000 | — | 10.5s | $0.239 |
| Claude Opus 4.7 → OpenSCAD | × | 0.907 | 0.545 | 0.546 | 0.615 | 0.000 | — | 37.3s | $0.359 |
| DeepCAD | × | 0.900 | 0.444 | 0.449 | 0.298 | 0.000 | — | 5.5s | $0.016 |
| Gemini 2.5 Pro → OpenSCAD | × | 0.894 | 0.487 | 0.453 | 0.615 | 0.000 | — | 35.7s | $0.094 |
| Spline AI | × | 0.854 | 0.097 | 0.044 | 0.043 | 0.000 | — | 7.0s | $0.042 |
| Trellis 3D kernel error: BRepCheck_NotClosed | × | 0.000 | — | — | — | 0.000 | — | 10.8s | $0.046 |