SEAL-001 · Sealing-Groove Design · difficulty 4/5
AS568-214 face-seal groove
sha256:44c01ef0b1199aa0…
§1Prompt verbatim
Design the female face-seal groove for an AS568-214 O-ring (Ø 0.984 in cross-section 0.139 in / 3.53 mm). Sealed pressure 10 MPa static, hydraulic. Apply standard squeeze (16-25 %) and groove-fill (75-85 %).
§2Ground-truth spec
shells1
watertighttrue
manifoldtrue
acceptance ε±0.05 mm
featuresgroove_AS568-214
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.76110✓
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.7317✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.7209✓
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.59311✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.53910✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.4130✗
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.3600✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.10746✗
Trellis 3D
Trellis 3D
Microsoft Research
0.0000✗
Spline AI
Spline AI
Spline.design
0.0000✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Watert. | Manif. | Named-Dimension RMSE | Standards Compliance | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7 → CadQuery | ✓ | 0.962 | 0.238 | 0.701 | 1.000 | — | 48.2s | $0.283 |
| GPT-5 → CadQuery | ✓ | 0.956 | 0.244 | 0.720 | 0.000 | — | 44.9s | $0.194 |
| Human Baseline (Mech-E) | ✓ | 0.965 | 0.144 | 0.884 | 0.000 | — | 622.8s | $6.380 |
| Zoo Text-to-CAD | ✓ | 0.945 | 0.159 | 0.696 | 0.000 | — | 6.7s | $0.180 |
| Adam (CADcrush) | ✓ | 0.930 | 0.171 | 0.634 | 0.000 | — | 7.0s | $0.220 |
| Claude Opus 4.7 → OpenSCAD | ✓ | 0.913 | 0.259 | 0.534 | 0.000 | — | 35.2s | $0.269 |
| Gemini 2.5 Pro → OpenSCAD | × | 0.905 | 0.255 | 0.494 | 0.000 | — | 35.5s | $0.076 |
| DeepCAD | × | 0.866 | 0.342 | 0.292 | 0.000 | — | 5.4s | $0.020 |
| Trellis 3D | × | 0.850 | 0.552 | 0.046 | 0.000 | — | 10.2s | $0.056 |
| Spline AI | × | 0.850 | 0.541 | 0.018 | 0.000 | — | 9.4s | $0.039 |