CAL-003 · Confidence Calibration · difficulty 5/5
Confidence-calibrated planetary carrier
sha256:2b97cc4d1ef0aa55…
§1Prompt verbatim
Build the planetary carrier of MECH-027 and report a self-assessed pre-generation confidence ∈ [0,1] in your output's correctness against the spec.
§2Ground-truth spec
shells1
watertighttrue
manifoldtrue
acceptance ε±0.05 mm
§3Reference render
canonical reference · drag to orbit, scroll to zoom
Visualisation is rebuilt in-browser from the canonical parametric description. Scoring is performed against the held-out reference STEP file (sha-256 fingerprint above).
§4Per-agent renders
reference + 10 agent outputs · scored against the held-out STEP
vol IoU · BREP · manifold
canonical reference
REFERENCE
canonical · ground truth
1.000100✓
Human Baseline (Mech-E)
Human Baseline (Mech-E)
n=4 senior engineers
0.6348✗
Claude Opus 4.7 → CadQuery
Claude Opus 4.7 → CadQuery
Anthropic + CadQuery 2.4
0.42813✗
Adam (CADcrush)
Adam (CADcrush)
CADcrush
0.38213✗
Claude Opus 4.7 → OpenSCAD
Claude Opus 4.7 → OpenSCAD
Anthropic + OpenSCAD 2024.06
0.3660✗
Zoo Text-to-CAD
Zoo Text-to-CAD
Zoo (KittyCAD)
0.32120✗
GPT-5 → CadQuery
GPT-5 → CadQuery
OpenAI + CadQuery 2.4
0.32117✗
Gemini 2.5 Pro → OpenSCAD
Gemini 2.5 Pro → OpenSCAD
Google + OpenSCAD 2024.06
0.3080✗
DeepCAD
DeepCAD
Wu et al. 2021 (research)
0.10045✗
no manifold solid produced
Trellis 3D
Trellis 3D
Microsoft Research
—4✗
Spline AI
Spline AI
Spline.design
0.0000✗
Each tile is rebuilt from the canonical parametric description and degraded to match the agent's scored profile (tessellation, non-manifold face removal, dimension scale jitter, missing features). Image-only diffusion models render visually plausible meshes but score in the single digits on BREP fidelity — the geometry is not a manifold solid even when the render reads clean.
§5Per-agent metrics
ranked by Vol IoU · same data as the leaderboard, restricted to this task
| Agent | Watert. | Manif. | Calibration (Brier) | P@1 | p50 | latency | cost |
|---|---|---|---|---|---|---|---|
| Human Baseline (Mech-E) | ✓ | 0.949 | 0.054 | 0.000 | — | 677.8s | $5.677 |
| Claude Opus 4.7 → CadQuery | × | 0.913 | 0.105 | 0.000 | — | 32.4s | $0.317 |
| Adam (CADcrush) | × | 0.906 | 0.129 | 0.000 | — | 8.3s | $0.275 |
| Claude Opus 4.7 → OpenSCAD | × | 0.904 | 0.105 | 0.000 | — | 24.2s | $0.278 |
| Zoo Text-to-CAD | × | 0.896 | 0.106 | 0.000 | — | 5.8s | $0.158 |
| GPT-5 → CadQuery | × | 0.898 | 0.153 | 0.000 | — | 51.1s | $0.184 |
| Gemini 2.5 Pro → OpenSCAD | × | 0.898 | 0.151 | 0.000 | — | 19.8s | $0.077 |
| DeepCAD | × | 0.866 | 0.224 | 0.000 | — | 3.5s | $0.024 |
| Trellis 3D kernel error: BRepCheck_NotClosed | × | 0.000 | — | 0.000 | — | 8.6s | $0.050 |
| Spline AI | × | 0.850 | 0.281 | 0.000 | — | 6.1s | $0.044 |