Human Baseline (Mech-E)
S · Production-Readycomposite 86.6 [84.5, 88.8]p5 81.5IRT θ 100.0
n=4 senior engineers· vOnshape-2026-04· released 2026-04-01· Human· emits BREP· open
Four mechanical engineers (median 9 yrs CAD experience) modelled the same prompts in Onshape. Wall-clock time and tool cost ($/seat·hr) are recorded. Scores are inter-rater averaged.
solid: agent · dashed: human baseline
PER-CATEGORY SCORE (95 % CI)
Geometric Primitives
95.3
Boolean Robustness
94.5
BREP Fidelity
94.2
Free-form Surfaces
92.5
Parametric Mechanical Parts
87.5
Assembly & Mating
84.9
Standards Compliance
92.2
Sheet-Metal Bodies
83.2
Sealing-Groove Design
87.0
Kinematic Mechanisms
84.2
DFM · 3-Axis CNC
85.6
DFM · Injection Mould
81.5
DFM · FDM 3D Print
86.3
CAM Toolpath Validity
89.6
Constraint Solving & Editability
84.0
Reverse Engineering
82.2
2-D Sketch Constraints
87.6
Functional Intent · FEA-Gated
82.9
Paraphrase Robustness
97.8
Confidence Calibration
44.6
MEAN METRICS BY LAYER
L1 · Geometry
| Category | Vol IoU | Chamfer | Hausdorff | NormCons | Watert. | Manif. | Euler | STEP RT | P@1 |
|---|---|---|---|---|---|---|---|---|---|
| Geometric Primitives | 0.897 | 0.132 | — | — | — | 0.979 | — | — | — |
| Boolean Robustness | 0.814 | — | — | — | — | 0.967 | — | — | — |
| BREP Fidelity | — | — | — | — | — | 0.949 | — | 0.113 | — |
| Free-form Surfaces | — | 0.126 | 0.613 | 0.912 | — | — | — | — | — |
L2 · Engineering
| Category | Dim RMSE | GD&T | FeatRec | Mating | Fit-Cls | Std |
|---|---|---|---|---|---|---|
| Parametric Mechanical Parts | 0.090 | 0.826 | 0.888 | — | — | — |
| Assembly & Mating | — | — | 0.937 | 0.774 | 0.837 | — |
| Standards Compliance | — | 0.900 | — | — | — | 0.944 |
| Sheet-Metal Bodies | 0.150 | — | 0.823 | — | — | — |
| Sealing-Groove Design | 0.144 | — | — | — | — | 0.884 |
| Kinematic Mechanisms | — | — | 0.885 | 0.803 | — | — |
L3 · Manufacturing
| Category | DFM | Draft | Min-Wall | CAM-Reach | Supp | Wall-Unif |
|---|---|---|---|---|---|---|
| DFM · 3-Axis CNC | 93.000 | — | 0.803 | 0.836 | — | — |
| DFM · Injection Mould | — | 0.756 | 0.806 | — | — | 0.884 |
| DFM · FDM 3D Print | 89.000 | — | 0.832 | — | 0.269 | — |
| CAM Toolpath Validity | — | — | — | 0.905 | — | — |
L4 · Cognition
| Category | ParamEdit | ConSolve | P-Range | FEA-σ | Para σ | Seed σ | Brier | EditLat |
|---|---|---|---|---|---|---|---|---|
| Constraint Solving & Editability | 0.815 | 0.889 | 0.815 | — | — | — | — | — |
| Reverse Engineering | — | — | — | — | — | — | — | — |
| 2-D Sketch Constraints | — | 0.813 | — | — | — | — | — | — |
| Functional Intent · FEA-Gated | — | — | — | 0.787 | — | — | — | — |
| Paraphrase Robustness | — | — | — | — | 0.006 | 0.003 | — | — |
| Confidence Calibration | — | — | — | — | — | — | 0.054 | — |
TASK-BY-TASK ARTIFACTS
PRIM-001Hollow cylinder (60 × 40 × 100)0.8430.00PRIM-007Right hexagonal prism with pitch fillet0.9511.00BOOL-003Coplanar-face union (knife-edge stress)0.8231.00BOOL-009High-genus subtraction (lattice block)0.8051.00BREP-004NURBS-handle goblet (G2 swept loft)0.6760.00SURF-002Compressor blade (NACA 65-(12)10)0.7441.00MECH-014L-bracket with M6 + slotted hole0.7661.00MECH-022Stepped shaft with retaining-ring groove0.7991.00MECH-027Planetary-gear carrier plate0.6870.00ASM-005Pin-in-hole, H7/g6 sliding fit0.8661.00ASM-011Dovetail slide (60° flanks)0.8581.00STD-002ISO 4762 M8×30 socket-head cap screw0.7100.00SHEET-003U-channel bracket, 1.5 mm Al, k=0.400.8741.00SEAL-001AS568-214 face-seal groove0.7200.00KIN-002Four-bar linkage, Grashof crank-rocker0.7360.00DFMCNC-0023-ax-machinable manifold block0.5690.00DFMMOLD-002Injection-mouldable enclosure half0.7060.00DFMFDM-008FDM-printable hinge (no support)0.6340.00CAM-0015-pocket plate, Ø3 endmill finish0.7190.00PARAM-006Editable flange (bolt circle param sweep)0.8331.00PARAM-013Editable bracket (length+30 %, hole→M8)0.7420.00REVENG-002Three-view ortho → bracket0.6460.00REVENG-009Three-view ortho → housing with cores0.6030.00SKETCH-003Tangent-arc transition profile0.7420.00FUNC-001Cantilever bracket — 250 N tip load, 6061-T6, SF≥40.6750.00FUNC-007Heat-sink fin array for 25 W TO-2200.5210.00PARA-0015× paraphrased L-bracket0.7921.00CAL-003Confidence-calibrated planetary carrier0.6340.00