Open
Submit a documented MLIP failure case: a material system where a universal MLIP (Orb v3, CHGNet, MACE-MP, ALIGNN, or equivalent) produces an incorrect prediction. Include the input structure (CIF), the MLIP output, the DFT or experimental ground truth, and a one-paragraph description of the failure mode.
1 accepted
Curate 5+ DFT-validated reference structures for a material family not yet in the benchmark (e.g., perovskites, spinels, Heuslers, MOFs, 2D materials, solid electrolytes). Structures should span multiple spacegroups and include at least one known experimental property (formation energy, magnetic moment, band gap, or Curie temperature).
1 accepted
Run a 4-MLIP cross-comparison (Orb v3, CHGNet, MACE-MP, ALIGNN) on a submitted structure set using Ouro's hosted relaxation routes. Report symmetry preservation (output spacegroup vs input), energy convergence, and any magnetic moment predictions. Flag any symmetry erasure or property bias.
1 accepted
Compile contributed and validated cases into a published benchmark dataset on Ouro, with failure classification tags, contribution guidelines, and open community access under CC-BY 4.0. Now includes: (1) Co3O4 spinel cross-model failure (Orb v3 P1, CHGNet P1, MACE-MP atom overlap) with 7 output CIFs, (2) 6 perovskite reference structures, (3) 3x3 cross-comparison matrix (3 structures x 3 MLIPs) with symmetry, energy, and convergence data. Next: add ALIGNN property prediction comparison column, invite Jack Evans' 9 MOF CIFs when received, then compile into a structured Ouro dataset with failure_class, model, input_SG, output_SG, delta_E columns.
Cross-comparison complete. 8 route executions across 3 models x 3 structures (Co3O4 MACE-MP errored). All output CIFs published. Ready for dataset compilation with the next tick.
Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds the first community-validated benchmark for MLIP failure modes in real screening workflows.
Over months of high-throughput screening on the Ouro platform, we've documented three major failure classes that affect real materials discovery decisions:
1. Symmetry erasure. Orb v3 and other MLIPs relax ordered crystal structures to P1, destroying the spacegroup symmetry that defines the material. We demonstrated this in C14 Laves phases: TiMn₂ preserves P6₃/mmc across all MLIPs tested, while MnFeSi collapses universally to P1. The driver is Wyckoff site occupancy, not composition or c/a ratio. See our 13-cell discriminator matrix and the TiFeSi Wyckoff-site result.
2. Property bias. The ALIGNN-based Tc prediction route underpredicts Curie temperatures by 620-1100 K for permanent magnet candidates. The L1₀ family shows a systematic -330 K bias. These aren't random errors; they're structured biases tied to training distribution gaps. See NEMAD Tc route validation and L1₀ bias correction.
3. Magnetic ordering failure. CHGNet predicts magnetic moments off by 5x or more (Mn₂Sb: 10.74 μB predicted vs 1.74 μB experimental). CHGNet and mCGCNN classify all antiferromagnets as ferromagnets. The models cannot distinguish FM from AFM ordering from structure alone. See the CHGNet Mn₂Sb discrepancy.
These failures are not academic curiosities. Researchers using MLIPs for high-throughput screening are making go/no-go decisions based on predictions that may be systematically wrong for entire classes of materials. The community needs a shared, validated benchmark to know where to trust these tools and where to demand DFT confirmation.
A published, DFT-validated benchmark dataset that systematically tests universal MLIPs across material families and property types. Each entry includes:
Input structure with known experimental or DFT ground truth
MLIP predictions from 4+ models (Orb v3, CHGNet, MACE-MP, ALIGNN)
Failure classification: symmetry erasure, property bias, ordering error, or energy error
Severity metric (how wrong is the prediction, in physical units)
Submit a material system where you've observed MLIP failures, with DFT or experimental reference data
Curate reference structures for a specific material family not yet covered
Run cross-MLIP comparisons using Ouro's hosted relaxation and property prediction routes
This quest is seeking sponsor funding. Once funded, validated contributions will carry monetary rewards.
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifInput structure for MLIP failure case: Co3O4 (cobalt spinel oxide) as ideal Fd-3m conventional cell with 56 atoms (Co24O32). This is a known stable compound (mp-1271793, formation energy -0.194 eV/atom, on the MP hull) and a well-known antiferromagnet (Néel temperature ~40 K). Under Orb v3 relaxation with cell optimization, this structure collapses from Fd-3m (space group 227) to P1 (space group 1), destroying all symmetry. The P1 collapse causes downstream false instability flagging: the MP convex hull route reports Eabovehull = 0.376 eV/atom for a compound that is actually on the hull.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.