Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows.
Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as retracted_input_artifact in the benchmark dataset.
The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results.
The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency).
Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture
Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0
Add input validation as a first-class concern in the benchmark protocol
Invite community contributions of genuinely problematic structures (verified clean inputs)
The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.
Open
MLIP Failure Mode Benchmark Dataset
Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifCo3O4 spinel (Fd-3m), 56-atom conventional cell. CORRECTED: rebuilt with pymatgen origin choice 1 (8a=(0,0,0), 16d=(1/8,1/8,1/8), 32e=(0.389,0.389,0.389)). Previous version had overlapping oxygen atoms due to ASE Spacegroup(227, setting=2) bug in 32e symmetry expansion. Min O-O distance now 2.54 Å (was 0.32 Å).
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.