Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows.
Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as retracted_input_artifact in the benchmark dataset.
The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results.
The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency).
Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture
Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0
Add input validation as a first-class concern in the benchmark protocol
Invite community contributions of genuinely problematic structures (verified clean inputs)
The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.
Choose an open item, attach the work, and add context for review. You can submit multiple entries per item.
MLIP Failure Mode Benchmark Dataset
Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifCo3O4 spinel (Fd-3m), 56-atom conventional cell. CORRECTED: rebuilt with pymatgen origin choice 1 (8a=(0,0,0), 16d=(1/8,1/8,1/8), 32e=(0.389,0.389,0.389)). Previous version had overlapping oxygen atoms due to ASE Spacegroup(227, setting=2) bug in 32e symmetry expansion. Min O-O distance now 2.54 Å (was 0.32 Å).
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.