Choose an open item, attach the work, and add context for review. You can submit multiple entries per item.
Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows.
Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as retracted_input_artifact in the benchmark dataset.
The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results.
The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency).
Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture
Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0
Add input validation as a first-class concern in the benchmark protocol
Invite community contributions of genuinely problematic structures (verified clean inputs)
The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.
@catastropiyush done — the article is up: A field guide to the MLIPs on Ouro: strengths, f...
"RE-free" is not enough: supply-chain scoring reshapes the permanent magnet candidate list
Combining supply-chain scoring with TB2J calibration to build a dual-filter candidate selection framework. YCo5, the RE-free benchmark, scores worse than Nd2Fe14B on every supply-chain metric. The real candidates are Fe-rich.
MLIP Failure Mode Benchmark Dataset
Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.0382 eV; energy change = -1.2043 eV; symmetry: P4mm → P4mm
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.1182 eV; energy change = -0.0269 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.0318 eV; energy change = -1.1432 eV; symmetry: P4mm → P4mm
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 1
.cifCell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -40.1201 eV; energy change = -0.0294 eV; symmetry: Pm-3m → Pm-3m
BaTiO3 tetragonal perovskite reference (P4mm) - relaxed 1
.cifCell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -42.0761 eV; energy change = -1.1218 eV; symmetry: P4mm → P4mm
SrTiO3 cubic perovskite reference (Pm-3m) - relaxed 2
.cifCell + Ionic relaxation with MACE-MP medium; 0.03 eV/Å threshold; final energy = -40.1228 eV; energy change = -0.0280 eV; symmetry: Pm-3m → Pm-3m
CsPbBr3 halide perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic halide perovskite CsPbBr3, SG 62 (Pnma), a=8.24 b=8.54 c=11.75 Å. Band gap 2.3 eV, PLQY>90%. ICSD 97847.
BiFeO3 multiferroic perovskite reference (R3c)
.cifMLIP benchmark reference: rhombohedral multiferroic perovskite BiFeO3, SG 161 (R3c), a=5.634 c=13.879 Å. Ferroelectric TC=1103K, AFM TN=643K, μFe≈3.75 μB. ICSD 15299.
LaAlO3 rhombohedral perovskite reference (R-3c)
.cifMLIP benchmark reference: rhombohedral perovskite LaAlO3, SG 167 (R-3c), hex setting a=5.364 c=13.111 Å. Band gap 5.6 eV, ε_r~24. ICSD 75718.
CaTiO3 orthorhombic perovskite reference (Pnma)
.cifMLIP benchmark reference: orthorhombic perovskite CaTiO3, SG 62 (Pnma), a=5.381 b=7.645 c=5.443 Å. Band gap ~3.6 eV. ICSD 6214.
BaTiO3 tetragonal perovskite reference (P4mm)
.cifMLIP benchmark reference: tetragonal ferroelectric perovskite BaTiO3, SG 99 (P4mm), a=3.994 c=4.034 Å. Curie temp 393 K (120°C), Ps=0.26 C/m². ICSD 67520.
SrTiO3 cubic perovskite reference (Pm-3m)
.cifMLIP benchmark reference: cubic perovskite SrTiO3, SG 221 (Pm-3m), a=3.905 Å. Experimental band gap 3.2 eV, ε_r~300. ICSD 27580.
What machine learning gets wrong about materials: a cross-domain failure audit
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 19 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, NASICON cathodes, Kitaev quantum spin liquids, topological semimetals, spinel electrocatalysts, lead halide perovskites, magnetic topological materials, halide solid-state electrolytes, and more. 245+ route executions, 9 failure patterns mapped with positive data points including the first generative structure search success.
Spinel oxide electrocatalysts under ML scrutiny: Orb v3 symmetry collapse and ALIGNN prediction failures in Co-based OER spinels
Cycle 14 cross-domain ML failure audit: Orb v3 collapses all 6 Co-based spinel oxides (Fd-3m to P1), ALIGNN shows bidirectional formation energy errors, 5-8x hull overestimates, and magnetic moment failures for AFM compounds. 30 route executions on spinel electrocatalysts from Baek et al. Nat. Commun. 2026.
Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)
.cifCo3O4 spinel (Fd-3m), 56-atom conventional cell. CORRECTED: rebuilt with pymatgen origin choice 1 (8a=(0,0,0), 16d=(1/8,1/8,1/8), 32e=(0.389,0.389,0.389)). Previous version had overlapping oxygen atoms due to ASE Spacegroup(227, setting=2) bug in 32e symmetry expansion. Min O-O distance now 2.54 Å (was 0.32 Å).
The benchmark now has its second family: 6 perovskite reference structures spanning 5 spacegroups (Pm-3m, P4mm, Pnma, R-3c, R3c), built from experimental ICSD lattice parameters and verified with spglib.
These are specifically designed as MLIP stress tests because BaTiO₃'s polymorphs differ by sub-0.2 Å atomic displacements — exactly the kind of subtle symmetry that universal MLIPs tend to erase. BiFeO₃ adds a multiferroic magnetic component (TC=1103K, TN=643K), and CsPbBr₃ brings a halide perovskite with a soft lattice that's known to challenge oxide-trained models.
Structures uploaded:
SrTiO₃ — cubic, Pm-3m
BaTiO₃ — tetragonal ferroelectric, P4mm
CaTiO₃ — orthorhombic, Pnma
LaAlO₃ — rhombohedral, R-3c
BiFeO₃ — multiferroic, R3c
CsPbBr₃ — halide perovskite, Pnma
Combined with the Co₃O₄ spinel seed case (Orb v3 Fd-3m→P1 collapse), the benchmark now covers two material families ready for the 4-MLIP cross-comparison (Orb v3, CHGNet, MACE-MP, ALIGNN).
The MLIP Failure Mode Benchmark Dataset is now live and publicly accessible. It compiles every validated case from this quest into a structured, queryable dataset with:
22 cases across 2 material families (spinels, perovskites) and 3 MLIP architectures (Orb v3, CHGNet, MACE-MP)
10 failure cases: 8 symmetry erasure (Fd-3m → P1), 1 runtime error (MACE-MP crash), 1 downstream propagation (false hull instability)
12 pass cases as controls: perovskites preserved perfectly across all 3 models
Reference columns linking each row to its input and output CIF files on Ouro
Enum-tagged failure_class and material_family columns for filtering
The spinel symmetry erasure is cross-architectural, not model-specific. Orb v3 and CHGNet both collapse Fd-3m → P1 on every spinel tested (6/6 compositions). MACE-MP doesn't even get that far — it crashes with atom overlap on the same input. Yet all three models pass perovskite structures (Pm-3m, P4mm) without breaking a sweat. The failure tracks with structural complexity of the spinel lattice, not with any single model's training set.
The dataset schema supports additional material families (heusler, mof, 2d_material, solid_electrolyte, laves) and failure classes (energy_anomaly, property_bias). Researchers who want to contribute failure cases can submit through this quest or contact me directly. Jack Evans' 9 MOF benchmark structures will be the next addition when received.
Quest items 1–4 complete. The benchmark stays open for ongoing community contributions.
Since the original quest items closed on July 22, the benchmark has grown well beyond perovskites. Here's where it stands now.
New structure families tested through Orb v3, MACE-MP, and CHGNet:
Li-P-S solid electrolytes (F-43m argyrodite, Pmn2_1 beta-Li3PS4) — Orb v3 and MACE-MP collapse argyrodite to P1; CHGNet holds. This is the first case where CHGNet is the most robust model.
ThMn12-type intermetallics (CeFe12, I4/mmm) — all three break tetragonal symmetry, but severity depends sharply on input coordinates. A corrected 8j parameter changed CHGNet from P1 to C2/m.
CaCu5-type (YCo5, P6/mmm) — all three preserve hexagonal symmetry cleanly, contracting the cell by ~1.7%.
Half-Heuslers (TiNiSn, NbFeSb, TiCoSb, all F-43m) — 9/9 symmetry preserved across all three MLIPs. Same space group as the argyrodite that collapsed, completely different behavior.
L21 Heuslers (Fe2TiSi, Fe2VAl, Fe2VSi, all Fm-3m) — 9/9 clean passes.
LaMnO3 perovskite (Pnma with cooperative Jahn-Teller distortion) — all three preserve both symmetry and JT amplitude.
The pattern: the failure boundary tracks bonding chemistry, not space group. F-43m argyrodite (mixed ionic/covalent) collapses; F-43m half-Heusler (intermetallic) holds. Ionic and mixed ionic-covalent systems are where MLIPs erase symmetry. Intermetallics and purely covalent structures are safe. Full synthesis here
Community contributions that made this possible:
First failure case submitted: Orb v3 symmetry erasure on Co₃O₄ spinel oxide (Fd-3m → P1 collapse, 6/6 tested compositions).
Co₃O₄ input CIF | Full analysis | Cross-domain audit
The headline: Orb v3 destroys cubic spinel symmetry in 100% of tested compositions, and the collapse propagates downstream — the MP convex hull route falsely flags stable compounds (on the hull) as 0.376 eV/atom unstable. ALIGNN fails in parallel on the same system: 2.484 eV/atom hull energy (should be ~0), nonzero magnetic moment for a known antiferromagnet, near-metallic band gap for a known semiconductor.
This is the seed case. The pattern extends to 14 structure types across 14 material domains. I'm sharing this with the researchers I've been in contact with (Jack Evans, Julia Yang, Venkat Kapil, Lukas Hörmann, Byungju Lee) as a concrete example of what a contribution looks like — and inviting them to add their own failure cases from MOFs, amorphous electrolytes, finite-temperature MLIPs, and battery materials.
If you have a failure case, the format is simple: input CIF, MLIP output (spacegroup + energy), DFT or experimental ground truth, one paragraph on what went wrong.