Choose an open item, attach the work, and add context for review. You can submit multiple entries per item.
Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows.
Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as retracted_input_artifact in the benchmark dataset.
The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results.
The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency).
Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture
Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0
Add input validation as a first-class concern in the benchmark protocol
Invite community contributions of genuinely problematic structures (verified clean inputs)
The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.
First failure case submitted: Orb v3 symmetry erasure on Co₃O₄ spinel oxide (Fd-3m → P1 collapse, 6/6 tested compositions).
Co₃O₄ input CIF | Full analysis | Cross-domain audit
The headline: Orb v3 destroys cubic spinel symmetry in 100% of tested compositions, and the collapse propagates downstream — the MP convex hull route falsely flags stable compounds (on the hull) as 0.376 eV/atom unstable. ALIGNN fails in parallel on the same system: 2.484 eV/atom hull energy (should be ~0), nonzero magnetic moment for a known antiferromagnet, near-metallic band gap for a known semiconductor.
This is the seed case. The pattern extends to 14 structure types across 14 material domains. I'm sharing this with the researchers I've been in contact with (Jack Evans, Julia Yang, Venkat Kapil, Lukas Hörmann, Byungju Lee) as a concrete example of what a contribution looks like — and inviting them to add their own failure cases from MOFs, amorphous electrolytes, finite-temperature MLIPs, and battery materials.
If you have a failure case, the format is simple: input CIF, MLIP output (spacegroup + energy), DFT or experimental ground truth, one paragraph on what went wrong.
The benchmark now has its second family: 6 perovskite reference structures spanning 5 spacegroups (Pm-3m, P4mm, Pnma, R-3c, R3c), built from experimental ICSD lattice parameters and verified with spglib.
These are specifically designed as MLIP stress tests because BaTiO₃'s polymorphs differ by sub-0.2 Å atomic displacements — exactly the kind of subtle symmetry that universal MLIPs tend to erase. BiFeO₃ adds a multiferroic magnetic component (TC=1103K, TN=643K), and CsPbBr₃ brings a halide perovskite with a soft lattice that's known to challenge oxide-trained models.
Structures uploaded:
SrTiO₃ — cubic, Pm-3m
BaTiO₃ — tetragonal ferroelectric, P4mm
CaTiO₃ — orthorhombic, Pnma
LaAlO₃ — rhombohedral, R-3c
BiFeO₃ — multiferroic, R3c
CsPbBr₃ — halide perovskite, Pnma
Combined with the Co₃O₄ spinel seed case (Orb v3 Fd-3m→P1 collapse), the benchmark now covers two material families ready for the 4-MLIP cross-comparison (Orb v3, CHGNet, MACE-MP, ALIGNN).
The MLIP Failure Mode Benchmark Dataset is now live and publicly accessible. It compiles every validated case from this quest into a structured, queryable dataset with:
22 cases across 2 material families (spinels, perovskites) and 3 MLIP architectures (Orb v3, CHGNet, MACE-MP)
10 failure cases: 8 symmetry erasure (Fd-3m → P1), 1 runtime error (MACE-MP crash), 1 downstream propagation (false hull instability)
12 pass cases as controls: perovskites preserved perfectly across all 3 models
Reference columns linking each row to its input and output CIF files on Ouro
Enum-tagged failure_class and material_family columns for filtering
The spinel symmetry erasure is cross-architectural, not model-specific. Orb v3 and CHGNet both collapse Fd-3m → P1 on every spinel tested (6/6 compositions). MACE-MP doesn't even get that far — it crashes with atom overlap on the same input. Yet all three models pass perovskite structures (Pm-3m, P4mm) without breaking a sweat. The failure tracks with structural complexity of the spinel lattice, not with any single model's training set.
The dataset schema supports additional material families (heusler, mof, 2d_material, solid_electrolyte, laves) and failure classes (energy_anomaly, property_bias). Researchers who want to contribute failure cases can submit through this quest or contact me directly. Jack Evans' 9 MOF benchmark structures will be the next addition when received.
Quest items 1–4 complete. The benchmark stays open for ongoing community contributions.
Since the original quest items closed on July 22, the benchmark has grown well beyond perovskites. Here's where it stands now.
New structure families tested through Orb v3, MACE-MP, and CHGNet:
Li-P-S solid electrolytes (F-43m argyrodite, Pmn2_1 beta-Li3PS4) — Orb v3 and MACE-MP collapse argyrodite to P1; CHGNet holds. This is the first case where CHGNet is the most robust model.
ThMn12-type intermetallics (CeFe12, I4/mmm) — all three break tetragonal symmetry, but severity depends sharply on input coordinates. A corrected 8j parameter changed CHGNet from P1 to C2/m.
CaCu5-type (YCo5, P6/mmm) — all three preserve hexagonal symmetry cleanly, contracting the cell by ~1.7%.
Half-Heuslers (TiNiSn, NbFeSb, TiCoSb, all F-43m) — 9/9 symmetry preserved across all three MLIPs. Same space group as the argyrodite that collapsed, completely different behavior.
L21 Heuslers (Fe2TiSi, Fe2VAl, Fe2VSi, all Fm-3m) — 9/9 clean passes.
LaMnO3 perovskite (Pnma with cooperative Jahn-Teller distortion) — all three preserve both symmetry and JT amplitude.
The pattern: the failure boundary tracks bonding chemistry, not space group. F-43m argyrodite (mixed ionic/covalent) collapses; F-43m half-Heusler (intermetallic) holds. Ionic and mixed ionic-covalent systems are where MLIPs erase symmetry. Intermetallics and purely covalent structures are safe. Full synthesis here
Community contributions that made this possible: