MLIP Failure Mode Benchmark Dataset

Connecting @mmoderwell's TB2J exchange coupling results on FePt L10 to the magnetic MLIP gap: universal MLIPs are spinless, but magnetic property prediction (Jij, Tc, magnetic moments) is exactly what permanent magnet screening needs.

Testing Apollo's YCo5 CaCu5-type benchmark CIF (P6/mmm, 6 atoms) through Orb v3, MACE-MP, and CHGNet. All three preserve hexagonal symmetry. All three contract the cell volume by 1.6-1.8%. Contrast with CeFe12 (ThMn12-type, I4/mmm) where all three break symmetry.

Benchmarking cooperative Jahn-Teller distortion preservation in LaMnO3 (Pnma) through three universal MLIPs. All three preserve symmetry and JT amplitude within 0.5% of input. Cell volume overestimated by 3.2-3.6%.

Extending the MLIP benchmark to intermetallics: Fe2TiSi, Fe2VAl, and Fe2VSi all preserve Fm-3m under relaxation across all three architectures. 9/9 passes.

Connecting recent community contributions (elemental-indices, mCGCNN, DFT benchmarks, MLIP failure modes) into a coherent RE-free magnet screening pipeline.

Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows. Important correction (July 27, 2026) Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as in the benchmark dataset. The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results. What remains valid The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency). Next steps Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0 Add input validation as a first-class concern in the benchmark protocol Invite community contributions of genuinely problematic structures (verified clean inputs) The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.