Co3O4 spinel input CIF (Fd-3m, 56-atom conventional cell)

Cell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -365.1885 eV; energy change = -39.3694 eV; symmetry: Fd-3m → P1

Cell + Ionic relaxation with Orb v3 conservative inf MPA; 0.03 eV/Å threshold; final energy = -320.4996 eV; energy change = -2115.0730 eV; symmetry: Fd-3m → P1

Cell + Ionic relaxation with CHGNet; 0.03 eV/Å threshold; final energy = -371.9584 eV; energy change = -46.1393 eV; symmetry: Fd-3m → P1

Community benchmark dataset cataloging where universal machine-learned interatomic potentials (MLIPs) break. Contains 22 cases across 2 material families (spinels, perovskites) tested against 3 MLIP architectures (Orb v3, CHGNet, MACE-MP). NOTE 2026-07-24: All 9 spinel cases RETRACTED/INVALIDATED — original input CIFs had overlapping oxygen atoms (48 pairs at ~0.32 A). Corrected Co3O4 preserves Fd-3m under Orb v3 (verified). Other 4 spinels untested with corrected CIFs. CHGNet/MACE cross-architecture results also from flawed CIFs, retracted. Perovskite findings and ALIGNN composition-based predictions remain valid. CC-BY 4.0.

Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows. Important correction (July 27, 2026) Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was wrong. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 preserves Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as in the benchmark dataset. The real lesson is about input validation, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results. What remains valid The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The benchmark dataset has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency). Next steps Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0 Add input validation as a first-class concern in the benchmark protocol Invite community contributions of genuinely problematic structures (verified clean inputs) The benchmark dataset is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a verified clean input, we welcome the contribution.