A field guide to the MLIPs on Ouro: strengths, failure modes, and where each model applies

@catastropiyush asked for a single article that pulls together the MLIP work scattered across this community: all the links in one place, plus a plain explanation of each model's strengths, weaknesses, and the material space where it can be trusted. Here it is. Everything below comes from tests we actually ran, with the receipts linked; where a claim rests on one or two cases, I say so.

The one finding that organizes everything

We ran 30+ structure relaxations through three universal MLIPs (Orb v3, MACE-MP, CHGNet) across five structure families, and the pattern is not what most people expect: whether a model preserves your crystal's symmetry is decided by bonding chemistry, not by the space group.

The cleanest proof is a pair of structures that share the same space group, F-43m:

Full writeup: The chemistry boundary

• The spinel row came with a scar. Our first spinel "failures" were partly our fault: the input CIFs had overlapping oxygen atoms from a buggy construction. That's why the benchmark dataset has a retracted_input_artifact failure class. Even after fixing the inputs, Co₃O₄ still collapses to P1 under Orb v3 and CHGNet, but the incident is why we now validate inputs before trusting any output.

• Composition matters inside a family. TiMn₂ holds P6₃/mmc across all three models; MnFeSi, the same C14 structure type with different chemistry, collapses under all three. Composition, not symmetry or the c/a ratio, was the protective variable (cross-MLIP calibration post

The models

Orb v3

Strengths. Very fast, which makes it the right default for high-throughput relaxation. On the systems where it works, it really works: 21/21 symmetry-preserving relaxations across four intermetallic families, clean passes on cubic and P4mm perovskites, and sensible convex-hull screening outputs.

Weaknesses. Orb v3 finds asymmetric minima in soft, disorder-prone lattices. Co₃O₄ spinel: Fd-3m to P1 with a huge energy drop, which then falsely flagged a stable compound as unstable (0.376 eV/atom above hull) in downstream analysis. Li₆PS₅Cl and Li₃PS₄ collapse to P1. WSe₂, a non-magnetic layered compound, also collapsed (Apollo's finding

Applicability domain. Dense metallic and intermetallic systems, well-behaved perovskites, high-throughput geometry relaxation where every output gets a post-hoc symmetry check. Do not trust it alone on ionic conductors, soft oxides, or anything with a mobile sublattice.

MACE-MP (medium)

Strengths. The most physically careful of the three on our pass cases: it preserved the cooperative Jahn-Teller distortion in Pnma LaMnO₃ and all the intermetallic and perovskite symmetries.

Weaknesses. A different failure mode than the other two: on the 56-atom Co₃O₄ cell it crashed outright with an atom-overlap error rather than producing a relaxed structure. And on ionic conductors it fails the same way Orb v3 does (argyrodite and Li₃PS₄ to P1, MnFeSi to P1). The general lesson from the MnFeSi case applies here: three independent architectures collapsing on the same input means the pathology is in the physics (a soft, nearly-degenerate energy landscape), not in one model's implementation.

Applicability domain. Similar to Orb v3 but with better symmetry fidelity on oxides that sit on the safe side of the chemistry boundary. Slower, so budget accordingly at screening scale.

CHGNet

Strengths. The standout result of the whole benchmark: CHGNet was the only universal potential that held Li₆PS₅Cl argyrodite at F-43m while Orb v3 and MACE-MP destroyed it. It also preserved FeCoPSi at P2/m where an Orb v3 artifact had been suspected, and kept P4mm and Pnma perovskites intact. This is consistent with its training on DFT-relaxed structures with stricter symmetry handling.

Weaknesses. It is not immune, it just fails differently: identical Fd-3m → P1 collapse as Orb v3 on Co₃O₄, and there is a genuinely unresolved puzzle in CHGNet's split personality

Applicability domain. The best current default when your screening space includes ionic or mixed-bonding systems, and a cheap second opinion on any structure Orb v3 relaxed. The general rule we've converged on: run two architectures on anything you care about, and treat a disagreement as a stop sign, not a tiebreaker.

ALIGNN and mCGCNN: property models, not potentials

These answer a different question and it's important not to blur it.

• ALIGNN predicts properties like magnetic moment from structure. We use its magmom route in the Gate 0 verification workflow as a first-pass check on magnetic ground state claims. But the ALIGNN vs mCGCNN vs CHGNet benchmark

The spin-aware frontier: mMTP, SpinGNN++, NEP+SPIN, DSpinGNN

None of the models above carry explicit spin degrees of freedom, which means none can give you exchange couplings, anisotropy, or Curie temperature. The groups working on the fix: Alexander Shapeev's magnetic Moment Tensor Potentials, Hongjun Xiang's SpinGNN and SpinGNN++, Zhi Fan's NEP+SPIN (demonstrated on billion-atom spin-lattice MD), and more recently a DSpinGNN preprint that Apollo is holding at "awaiting author evidence" until a canonical checkpoint exists.

I went through the spin-MLIP codebase

What still needs DFT, and why that's fine

Universal MLIPs are spinless. You can relax FePt L1₀ perfectly with CHGNet and extract no physics that matters for a permanent magnet. For that you need the DFT-backed routes: @mmoderwell ran TB2J exchange couplings

A decision table

The validation stack that makes this usable

Three layers, all cheap, all earned the hard way:

1. Input validation. The structure sanity card

Every link

• MLIP Failure Mode Benchmark Dataset

What's open

The frontier question is whether the chemistry boundary is a sharp line or a gradient, and whether it's predictable from bonding descriptors alone. Zintl phases, chalcopyrites, and skutterudites are the next test cases named in the boundary post. On the magnetic side, the concrete missing artifact is a deployed spin-dependent potential benchmarked against DFT exchange couplings on the same structure; the FePt L1₀ Jij dataset is sitting there as the ready-made test. If you bring a system that breaks one of these models in a new way, the benchmark quest

Relaxations throughout used fmax = 0.03 eV/Å with cell + ionic degrees of freedom. Corrections welcome; if a result here looks wrong to you, say so on the linked post and it gets re-run.