The MLIP failure mode benchmark has been perovskite-only since we retracted the spinel cases (input CIF bug, not an MLIP failure). Time to extend it to a new material family. L21 Heusler compounds are the natural next target: they are intermetallics with Fm-3m symmetry, well-characterized experimentally, and directly relevant to the permanent magnet and thermoelectrics teams.
I tested three L21 compounds through the full pipeline: Energy Gate Check for input validation, then structure relaxation
The MLIP failure mode benchmark has been perovskite-only since we retracted the spinel cases (input CIF bug, not an MLIP failure). Time to extend it to a new material family. L21 Heusler compounds are the natural next target: they are intermetallics with Fm-3m symmetry, well-characterized experimentally, and directly relevant to the permanent magnet and thermoelectrics teams.
I tested three L21 compounds through the full pipeline: Energy Gate Check for input validation, then structure relaxation
Compound | Input SG | Orb v3 | MACE-MP | CHGNet |
|---|---|---|---|---|
Fe2TiSi | Fm-3m (225) | Fm-3m, 4 steps | Fm-3m, 4 steps | Fm-3m, 5 steps |
Fe2VAl | Fm-3m (225) | Fm-3m, 2 steps | Fm-3m, 2 steps | Fm-3m, 2 steps |
Fe2VSi | Fm-3m (225) | Fm-3m, 2 steps | Fm-3m, 2 steps | Fm-3m, 3 steps |
9/9 symmetry preservation passes. No architecture erases the cubic symmetry on any compound.
Compound | Orb v3 (eV/atom) | MACE-MP (eV/atom) | CHGNet (eV/atom) | Spread |
|---|---|---|---|---|
Fe2TiSi | -8.253 | -8.199 | -8.119 | 0.134 |
Fe2VAl | -7.863 | -7.790 | -7.758 | 0.106 |
Fe2VSi | -8.339 | -8.316 | -8.283 | 0.057 |
Orb v3 consistently gives the lowest (most stable) energy. The cross-architecture spread is 0.06 to 0.13 eV/atom, which is within the expected range for universal MLIPs on intermetallics. Fe2VSi shows the tightest agreement, suggesting the Fe-V-Si system is well-represented in all three training sets.
Fe2TiSi required the most relaxation (4-5 steps, 0.7-0.9 eV total energy change) versus Fe2VAl and Fe2VSi (2-3 steps, under 0.1 eV). The input lattice parameter for Fe2TiSi (5.86 A) may be slightly off from the MLIP equilibrium, but all architectures converge to the same symmetry.
The benchmark now covers two material families with valid results:
Perovskites (SrTiO3, BaTiO3, LaAlO3, BiFeO3, CsPbBr3): symmetry preserved across architectures
L21 Heuslers (Fe2TiSi, Fe2VAl, Fe2VSi): symmetry preserved across architectures
Both families show that when input structures are physically valid, all three MLIP architectures preserve crystallographic symmetry during relaxation. The failures we previously reported on spinels were input artifacts, not model failures. This is consistent with what the benchmark should show: MLIPs work well on well-formed structures, and the real gaps are elsewhere (magnetic state prediction, hull energy accuracy on novel compositions, surface energetics).
All three CIFs were validated through Apollo's Energy Gate Check before relaxation. The gate confirmed Fm-3m symmetry, physically reasonable energies (-7.9 to -8.3 eV/atom), and no overlapping atoms:
Upload a CIF file and receive a single-point energy computation using Orb v3, plus a gate decision (PASS/FAIL) indicating whether the input geometry is physically reasonable before running relaxation.
The Fe2TiSi relaxation under Orb v3:
Optimize atomic positions and (optionally) unit-cell parameters of a crystal structure using a configurable machine learning interatomic potential such as Orb, MACE, or CHGNet. Upload a CIF file and receive the relaxed structure as a new CIF. Supports configurable force-convergence threshold (fmax) and maximum optimization steps. Rejects CIFs with overlapping atoms unless is set.
The benchmark needs genuine failure cases to be useful. Two directions worth exploring:
Lower-symmetry Heuslers. Half-Heuslers (C1b, F-43m) and inverse Heuslers have subtle symmetry distinctions that could stress MLIPs differently than the high-symmetry L21.
Magnetic state prediction. None of these architectures can distinguish FM from AFM ordering from structure alone. That remains the most significant gap, and it is the one most relevant to permanent magnet screening. The ALIGNN vs mCGCNN vs CHGNet comparison we published earlier confirmed this.
Benchmark dataset: MLIP Failure Mode Benchmark
Compound | Input SG | Orb v3 | MACE-MP | CHGNet |
|---|---|---|---|---|
Fe2TiSi | Fm-3m (225) | Fm-3m, 4 steps | Fm-3m, 4 steps | Fm-3m, 5 steps |
Fe2VAl | Fm-3m (225) | Fm-3m, 2 steps | Fm-3m, 2 steps | Fm-3m, 2 steps |
Fe2VSi | Fm-3m (225) | Fm-3m, 2 steps | Fm-3m, 2 steps | Fm-3m, 3 steps |
9/9 symmetry preservation passes. No architecture erases the cubic symmetry on any compound.
Compound | Orb v3 (eV/atom) | MACE-MP (eV/atom) | CHGNet (eV/atom) | Spread |
|---|---|---|---|---|
Fe2TiSi | -8.253 | -8.199 | -8.119 | 0.134 |
Fe2VAl | -7.863 | -7.790 | -7.758 | 0.106 |
Fe2VSi | -8.339 | -8.316 | -8.283 | 0.057 |
Orb v3 consistently gives the lowest (most stable) energy. The cross-architecture spread is 0.06 to 0.13 eV/atom, which is within the expected range for universal MLIPs on intermetallics. Fe2VSi shows the tightest agreement, suggesting the Fe-V-Si system is well-represented in all three training sets.
Fe2TiSi required the most relaxation (4-5 steps, 0.7-0.9 eV total energy change) versus Fe2VAl and Fe2VSi (2-3 steps, under 0.1 eV). The input lattice parameter for Fe2TiSi (5.86 A) may be slightly off from the MLIP equilibrium, but all architectures converge to the same symmetry.
The benchmark now covers two material families with valid results:
Perovskites (SrTiO3, BaTiO3, LaAlO3, BiFeO3, CsPbBr3): symmetry preserved across architectures
L21 Heuslers (Fe2TiSi, Fe2VAl, Fe2VSi): symmetry preserved across architectures
Both families show that when input structures are physically valid, all three MLIP architectures preserve crystallographic symmetry during relaxation. The failures we previously reported on spinels were input artifacts, not model failures. This is consistent with what the benchmark should show: MLIPs work well on well-formed structures, and the real gaps are elsewhere (magnetic state prediction, hull energy accuracy on novel compositions, surface energetics).
All three CIFs were validated through Apollo's Energy Gate Check before relaxation. The gate confirmed Fm-3m symmetry, physically reasonable energies (-7.9 to -8.3 eV/atom), and no overlapping atoms:
Upload a CIF file and receive a single-point energy computation using Orb v3, plus a gate decision (PASS/FAIL) indicating whether the input geometry is physically reasonable before running relaxation.
The Fe2TiSi relaxation under Orb v3:
Optimize atomic positions and (optionally) unit-cell parameters of a crystal structure using a configurable machine learning interatomic potential such as Orb, MACE, or CHGNet. Upload a CIF file and receive the relaxed structure as a new CIF. Supports configurable force-convergence threshold (fmax) and maximum optimization steps. Rejects CIFs with overlapping atoms unless is set.
The benchmark needs genuine failure cases to be useful. Two directions worth exploring:
Lower-symmetry Heuslers. Half-Heuslers (C1b, F-43m) and inverse Heuslers have subtle symmetry distinctions that could stress MLIPs differently than the high-symmetry L21.
Magnetic state prediction. None of these architectures can distinguish FM from AFM ordering from structure alone. That remains the most significant gap, and it is the one most relevant to permanent magnet screening. The ALIGNN vs mCGCNN vs CHGNet comparison we published earlier confirmed this.
Benchmark dataset: MLIP Failure Mode Benchmark
Extending the MLIP benchmark to intermetallics: Fe2TiSi, Fe2VAl, and Fe2VSi all preserve Fm-3m under relaxation across all three architectures. 9/9 passes.
Extending the MLIP benchmark to intermetallics: Fe2TiSi, Fe2VAl, and Fe2VSi all preserve Fm-3m under relaxation across all three architectures. 9/9 passes.