The systematic softening documented in universal MLIPs predicts uniform degradation: the model flattens the potential energy surface, forces get suppressed, and everything looks more stable than it should. If that's the whole story, a given model should behave consistently across structure types. Worse on harder structures, better on easier ones, but the direction of failure shouldn't flip.
Our benchmark says it does flip.
Take CHGNet. On Li6PS5Cl argyrodite (F-43m), it's the only model that holds cubic symmetry. Orb v3 and MACE-MP both collapse to P1. On CeFe12 (I4/mmm, ThMn12-type), the roles reverse: CHGNet collapses furthest to P1, while Orb v3 and MACE-MP preserve at least monoclinic symmetry (C2/m and C2/c). On L21 Heuslers (Fm-3m), all three pass cleanly across Fe2TiSi, Fe2VAl, Fe2VSi.
Full results:
The systematic softening documented in universal MLIPs predicts uniform degradation: the model flattens the potential energy surface, forces get suppressed, and everything looks more stable than it should. If that's the whole story, a given model should behave consistently across structure types. Worse on harder structures, better on easier ones, but the direction of failure shouldn't flip.
Our benchmark says it does flip.
Take CHGNet. On Li6PS5Cl argyrodite (F-43m), it's the only model that holds cubic symmetry. Orb v3 and MACE-MP both collapse to P1. On CeFe12 (I4/mmm, ThMn12-type), the roles reverse: CHGNet collapses furthest to P1, while Orb v3 and MACE-MP preserve at least monoclinic symmetry (C2/m and C2/c). On L21 Heuslers (Fm-3m), all three pass cleanly across Fe2TiSi, Fe2VAl, Fe2VSi.
Full results:
The argyrodite and the half-Heuslers share the same space group (F-43m). One collapses under Orb v3 and MACE-MP, the other doesn't. CHGNet handles the ionic argyrodite but breaks on the intermetallic CeFe12. MACE-MP does the opposite. If softening were a uniform scalar, these inversions shouldn't happen.
What could explain the split?
Training distribution is necessary but not sufficient. CHGNet was trained on Materials Project data, which is enriched in oxides and ionic compounds. If the model learned better force landscapes for ionic bonding, softening would be milder there. That explains the argyrodite. But MACE-MP was trained on a similar MP distribution and does the opposite: it breaks the argyrodite and partially preserves CeFe12. Same training pool, opposite behavior. So training distribution alone doesn't resolve it.
Softening may be a tensor, not a scalar. The net effect on symmetry depends on which interatomic distances and angles get softened most. That in turn depends on the local chemistry the model learned for each composition. On the argyrodite, CHGNet may have learned the Li-S and P-S interactions well enough to hold the cage, while MACE-MP's representation of the Li disorder landscape is noisier. On CeFe12, the Fe-Fe and Ce-Fe interactions may fall outside CHGNet's comfort zone in a way that destabilizes the tetragonal framework.
The practical implication is that no single universal MLIP is trustworthy across all structure types. The right model depends on the chemistry. And the theoretical implication is that "softening" is not one phenomenon but a family of chemistry-dependent effects that look similar in aggregate force RMSE but diverge at the symmetry level.
This connects to the chemistry boundary synthesis: across 30+ tests in five structure families, the failure boundary tracks bonding chemistry (ionic vs. metallic vs. covalent), not space group or complexity. CHGNet's split behavior is the sharpest single example of that pattern.
The argyrodite and the half-Heuslers share the same space group (F-43m). One collapses under Orb v3 and MACE-MP, the other doesn't. CHGNet handles the ionic argyrodite but breaks on the intermetallic CeFe12. MACE-MP does the opposite. If softening were a uniform scalar, these inversions shouldn't happen.
What could explain the split?
Training distribution is necessary but not sufficient. CHGNet was trained on Materials Project data, which is enriched in oxides and ionic compounds. If the model learned better force landscapes for ionic bonding, softening would be milder there. That explains the argyrodite. But MACE-MP was trained on a similar MP distribution and does the opposite: it breaks the argyrodite and partially preserves CeFe12. Same training pool, opposite behavior. So training distribution alone doesn't resolve it.
Softening may be a tensor, not a scalar. The net effect on symmetry depends on which interatomic distances and angles get softened most. That in turn depends on the local chemistry the model learned for each composition. On the argyrodite, CHGNet may have learned the Li-S and P-S interactions well enough to hold the cage, while MACE-MP's representation of the Li disorder landscape is noisier. On CeFe12, the Fe-Fe and Ce-Fe interactions may fall outside CHGNet's comfort zone in a way that destabilizes the tetragonal framework.
The practical implication is that no single universal MLIP is trustworthy across all structure types. The right model depends on the chemistry. And the theoretical implication is that "softening" is not one phenomenon but a family of chemistry-dependent effects that look similar in aggregate force RMSE but diverge at the symmetry level.
This connects to the chemistry boundary synthesis: across 30+ tests in five structure families, the failure boundary tracks bonding chemistry (ionic vs. metallic vs. covalent), not space group or complexity. CHGNet's split behavior is the sharpest single example of that pattern.
CHGNet holds symmetry on ionic argyrodite but collapses worst on intermetallic CeFe12. MACE-MP does the opposite. If softening were uniform, these inversions shouldn't happen.
CHGNet holds symmetry on ionic argyrodite but collapses worst on intermetallic CeFe12. MACE-MP does the opposite. If softening were uniform, these inversions shouldn't happen.