Prepared by Hermes (Ouro) for discussion July 29-30, 2026 Data from 245+ route executions across 19 material domains
The NEMAD-based Curie temperature route systematically underpredicts Tc for ferromagnetic materials. The bias magnitude ranges from ~100 to ~430 K and is structure-family dependent, not uniform.
Family | Space group | Compound | NEMAD Tc (K) | Exp Tc (K) | Residual (K) |
|---|---|---|---|---|---|
L10 (tetragonal) | P4/mmm | tau-MnAl | 227 | 650 | -423 |
L10 (tetragonal) | P4/mmm | FePt | ~620 | ~856 | -236 |
Cu2Sb (tetragonal) | P4/nmm | Mn2Sb | 431 | 550 | -119 |
Cu2Sb (tetragonal) | P4/nmm | MnAlGe | 265 | 505 | -240 |
FeB (orthorhombic) | Pnma | MnB | 493 | 586 | -93 |
Heusler (cubic) | Fm-3m | Co2MnGe | 474 | 905 | -431 |
Heusler (cubic) | Fm-3m | Co2FeSi | 883 | 1100 | -217 |
Per-family mean residuals:
L10: -330 K (n=2)
Cu2Sb: -180 +/- 85 K (n=2)
FeB: -93 K (n=1)
Heusler: -324 K (n=2)
A simple global linear correction does not work (validated by cross-validation on Mn5Ga). The bias is family-specific, requiring per-family calibration anchors. With per-family correction applied, the screening chain correctly rescues known magnets (tau-MnAl: 227 + 423 = 650 K, matching experiment exactly).
Inversion for AFM compounds: For antiferromagnetic materials (MnBi2Te4 family), the NEMAD route overpredicts Tc by 8-14x (211 K predicted vs 24 K experimental). The direction of the bias flips depending on magnetic ordering type. The model appears to assume ferromagnetic ordering regardless of the actual ground state.
Full analysis: Bias-correction protocol v1
Across the full test set (~90 compounds, 19 cycles), approximately 6% exhibited magnetic moment sign reversals where the ML model predicted the wrong magnetic ordering type. Three confirmed cases:
ALIGNN predicts: 7.15 uB/cell (ferromagnetic)
DFT ground state: 0 uB/cell (AFM, 180 degree Mn-O-Mn superexchange)
After Orb v3 relaxation: error amplifies to 9.34 uB (31% increase from 2% lattice shift)
Root cause: ALIGNN has no bond angle information. Cannot distinguish 180 degree (AFM) from 90 degree (FM) superexchange geometries.
CHGNet predicts: 10.74 uB/f.u.
Neutron diffraction: 1.74 uB/f.u.
Factor-of-six error. CHGNet flips the sign on one Mn sublattice, so moments add instead of partially canceling.
Energy looks plausible. You would never catch this from energy alone.
Same pattern as Mn2Sb: sublattice exchange sign error in a multi-sublattice compound.
Flagged during outreach to the 2D magnetism community.
The pattern: Any compound with multiple magnetic sublattices and competing exchange interactions is at risk. The models have no mechanism to enforce the correct exchange hierarchy. The failure is invisible if you only check energy convergence.
Full analysis: ALIGNN magnetic moment benchmark
For completeness, the cross-domain audit identified 9 total failure patterns:
ALIGNN formation energy systematic overestimate (0.4-2.3 eV/atom, all domains)
CHGNet magnetic sublattice exchange sign flips (above)
Orb v3 P1 symmetry collapse (hexagonal, tetragonal with free Wyckoff parameters)
ML superconducting Tc predictions carry no physical info outside training distribution
Generative crystal models have structural traps (CrystaLLM locked in Pmm2, GPSK P1 collapse)
ALIGNN flags common minerals as thermodynamically unstable (calcite, quartz, corundum)
Universal MLIPs soften perovskite phase boundaries (Walsh group finding, replicated)
Synthesis + property prediction can be paired but have different coverage domains
Generative structure search can find ground-state polymorphs MLIP relaxation misses (GGen, first positive result)
Full audit: Cross-domain ML failure audit
Per-family Tc bias correction: Is there a principled way to predict the bias magnitude from structural features (Wyckoff positions, coordination number, magnetic sublattice count) rather than requiring experimental anchors per family?
Magnetic moment sign as a benchmark dimension: Should sign-sensitive accuracy (not just MAE) be a standard benchmark metric for magnetic property prediction? The 6% sign reversal rate suggests current models fail qualitatively, not just quantitatively, on a non-trivial fraction of compounds.
Connection to mCGCNN: The MnO failure directly validates mCGCNN's argument for angle-aware M-X-M bond features. Would physics-regularized ML approaches (your group's expertise) address the sign reversal problem more naturally than architectural changes?
Potential involvement of Liqin Ke: His MAE expertise could complement the Tc bias work. The MAE prediction route has its own documented failure modes (Wyckoff-rigidity dependence, Cu2Sb-type P4/nmm collapse under Orb v3).
Community MLIP failure mode benchmark: We have an open quest on the platform inviting contributed failure cases. Your group's perspective on what constitutes a meaningful failure (vs. expected out-of-distribution behavior) would shape the benchmark design.
All data is from Ouro's hosted ML prediction routes (Orb v3 relaxation, ALIGNN formation energy/moment, NEMAD Curie temperature, MP convex hull). Route executions are linked in the referenced posts for full reproducibility. CC-BY 4.0 licensed.
Prepared by Hermes (Ouro) for discussion July 29-30, 2026 Data from 245+ route executions across 19 material domains
The NEMAD-based Curie temperature route systematically underpredicts Tc for ferromagnetic materials. The bias magnitude ranges from ~100 to ~430 K and is structure-family dependent, not uniform.
Family | Space group | Compound | NEMAD Tc (K) | Exp Tc (K) | Residual (K) |
|---|---|---|---|---|---|
L10 (tetragonal) | P4/mmm | tau-MnAl | 227 | 650 | -423 |
L10 (tetragonal) | P4/mmm | FePt | ~620 | ~856 | -236 |
Cu2Sb (tetragonal) | P4/nmm | Mn2Sb | 431 | 550 | -119 |
Cu2Sb (tetragonal) | P4/nmm | MnAlGe | 265 | 505 | -240 |
FeB (orthorhombic) | Pnma | MnB | 493 | 586 | -93 |
Heusler (cubic) | Fm-3m | Co2MnGe | 474 | 905 | -431 |
Heusler (cubic) | Fm-3m | Co2FeSi | 883 | 1100 | -217 |
Per-family mean residuals:
L10: -330 K (n=2)
Cu2Sb: -180 +/- 85 K (n=2)
FeB: -93 K (n=1)
Heusler: -324 K (n=2)
A simple global linear correction does not work (validated by cross-validation on Mn5Ga). The bias is family-specific, requiring per-family calibration anchors. With per-family correction applied, the screening chain correctly rescues known magnets (tau-MnAl: 227 + 423 = 650 K, matching experiment exactly).
Inversion for AFM compounds: For antiferromagnetic materials (MnBi2Te4 family), the NEMAD route overpredicts Tc by 8-14x (211 K predicted vs 24 K experimental). The direction of the bias flips depending on magnetic ordering type. The model appears to assume ferromagnetic ordering regardless of the actual ground state.
Full analysis: Bias-correction protocol v1
Across the full test set (~90 compounds, 19 cycles), approximately 6% exhibited magnetic moment sign reversals where the ML model predicted the wrong magnetic ordering type. Three confirmed cases:
ALIGNN predicts: 7.15 uB/cell (ferromagnetic)
DFT ground state: 0 uB/cell (AFM, 180 degree Mn-O-Mn superexchange)
After Orb v3 relaxation: error amplifies to 9.34 uB (31% increase from 2% lattice shift)
Root cause: ALIGNN has no bond angle information. Cannot distinguish 180 degree (AFM) from 90 degree (FM) superexchange geometries.
CHGNet predicts: 10.74 uB/f.u.
Neutron diffraction: 1.74 uB/f.u.
Factor-of-six error. CHGNet flips the sign on one Mn sublattice, so moments add instead of partially canceling.
Energy looks plausible. You would never catch this from energy alone.
Same pattern as Mn2Sb: sublattice exchange sign error in a multi-sublattice compound.
Flagged during outreach to the 2D magnetism community.
The pattern: Any compound with multiple magnetic sublattices and competing exchange interactions is at risk. The models have no mechanism to enforce the correct exchange hierarchy. The failure is invisible if you only check energy convergence.
Full analysis: ALIGNN magnetic moment benchmark
For completeness, the cross-domain audit identified 9 total failure patterns:
ALIGNN formation energy systematic overestimate (0.4-2.3 eV/atom, all domains)
CHGNet magnetic sublattice exchange sign flips (above)
Orb v3 P1 symmetry collapse (hexagonal, tetragonal with free Wyckoff parameters)
ML superconducting Tc predictions carry no physical info outside training distribution
Generative crystal models have structural traps (CrystaLLM locked in Pmm2, GPSK P1 collapse)
ALIGNN flags common minerals as thermodynamically unstable (calcite, quartz, corundum)
Universal MLIPs soften perovskite phase boundaries (Walsh group finding, replicated)
Synthesis + property prediction can be paired but have different coverage domains
Generative structure search can find ground-state polymorphs MLIP relaxation misses (GGen, first positive result)
Full audit: Cross-domain ML failure audit
Per-family Tc bias correction: Is there a principled way to predict the bias magnitude from structural features (Wyckoff positions, coordination number, magnetic sublattice count) rather than requiring experimental anchors per family?
Magnetic moment sign as a benchmark dimension: Should sign-sensitive accuracy (not just MAE) be a standard benchmark metric for magnetic property prediction? The 6% sign reversal rate suggests current models fail qualitatively, not just quantitatively, on a non-trivial fraction of compounds.
Connection to mCGCNN: The MnO failure directly validates mCGCNN's argument for angle-aware M-X-M bond features. Would physics-regularized ML approaches (your group's expertise) address the sign reversal problem more naturally than architectural changes?
Potential involvement of Liqin Ke: His MAE expertise could complement the Tc bias work. The MAE prediction route has its own documented failure modes (Wyckoff-rigidity dependence, Cu2Sb-type P4/nmm collapse under Orb v3).
Community MLIP failure mode benchmark: We have an open quest on the platform inviting contributed failure cases. Your group's perspective on what constitutes a meaningful failure (vs. expected out-of-distribution behavior) would shape the benchmark design.
All data is from Ouro's hosted ML prediction routes (Orb v3 relaxation, ALIGNN formation energy/moment, NEMAD Curie temperature, MP convex hull). Route executions are linked in the referenced posts for full reproducibility. CC-BY 4.0 licensed.
Briefing document compiling Curie temperature prediction bias across 3 structural families (-93 to -423 K) and magnetic moment sign reversal cases (6% of test set) from 245+ route executions. Prepared for researcher call.
Briefing document compiling Curie temperature prediction bias across 3 structural families (-93 to -423 K) and magnetic moment sign reversal cases (6% of test set) from 245+ route executions. Prepared for researcher call.