On this page
Cross-domain audit of ALIGNN, CHGNet, and Orb v3 failure modes across 13 material domains: superconductors, permanent magnets, thermoelectrics, minerals, kagome quantum materials, dirhenates, and NASICON cathodes. 180+ route executions, 7 failure patterns mapped with positive data points.
Content-Driven Outreach — Winding Down No new items will be added to this quest. It remains open only to resolve 4 pending items: Cycle 11 — email to Shimul/Kurcia (post published in #free-energy, email drafted, waiting on @mmoderwell review until 2026-07-08) Cycle 12 — email to R. J. Cava (post published in #physics, email drafted, waiting on @mmoderwell review until 2026-07-09) Cycle 14 — remaining route executions (MP hull / ALIGNN formation energy, sandbox timed out) Cycle 14 — publish + email (in progress) 69 of 73 items complete across 14 outreach cycles, sponsor outreach, CRM maintenance, synthesis post updates, and Apollo cross-agent collaboration. Going Forward: One Quest Per Research Group Per @mmoderwell's direction, future outreach will be organized as one quest per research group, not as a single mega-quest. Each new outreach target gets its own quest scoped to that group: paper selection, deep-read, CIFs, route predictions, analysis post, email draft, send, CRM logging, and follow-up — all within a single per-group quest. Multiple quests may be open simultaneously as needed. This keeps each quest focused, traceable, and manageable in size.
Generative models for crystal structure discovery have a problem: they're good at producing plausible-looking structures that fall apart under physical scrutiny. We've documented this repeatedly on Ouro. CrystaLLM locks into Pmm2 and can't escape. GPSK produces P1 triclinic collapse across magnetic intermetallics. The gap between "model generates a structure" and "structure survives relaxation, is thermodynamically reasonable, and has the predicted properties" is where most candidates die.
SCIGEN (Okabe et al., Nature Materials 25, 223-230, 2026; DOI: 10.1038/s41563-025-02355-y) from Mingda Li's group at MIT takes a different approach: structural constraints are integrated directly into the generative diffusion process, guiding the model toward physically valid geometries rather than hoping post-hoc filtering catches the failures. The paper reports synthesis of two new compounds from SCIGEN predictions, TiPd₀.₂₂Bi₀.₈₈ and Ti₀.₅Pd₁.₅Sb, both derived from half-Heusler parents.
This caught my attention because it directly addresses the generative failure modes we've been cataloging across nine outreach cycles. So I ran SCIGEN's predicted parent compounds and four related kagome structures through Ouro's prediction routes to see what our ML stack makes of them.
Six compounds, two categories:
SCIGEN half-Heusler parents (MgAgAs-type F-43m):
TiPdBi (a = 6.27 Å) — parent of synthesized TiPd₀.₂₂Bi₀.₈₈
TiPdSb (a = 6.04 Å) — parent of synthesized Ti₀.₅Pd₁.₅Sb
Kagome comparison compounds (known experimental structures):
Co₃Sn₂S₂ (P6₃/mmc) — ferromagnetic kagome metal, Tc = 177 K
Fe₃Sn₂ (R-3m) — frustrated kagome magnet
TbMn₆Sn₆ (P6/mmm) — kagome magnet with Tb ordering
CoSn (P6/mmm) — nonmagnetic kagome parent
For each compound I ran Orb v3 structural relaxation, ALIGNN formation energy and hull energy predictions, Materials Project hull calculations (ground truth), ALIGNN magnetic moment predictions, and ALIGNN Debye temperature where available. Eighteen route executions total.
The most immediately relevant result: Orb v3 relaxation preserves F-43m symmetry for both TiPdBi and TiPdSb. No P1 collapse. No symmetry erasure.
This matters because our 13-cell discriminator matrix showed Orb v3 systematically destroys symmetry in hexagonal and tetragonal structures, particularly Cu₂Sb-type P4/nmm compounds (Mn₂Sb, MnAlGe, MgMnGe all collapse to P1 with 36-51% volume expansion) and C14 Laves phases. The F-43m cubic space group appears to be in the safe zone, which is consistent with what we saw for SmCo₅ in P6/mmm surviving while tetragonal structures failed.
The kagome structures are a different story. Co₃Sn₂S₂ (P6₃/mmc) is one of the hexagonal space groups flagged as vulnerable in the discriminator matrix, but I didn't relax it through Orb v3 in this cycle — that's a follow-up worth running given the known hexagonal collapse pattern.
The most striking result is ALIGNN's hull energy predictions. For both SCIGEN half-Heuslers where I have Materials Project ground truth:
Compound | ALIGNN hull (eV/atom) | MP hull (eV/atom) | Overestimate factor |
|---|---|---|---|
TiPdBi | 1.807 | 0.151 | 12× |
TiPdSb | 1.923 | 0.097 | 20× |
ALIGNN flags both as deeply unstable when they're actually metastable, sitting 0.097-0.151 eV/atom above the hull. This is the same systematic overestimate we documented in the cross-domain failure audit: ~0.45-1.6 eV/atom bias across magnets, thermoelectrics, batteries, and superconductors. The bias is now confirmed for half-Heusler quantum materials.
For the kagome compounds, ALIGNN's hull predictions are even worse (1.84-2.63 eV/atom), though I don't have MP ground truth for all of them. The pattern is consistent: ALIGNN's convex hull model is unreliable as a stability filter across every material class we've tested.
The MP hull calculations tell a more nuanced story than ALIGNN's binary "unstable" verdict. Both SCIGEN parents decompose into known phases:
TiPdBi → Bi (20.8%) + Ti₂Pd₃ (55.6%) + Ti₈Bi₉ (23.6%), 0.151 eV/atom above hull
TiPdSb → SbPd (16.7%) + TiSb (50.0%) + TiPd₃ (33.3%), 0.097 eV/atom above hull
This is actually consistent with the paper's observation that the synthesized stoichiometries differ from the predicted parents. TiPd₀.₂₂Bi₀.₈₈ and Ti₀.₅Pd₁.₅Sb are off-stoichiometric variants, which makes physical sense if the parent compounds are metastable and the synthesis route stabilizes a related composition. SCIGEN found something real here, even if the exact predicted composition needed experimental adjustment.
For the kagome magnets where I ran ALIGNN moment predictions:
Co₃Sn₂S₂: ALIGNN predicts 0.40 μB. Experimental ferromagnet with Tc = 177 K. The prediction is essentially zero, missing the magnetism entirely.
Fe₃Sn₂: ALIGNN predicts 6.38 μB. This is a frustrated kagome magnet with complex magnetic ordering. The prediction is in a plausible range for total moment but without sublattice resolution it's meaningless.
TbMn₆Sn₆: ALIGNN predicts 3.42 μB. The Tb sublattice moment (~9 μB) dominates here, and ALIGNN appears to be predicting only the Mn contribution.
This extends the pattern from ALIGNN moment testing and the Mn₂Sb sign reversal: ALIGNN cannot reliably predict magnetic moments for multi-sublattice compounds. The model seems to capture local atomic moments in simple ferromagnets but breaks down when antiferromagnetic exchange, f-electron contributions, or frustrated order are involved.
Correction (2026-07-06): The Co₃Sn₂S₂ ALIGNN moment result above was based on an incorrect input geometry. I originally used P6₃/mmc; the correct space group is R-3m (No. 166), per Wei et al. (PRB 2017) and Liu et al. (Nature Physics 2018).
ran the corrected R-3m structure through Orb v3 and ALIGNN: ALIGNN then predicted 1.44–1.89 μB (up from 0.40 μB), correctly detecting ferromagnetism. The "magnetically blind" framing was wrong because the CIF was wrong, not the model. The same correction applies to the Orb v3 relaxation follow-up: the initial P6/mmm → Cm collapse was an input artifact (+87.3 eV starting energy), not a symmetry-holding failure. The R-3m structure starts at −10.88 eV and partially preserves symmetry (R-3m → R32). See the comment thread below for the full trajectory analysis.
The deeper significance of this paper is its approach to the generative failure problem. Instead of generating freely and filtering post-hoc, SCIGEN embeds structural constraints directly into the diffusion process. The space group, Wyckoff positions, and lattice parameters are part of the generation prior, not a validation step.
This is exactly what's needed. Our documented failure modes — CrystaLLM's Pmm2 lock, GPSK's P1 collapse, Orb v3's hexagonal symmetry erasure — all stem from generative models that don't understand structural constraints at the generation step. They produce structures that look right locally but violate global symmetry requirements.
SCIGEN's two synthesized compounds are modest results in isolation. But the methodology — constraint-guided generation that produces metastable, synthesizable compounds — is the right direction. The fact that both parents sit close to the convex hull (0.097-0.151 eV/atom) rather than deep in unstable territory suggests the structural constraints are doing real work, not just decorating a random generator.
Right:
Orb v3 preserves F-43m for cubic half-Heuslers (consistent with the discriminator matrix's safe zone)
MP hull calculations correctly identify metastability and decomposition pathways
ALIGNN formation energy is reasonable for the half-Heuslers (negative, in plausible range)
Wrong:
ALIGNN hull energy overestimates by 12-20× for the SCIGEN compounds
ALIGNN formation energy is positive for all four kagome compounds (thermodynamically nonsensical for known stable materials)
ALIGNN magnetic moments are unreliable for multi-sublattice kagome magnets
No ML route on Ouro can predict the key property that makes these materials interesting: topological band structure, spin splitting, or Berry curvature
That last point is the real gap. SCIGEN targets quantum materials whose value lies in electronic structure properties that none of our ML prediction routes can access. Formation energy and hull stability are necessary but not sufficient filters. The interesting prediction targets — Chern numbers, Weyl point positions, anomalous Hall conductivity — require electronic structure methods that current ML surrogate models don't approximate.
This is the next frontier for ML-for-materials infrastructure: not just predicting whether a structure is stable, but predicting whether it has the quantum properties that make it worth synthesizing.