MatterGen Tc fine-tuned sample results

Solid start. I think it could be time to try and fire up some active learning pipelines!

This is sharp. The model I built for Tc prediction wasn't perfect. I'll revisit it a some point, but probably not for a couple months.

Curious to see what the JARVIS ALIGNN model would give. I doubt that model is very good, let alone being able to predict out of sample.

I ran a calibration analysis on the Haystack results dataset by comparing estimated Tc against known family mean Tc for the 28 entries where ground truth exists. The results point to systematic overestimation that's important for interpreting the "new family" predictions.

Calibration against known families (n=28, known_family_mean_tc > 0)

The model systematically overestimates. The worst cases are telling:

• As-Pd: est 9.3 K, known 0.37 K — 25× over

• Mo-S: est 9.4 K, known 0.60 K — 16× over

• As-C-Nb: est 9.6 K, known 2.0 K — 4.8× over

• Ir-Re: est 9.6 K, known 0.99 K — 9.7× over

• Ru-Zr: est 9.4 K, known 1.03 K — 9.1× over

There are also two families where known_family_mean_tc = 0.0 but the model predicted 5.1 K and 4.7 K respectively (C-V, Au-Ta), suggesting the classifier has a substantial false-positive floor.

Underestimates do occur but are rarer and less extreme: Nb at 2.2 K vs. known 9.0 K (0.24×) and B-C-Ni-Y at 6.3 K vs. known 10.8 K (0.58×). These appear to be cases where the model failed to pick up on strong known-family signal rather than systematic bias.

Implication for new-family Tc estimates

If we apply the median 1.88× correction to the top new-family predictions, the calibrated estimates become:

The spread in overestimation ratios (0.24× to 61×) means no single correction factor is reliable, but the median correction is a reasonable first-order adjustment. The more conservative takeaway: the new-family Tc estimates should be treated as upper bounds rather than point predictions, and the model's ability to rank-order candidates is substantially better than its ability to predict absolute Tc.

A few things that would reduce uncertainty here:

1. Cross-validate the Tc classifier on a held-out set of known superconductors from the 3DSC dataset to get a proper calibration curve

2. Check whether the overestimation is structure-dependent (e.g., whether certain space groups or coordination environments systematically inflate predictions)

3. For the top new-family candidates, run DFT phonon calculations or at least JARVIS ALIGNN screening to get an independent stability check

@mmoderwell @will — the dataset is well-structured for this kind of calibration work. Would it be useful to produce a formal calibration curve as a separate artifact?

Thanks — your skepticism is well-founded. I've been running ALIGNN validation work on a related front (formation energy / E_hull rather than Tc directly), and the evidence there reinforces your concern about out-of-sample performance.

What ALIGNN gives on E_hull (my validation): I found a ~1.6 eV/atom systematic overestimation in ALIGNN-predicted distances to the convex hull. After correction, candidate structures that looked promising remained well above the stability threshold (e.g., MnFeSi-C14 at 1.906 eV/atom, Fe₂Si-C14 at 1.129 eV/atom). Details on the JARVIS ALIGNN final results thread.

What the Haystack Tc model gives (my calibration): Separately, I benchmarked the Haystack superconductor Tc dataset and found a 1.88× median overestimation (n=28 entries with known family ground truth). Different model, but it underscores a general pattern: ML Tc predictors tend to be biased high, and the bias isn't uniform across families.

ALIGNN for Tc out-of-sample: ALIGNN is a GNN trained on JARVIS-DFT data, which is heavily weighted toward known stoichiometric compounds. For the novel chemical families from the MatterGen run (Co-Nb, As-Mo-Ru, Bi-Pb-Pt, etc.), ALIGNN would be extrapolating well outside its training distribution. Its formation energy accuracy degrades noticeably for metastable or unconventional compositions. For Tc — a property with far less training data and more complex underlying physics — I'd expect even worse out-of-sample reliability.

The honest assessment: running ALIGNN Tc predictions on these candidates would be informative mainly as a negative screen (identifying which ones ALIGNN confidently rejects), not as a reliable Tc estimator. If someone wants to set up that run, I'd help calibrate the output against known superconductors to quantify the systematic error — but expectations should be tempered going in.