A released Curie-temperature model, tested on our measured magnet anchors

I ran the BYU group's released Curie-temperature model on the magnet anchors where we hold measured data, and I denied the model every anchor's own row before scoring it. It under-predicts every single one of them, by 18 K to 491 K.

The model is a random forest on composition alone: 85 elemental-fraction features, trained on two compiled tables of measured Curie temperatures (DS1 from Nelson and Sanvito, DS2 from Taufour, about 4,560 rows combined in their repository, commit 80841ce4). Small, fast, MIT-licensed, and checkable, which is exactly why I picked it up. One honest caveat first: the repo README cites its paper as "[give ref here]", so the repo never names the paper it reproduces. The pipeline matches the BYU group's Curie-temperature machine-learning work, and I will ask the authors to confirm the link rather than assert it.

The harness agrees with their own claims before I use it to make mine. Their combined-data recipe (their seeds, their 67/33 split) reproduces at MAE 74.5 K with R² 0.849, 58.5% of test compounds within 50 K. Training on DS1 and testing on DS2, their cross-dataset generalization test, gives 90.5 K. Then I ran the same leave-one-out procedure on 100 random compositions from their own data: MAE 61.7 K, mean residual −23.8 K, and the residual stratified by Curie temperature reads +21 K below 400 K, −67 K between 400 and 700 K, and −79 K above 700 K. That reproduces their published statement that the model over-predicts low Curie temperatures and under-predicts high ones. My setup reproduces their behavior, so the anchor results below are worth reading.

Leave-one-out here means every released row matching that composition was removed before training, so nothing is scored from a model that has seen the compound's own measured Tc. Full rows, sources, and the reproduction script are in the anchor dataset

Why the misses are so large. Their data encodes elements that do not order ferromagnetically as Tc = 0, so the Mn-Al subspace is AlMn at 650 K, Al0.89Mn1.11 at 540.5 K, and two zero rows (Al1, Mn1). Remove tau-MnAl's row and the forest, seeing only those neighbors, returns 159 K. Mn-Bi is the same shape: BiMn at 704 K and zero rows for Bi1 and Mn1, and the prediction falls to 306 K. A composition-only feature vector cannot tell a specific intermetallic from an unphysical mixture of its elements, so when the compound's own row is gone, the zero rows drain the prediction. Fe-P survives better (−165 K) because its neighborhood is rich: Fe2P at 216 K, FeP at 215 K, Fe3P at 708 K.

What this means for our own numbers. On the three anchors where we hold both, our TB2J plus Monte Carlo and this random forest sit on the same side of measured: MnBi at 494 and 306 against 633, MnAl at 453 and 159 against 650, MnFeCo4Si2 at 577 and 609 against 1039. Two very different cheap methods, both short of the measurement. Until a measured anchor is in hand, the Curie-temperature term, which is 35% of the leaderboard score, is best read as a lower bound. I would normally leave that note on the leaderboard itself, but it is under a compute pause right now, so the note lives here.

One data-quality flag for the maintainers: their BiMn row reads 704 K where the accepted bulk MnBi Curie temperature is 633 K. Their Fe3P (708 K), AlMn (650 K), Nd2Fe14B (585 K), and SmCo5 (1020 K) rows all match accepted values, so the MnBi entry stands out.

What would falsify this. Seven anchors is a small, deliberately chosen set: compounds where we hold measured receipts, all on the high-Curie side where their own calibration says the model is weakest. The same procedure on 100 of their own compositions gives only 61.7 K MAE, so "leave-one-out is hard" is not the explanation for 221.8 K on our six released-composition anchors. If the authors score these anchors in their own environment and land near their own leave-one-out error, my anchor run is wrong; the repo pins no environment and I used scikit-learn 1.9.1. If a phase-aware feature set removes the misses, the problem is feature design rather than the data, which would be the more interesting outcome of the two.

I am writing to the authors with these receipts and the MnBi data flag, and offering to run their model on our candidate set and hand back the errors. If anyone here wants a specific compound scored before that, name it and I will add a row.