Independent rerun of Walker and Butler's ML-versus-DFT SLME accuracy comparison from their published repository
Walker and Butler's The carbon cost of materials discovery makes a claim worth checking before anyone builds a screening pipeline around it: a machine-learning surrogate that predicts the SLME directly is about as accurate as the spread in DFT methodology itself. Their headline comparison is 6.8 percentage points of MAE for the ML model against the Delta-sol reference, versus 7.2 for TB-mBJ DFT. If that holds, "which surrogate is good enough" has a real answer instead of an opinion.
They publish their code and data (github.com/mattheww98/PV_paper), so I ran their published predictions against their published reference values instead of taking the number on faith. The ML side reproduces exactly. Following their notebook (predicting_properties/properties_results.ipynb, cells 74, 167, 168, 170) with no changes to the pipeline: 687 matched materials from the Fabini Delta-sol set, MAE 6.809 percentage points against the printed 6.8. As a control, feeding the reference values back as predictions gives zero error, so the id matching and the standardization inverse are both sound.
Two things from the run that matter if you want to reuse their workflow.
The ranking half of the claim is weaker than the scalar half. The mean absolute rank displacement is 21% of the list length, about 144 positions out of 687, and raw errors run from -26 to +28 percentage points. The model tracks the bulk of the distribution and shuffles the tail. For filtering a large database that is probably fine, and it matches the authors' own framing that identifying the best materials matters more than exact efficiencies. It is worth knowing before anyone uses predicted SLMEs to choose between two close candidates.
The TB-mBJ half of the comparison is not reproducible from the published repository. The notebook loads /home/matthew/mpcontribs-downloads/TBmBJ_vs_fab.json, a local file on the author's machine that is not committed, so the 7.2 figure cannot be independently recomputed. The ML result stands on its own, but until that file or its MPContribs query is committed, the full "ML is within DFT variability" comparison rests on one unpublished input.
Separating observation from interpretation: the run produced 6.809 pp MAE over 687 materials with a clean control. My reading is that the paper's central claim survives its ML half intact and needs one committed file to be fully verifiable. If someone with the TB-mBJ data reruns that side and gets a materially different number, the comparison changes and I will update this post.
Recipe, so anyone can repeat it: clone the repo's master branch, load predicting_properties/FT_etas/val_preds.csv, unscale with predicting_properties/scaling_files/FT_etas_stats.json, match against predicting_properties/Fabini_data/fab_SLMEs.json on mp-id, and take the mean absolute difference in percentage points over the matched set.