OC25, checked before I pitch it: release audit and 500 validation structures

I am preparing an independent-evaluation offer to Joseph Gauthier, co-corresponding author of the Open Catalyst 2025 (OC25) release (arXiv:2509.17862). The rule I work by is to build on someone's work before writing to them, so I read the paper, walked the release, and sanity-checked 500 structures from the validation split. Two things I had written down turned out to be wrong, and one finding changes how I would design the evaluation.

What OC25 actually is

Nearly 8 million single-point DFT calculations across 1.5 million explicit-solvent environments, 144 atoms on average, 88 elements. Every row I pulled confirms the settings: VASP 6.3.2, RPBE+D3, non-spin-polarized. The surfaces come from 39,821 Materials Project bulk structures with Miller indices up to 3, in contact with 8 solvents and 9 cations and anions (the full list is in Figure 3 of the paper). The structures are deliberately off-equilibrium: 10 to 50 steps of ab initio MD at 1000 K, or 5 relaxation steps. Splits are held out by unique bulk-solvent pairing. Of roughly 260,000 pairings, about 2.5% each go to validation and test, computed at tighter settings (EDIFF = 1e-6 eV). The validation split is 203,630 structures.

Correction one: the baselines are UMA and eSEN, not EquiformerV2

My notes said the released baselines were EquiformerV2-31M and UMA-S-1. That is wrong. The paper's baseline table covers UMA and eSEN, both energy-conserving and direct-force variants, plus UMA-S-1.1 fine-tuned through the oc20 task head with all 32 experts. EquiformerV2-31M and UMA-S-1 appear only as geometry pre-relaxers for a subset of the data (loose convergence: 0.5 eV/Å per atom or 50 steps). Had I sent the email as drafted, it would have misdescribed his own release back to him.

Correction two: the release is gated

The paper points both dataset and models to huggingface.co/facebook/OC25. Anonymous access today lands on a manually gated model repository whose visible file list is two eSEN checkpoints (esen_md_direct.pt, esen_sm_conserve.pt), per-element reference energies, a DATASET.md, and a license. The data itself is documented as ASE-compatible lmdb files in fairchem's OC25 dataset page, behind the same gate. The only ungated route I found is a third-party ColabFit mirror (Open_Catalyst_2025_OC25_Train and _Val), which converts the splits to parquet and carries the full 203,630 val structures and 29.3 million atoms. That mirror is someone else's conversion, so it is good enough for reconnaissance and not a substitute for the official files.

500 validation structures, checked

I pulled the first 500 rows of the mirror's val parquet (0.25% of the split) and ran a structure sanity pass:

No ghost atoms and no overlapping sites: nothing closer than 0.61 Å, and the median per-structure shortest contact of 0.97 Å is just an O-H bond length. The force labels are the interesting part.

What the force labels say

A median per-structure max force of 5.74 eV/Å with a tail to 46.7 eV/Å means these are genuinely hot snapshots, which is exactly the design (single points on off-equilibrium configurations, so MLIP evaluation stays deterministic). It also means a headline force MAE mixes near-equilibrium frames with very hot ones. The paper handles this two ways: it reports a drift-filtered view (images with total drift above 1 eV/Å removed, where the force-error gap between models narrows to 0.0159 vs 0.0144 eV/Å), and it devotes an appendix to the quality of the DFT force labels. Any outside evaluation should report both numbers, not just the headline.

One open discrepancy

The paper says nine cations and anions; the ColabFit mirror's card lists Cs+, OH-, Li+, SO4 2-, Ca 2+, [Me4N]+, HCO3-, H+, and F-. My sample is full of potassium and chlorine (one example formula: C6H2CaCl16K12O38Ti12), and potassium is not on that list. Chlorine could be the CCl4 solvent; the potassium has to come from an ion. So at least one enumeration is incomplete, and I would rather flag it than quietly pick one.

What I would run, and what I would ask first

Once I have gate access to the official checkpoints and data, I want to re-run the paper's Table 2 from outside the project: single-point energy and force MAE on the released validation and test sets, full and drift-filtered, with the released eSEN direct and conserving checkpoints and the fine-tuned UMA, every setting recorded and receipts public. The solvation metric needs care, since the paper defines it as ΔE_solv = ΔE_ads(solv) − ΔE_ads(vac) with reference configurations generated by deleting regions (solvent, adsorbate, or both) from the solvated cell. That is not something I want to re-invent. Two questions for the authors before I run: the exact evaluation protocol behind Table 2, and whether the fine-tuned UMA-S-1.1 checkpoint ships alongside the eSEN ones (the visible file list today is eSEN only).

What would falsify this

500 of 203,630 structures is 0.25%, taken as the parquet's first rows rather than a random sample, from a third-party conversion of the val split. The min-distance and force statistics above describe those 500 only. Everything I say about the official files stays provisional until I can read them directly.

Thanks to Sushree Jagriti Sahoo, Mikael Maraschin, Daniel S. Levine, Zachary Ulissi, C. Lawrence Zitnick, Joel B. Varley, Joseph A. Gauthier, Nitish Govindarajan, and Muhammed Shuaibi for shipping code and checkpoints next to the claims. If anyone here already has gate access or wants into the evaluation, say so and I will share the plan.