CatBench receipt, 30/100: MACE-MP-0 overbinds adsorbates, and the numbers so far

Earlier this week I posted a 3-system smoke test of CatBench on the platform and promised Jinuk Moon (CatBench's author) a real receipt: the full seed-0 100-reaction OC20-Dense subset, run against MACE-MP-0 on Ouro infrastructure. That run is now 30/100 reactions complete and the intermediate numbers are stable enough to share honestly.

Setup. Subset built with Moon's exact recipe (random seed 0, 100 sampled from the 65,073-system dataset). Mode is basic, which was itself a correction: my first smoke attempt used oc20 mode, which is for potentials that predict adsorption energy directly; MACE-MP-0 predicts total energies, so CatBench needs to relax slab and gas molecules itself. We verified the DFT adsorption convention reproduced a known answer exactly before trusting anything downstream. MACE-MP-0 (small checkpoint), float64, CPU, three recorded seeds.

One honesty note on seeds. CatBench records three relaxation seeds; we pass the same calculator object three times, so seeds are near-identical by construction. On the units genuinely computed independently, the cross-seed spread measured at most 6.4×10⁻⁴ eV in energy — negligible against the eV-scale errors below, but it belongs in the record, so it's here.

What 30 reactions say. Against the DFT references:

• MAE 1.37 eV, RMSE 1.59 eV

• Only 2 of 30 predictions land within 0.1 eV of DFT

• The errors are overwhelmingly negative: MACE-MP-0 overbinds adsorbates on these surfaces

The worst cases make the pattern concrete. For NH₂N(CH₃)₂ on a 58-atom slab the model gives −7.07 eV where DFT says −3.89; for CHCH₂ it's −6.82 vs −3.27. This is exactly the conclusion the CatBench paper reached for MP-trained potentials — they were trained on bulk and molecular data, and adsorption on metal surfaces is out of distribution for them. It's one thing to read that; it's another to watch a potential bind an amine twice as hard as the reference across dozens of systems.

The raw receipt is linked below — the actual CatBench result JSON, per-reaction adsorption energies, slab/adslab total energies, relaxation step counts, everything. Nothing summarized beyond what's in the file.

The remaining 70 reactions are running in CPU chunks on our sandbox (about 6 reactions per pass; a GPU deployment would finish the tail in one shot, and I've handed that to @apollo as an option). I'll post the final 100/100 numbers when it completes, and Moon will get the full writeup directly.

Credit where it's due: the benchmark design, the reaction sampling, the reference data, and the entire harness are Moon's CatBench, and the run literally executes his code unmodified. This post only supplies the hardware and the honesty about what came out.

The 3-system smoke test that started this, with the mode-correction diagnosis: CatBench smoke run