20 awaiting review0 of 3 resolvedOpen
How many stable, unique, novel crystals does a generative model find per GPU-second?
GenBench runs each crystal generator hosted on Ouro on the same GPU (an A10G) for a fixed budget of GPU time. Every sample it returns is:
relaxed with one fixed ORB v3 recipe (that time counts against the model);
checked for uniqueness;
matched against Materials Project and Alexandria for novelty;
placed on the ORB v3 convex hull of published phases.
Ouro records when each call started and finished, so the cost isn't self-reported: a model can be slower than it says, never faster.
There are three tracks, one item each, each with its own leaderboard:
A. Unconditional: 30 GPU-minutes. Score: SUN (stable, unique, novel) structures per GPU-hour.
B. Chemical systems: 10 systems (Li-Mn-O, Al-Fe-Si, B-Co-Fe, Bi-Mn, Bi-Se-Te, Mg-Si-Sn, Al-N-Ti, Cu-S-Sn-Zn, Ba-O-Ti, Ni-Sn-Zr), 6 GPU-minutes each. Score: SUN per GPU-hour over all ten.
C. Known ground states: GaN, TiO2, SrTiO3, LiCoO2, Al2O3, MgAl2O4, Fe2B, FeSn2, 5 GPU-minutes each. Score: median GPU-seconds until a sample relaxes to within 5 meV/atom of the best published polymorph (lower is better).
Enter a model. Your model needs a route that answers GenBench's sample contract. Enter that route in each track it can do. The quest's owner starts each evaluation and pays for the runs. The contract, a route checker, the scoring code and a template app are at https://github.com/ourofoundation/genbench. Season 0 scores models hosted on Ouro, so everyone runs on the same GPU; to have yours hosted, send it at https://ouro.foundation/host.
Season 0 reference: the shared structure database as of 2026-10-16. Stability is judged on ORB v3, not DFT.