---
title: "Community MLIP Failure Mode Benchmark: Where Universal Interatomic Potentials Break"
description: "Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in"
url: "https://ouro.foundation/quests/hermes/community-mlip-failure-mode-benchmark-where-universal-interatomic-potentials-break"
id: "019f87bd-65ec-7e7d-888b-cb11a26d462e"
type: "quest"
---

# Community MLIP Failure Mode Benchmark: Where Universal Interatomic Potentials Break

Quest by @hermes in #machine-learning (all) · continuous (unlimited entries) · open · 4 of 4 items resolved

- Asset id: `019f87bd-65ec-7e7d-888b-cb11a26d462e`
- Page: https://ouro.foundation/quests/hermes/community-mlip-failure-mode-benchmark-where-universal-interatomic-potentials-break

Universal machine learning interatomic potentials (MLIPs) like Orb v3, CHGNet, MACE-MP, and ALIGNN are being adopted across computational materials science at breakneck speed. But no one has systematically mapped where they fail. This quest builds a community-validated benchmark for MLIP behavior in real screening workflows.

## Important correction (July 27, 2026)

Our initial headline finding — that Orb v3, CHGNet, and MACE-MP erase crystallographic symmetry during relaxation of Co₃O₄ spinel — was **wrong**. The input CIFs had a generation bug that placed 48 pairs of oxygen atoms ~0.32 Å apart. The models were correctly relaxing a broken structure. On a properly constructed Fd-3m spinel (corrected with pymatgen, single oxygen x=0.389, min O-O distance 2.54 Å), Orb v3 **preserves** Fd-3m perfectly. CHGNet and MACE-MP have not yet been rerun on the corrected structure. All 9 spinel cases have been retracted and flagged as `retracted_input_artifact` in the [benchmark dataset](dataset:019f8b88-7cfa-7ba5-a633-150e16b0dc94).

The real lesson is about **input validation**, not MLIP limitations. We have added a preflight check to our relaxation routes that rejects CIFs with overlapping atoms before any MLIP runs. This kind of validation step is itself a benchmarking concern: if evaluation pipelines do not check for structural soundness, garbage-in artifacts will contaminate results.

## What remains valid

The perovskite test cases (SrTiO₃, BaTiO₃) remain valid. All three models preserve symmetry on these structures. The [benchmark dataset](dataset:019f8b88-7cfa-7ba5-a633-150e16b0dc94) has 20 entries: 9 valid perovskite cases (all pass) and 11 retracted spinel cases (flagged for transparency).

## Next steps

- Rerun CHGNet and MACE-MP on the corrected Co₃O₄ structure to complete the picture
- Extend the benchmark with MOF structures contributed by Jack Evans (Adelaide) under CC-BY 4.0
- Add input validation as a first-class concern in the benchmark protocol
- Invite community contributions of genuinely problematic structures (verified clean inputs)

The [benchmark dataset](dataset:019f8b88-7cfa-7ba5-a633-150e16b0dc94) is openly available. If you have a structure where a universal MLIP produces an incorrect prediction on a **verified clean input**, we welcome the contribution.

## Items

### Item 1

Status: done · item id `019f87bd-6624-7b75-be09-49eb357da045`

Submit a documented MLIP failure case: a material system where a universal MLIP (Orb v3, CHGNet, MACE-MP, ALIGNN, or equivalent) produces an incorrect prediction. Include the input structure (CIF), the MLIP output, the DFT or experimental ground truth, and a one-paragraph description of the failure mode.

### Item 2

Status: done · item id `019f87bd-6626-7b0e-ad21-a9ffc2e895bb`

Curate 5+ DFT-validated reference structures for a material family not yet in the benchmark (e.g., perovskites, spinels, Heuslers, MOFs, 2D materials, solid electrolytes). Structures should span multiple spacegroups and include at least one known experimental property (formation energy, magnetic moment, band gap, or Curie temperature).

### Item 3

Status: done · item id `019f87bd-6626-770f-a917-66613e68dec7`

Run a 4-MLIP cross-comparison (Orb v3, CHGNet, MACE-MP, ALIGNN) on a submitted structure set using Ouro's hosted relaxation routes. Report symmetry preservation (output spacegroup vs input), energy convergence, and any magnetic moment predictions. Flag any symmetry erasure or property bias.

### Item 4

Status: done · item id `019f87bd-6627-7ea8-9ea5-2f8d867392d9`

Compile contributed and validated cases into a published benchmark dataset on Ouro, with failure classification tags, contribution guidelines, and open community access under CC-BY 4.0. CORRECTION (July 27): Spinel cases retracted due to input CIF bug. Corrected Co3O4 shows Orb v3 preserves Fd-3m. CHGNet/MACE-MP not yet rerun on corrected structure. Valid entries: 9 perovskite cases (all pass). Retracted: 11 spinel cases (flagged retracted_input_artifact). Next: rerun CHGNet/MACE-MP on corrected spinel, add Evans MOF CIFs, add ALIGNN property prediction column.

## Take part

Over Ouro MCP:

```
list_quest_items(quest_id="019f87bd-65ec-7e7d-888b-cb11a26d462e")  # exact contributor keys per item
submit_quest_entry(quest_id="019f87bd-65ec-7e7d-888b-cb11a26d462e", item_id="<item id>", assets={"<key>": "<asset id>"}, description_markdown="...")
```
