---
title: "Aggregate computed results into a dataset"
description: "Collect useful results from route actions into a queryable dataset with typed references."
date: "2026-08-18"
last_updated: "2026-08-18"
---

A dataset on Ouro can be more than a table you upload at the end of a project. It can aggregate useful results from many files and computations into one queryable index.

We used this pattern to turn the CIFs visible across Ouro into a working list of rare-earth-free permanent-magnet candidates. The CIF files supplied the structures. Successful route actions supplied calculated properties. A reference-backed dataset joined them into one queryable index.

## The important distinction

A CIF file contains a crystal structure. It does not change each time someone predicts a property from that structure.

The prediction belongs to the **action** that ran the model:

- The **file** answers: what structure did we evaluate?
- The **route** answers: what method or service did we use?
- The **action** answers: what happened in this particular run?
- The **dataset** answers: which results are useful to analyze together?

<Callout type="info">
  Computed properties live in successful action responses, not in file metadata.
  Updating a file with every new prediction would erase the distinction between
  the source artifact and the computations performed on it.
</Callout>

That separation matters when two routes predict the same property, a model is updated, or a result later proves unreliable. The original structure stays intact, and every value remains connected to the exact computation that produced it.

## The aggregate dataset

For the permanent-magnet search, we wanted to answer questions such as:

- Which CIFs have already been evaluated?
- Which properties are available for each structure?
- Which route and model produced each value?
- Which structures have promising magnetic properties without rare-earth elements?
- Which candidates still need relaxation, stability, or anisotropy calculations?

The collection script stores one scalar property per row:

| Column          | Meaning                                                  |
| --------------- | -------------------------------------------------------- |
| `file_id`       | Reference to the CIF that was evaluated                  |
| `action_id`     | Reference to the route execution that produced the value |
| `route_id`      | Reference to the route used for the calculation          |
| `route_name`    | Human-readable route name                                |
| `model`         | Model reported by the action response                    |
| `property_name` | Name of the calculated field                             |
| `value_numeric` | Numeric value, when applicable                           |
| `value_text`    | Text value, when applicable                              |
| `unit`          | Unit reported by the route                               |

This long-form shape is deliberate. Property routes do not all return the same fields: one may return a named prediction, another formation energy and energy above hull, and another relaxation energies and step counts. One row per property avoids a wide, mostly empty table and lets new result types appear without a schema migration.

## Collect files, then inspect their actions

Start with the files visible to your account. `scope="all"` means all files you have permission to see, including files outside your own profile; it does not bypass access controls.

```python showLineNumbers
from ouro import Ouro

ouro = Ouro()
cifs = ouro.files.search(extension="cif", scope="all", limit=None)
```

For each CIF, retrieve successful actions that used it as an input. `include_response=True` is required because that response contains the calculated values.

```python showLineNumbers
for cif in cifs:
    connections = ouro.assets.actions(
        str(cif.id),
        role="input",
        status="success",
        include_response=True,
    )

    for action in connections["as_input"]:
        route_name = (action.route or {}).get("name")
        properties = action.response or {}
```

Use `role="input"` for property calculations because the CIF was consumed by the route. If you need to find the action that created a relaxed or generated CIF, inspect its output connection instead.

For a platform-wide collection, do not run this loop manually in a notebook and hope it finishes. Use bounded concurrency and a local checkpoint so an interrupted job can resume. The complete [`build_cif_properties_dataset.py`](https://github.com/ourofoundation/ouro-py/blob/main/examples/build_cif_properties_dataset.py) example in `ouro-py` handles those details and normalizes common response shapes.

Test the collection without creating an Ouro asset:

```shell
python examples/build_cif_properties_dataset.py --dry-run
```

Then create the dataset in an organization and team:

```shell
python examples/build_cif_properties_dataset.py \
  --org-id "$OURO_ORG_ID" \
  --team-id "$OURO_TEAM_ID"
```

## Preserve provenance with references

The three ID columns should not be anonymous UUID strings. Declare their meaning when you create the dataset:

```python showLineNumbers
dataset = ouro.datasets.create(
    name="CIF calculated properties",
    description=(
        "Calculated properties extracted from successful route actions "
        "that used each CIF as an input."
    ),
    visibility="public",
    org_id=org_id,
    team_id=team_id,
    data=rows,
    refs={
        "file_id": "file",
        "action_id": "action",
        "route_id": "route",
    },
)
```

References make the table a provenance graph rather than a detached export. Ouro can validate that referenced objects exist, expose the columns as semantic references, and resolve IDs back to asset or action details.

After creation, verify the inferred schema:

```python showLineNumbers
for column in ouro.datasets.schema(str(dataset.id)):
    print(
        column["column_name"],
        column.get("semantic_type"),
        column.get("ref_kind"),
        column.get("asset_type"),
    )
```

When an application needs names and URLs alongside raw IDs, request resolved references on a paginated read:

```python showLineNumbers
page = ouro.datasets.query(
    str(dataset.id),
    limit=100,
    offset=0,
    resolve_refs=True,
)

rows = page["data"]
references = page["resolved_refs"]
```

## From property history to a candidate list

The collected dataset is not yet a claim that every row is a good magnet. It is an evidence layer. The candidate-list step reshapes that evidence around each CIF, excludes rare-earth compositions, applies selection thresholds, and records which calculations are still missing.

For exploratory analysis, pivot numeric properties into one row per structure:

```python showLineNumbers
property_rows = ouro.datasets.query(dataset_id)

numeric = property_rows[property_rows["value_numeric"].notna()]
candidates = numeric.pivot_table(
    index=["file_id", "file_name"],
    columns="property_name",
    values="value_numeric",
    aggfunc="last",
).reset_index()
```

Before ranking, decide how to handle repeated calculations. `aggfunc="last"` is convenient for exploration, but production aggregation logic should choose results by an explicit rule such as approved route, model version, action timestamp, or lowest-energy relaxation. Keeping `action_id` and `route_id` in the source table makes that choice auditable.

The rare-earth-free permanent-magnet workflow then becomes:

1. Collect every accessible CIF.
2. Recover calculated properties from successful actions.
3. Exclude compositions containing rare-earth elements.
4. Rank structures with the magnetic, stability, cost, and anisotropy evidence available.
5. Mark missing evidence as the next work queue.
6. Append new action-backed results as routes finish.

The aggregate holds both the current candidate list and the gaps that determine the next action. A later screening run does not need to reconstruct context from posts, filenames, or agent memory; it can query the dataset, follow the references, and continue from the recorded results.

## Design rules

- Keep source artifacts as files and computed values on actions.
- Store references to files, actions, and routes instead of copying provenance into text fields.
- Prefer long-form rows when routes return heterogeneous properties.
- Record failures and missing stages as state, not only successful final candidates.
- Keep the action-level evidence even when you publish a separate ranked or wide-form view.
- Treat the dataset as a queryable aggregate, not as a replacement for the underlying files and actions.

The same model works beyond materials science. A dataset can aggregate document extractions, image labels, model evaluations, due-diligence findings, or any other useful results produced by repeatable actions.