Aggregate computed results into a dataset

A dataset on Ouro can be more than a table you upload at the end of a project. It can aggregate useful results from many files and computations into one queryable index.

We used this pattern to turn the CIFs visible across Ouro into a working list of rare-earth-free permanent-magnet candidates. The CIF files supplied the structures. Successful route actions supplied calculated properties. A reference-backed dataset joined them into one queryable index.

The important distinction

A CIF file contains a crystal structure. It does not change each time someone predicts a property from that structure.

The prediction belongs to the action that ran the model:

• The file answers: what structure did we evaluate?
• The route answers: what method or service did we use?
• The action answers: what happened in this particular run?
• The dataset answers: which results are useful to analyze together?

That separation matters when two routes predict the same property, a model is updated, or a result later proves unreliable. The original structure stays intact, and every value remains connected to the exact computation that produced it.

The aggregate dataset

For the permanent-magnet search, we wanted to answer questions such as:

• Which CIFs have already been evaluated?
• Which properties are available for each structure?
• Which route and model produced each value?
• Which structures have promising magnetic properties without rare-earth elements?
• Which candidates still need relaxation, stability, or anisotropy calculations?

The collection script stores one scalar property per row:

This long-form shape is deliberate. Property routes do not all return the same fields: one may return a named prediction, another formation energy and energy above hull, and another relaxation energies and step counts. One row per property avoids a wide, mostly empty table and lets new result types appear without a schema migration.

Collect files, then inspect their actions

Start with the files visible to your account. scope="all" means all files you have permission to see, including files outside your own profile; it does not bypass access controls.

from ouro import Ouro
 
ouro = Ouro()
cifs = ouro.files.search(extension="cif", scope="all", limit=None)

For each CIF, retrieve successful actions that used it as an input. include_response=True is required because that response contains the calculated values.

for cif in cifs:
    connections = ouro.assets.actions(
        str(cif.id),
        role="input",
        status="success",
        include_response=True,
    )
 
    for action in connections["as_input"]:
        route_name = (action.route or {}).get("name")
        properties = action.response or {}

Use role="input" for property calculations because the CIF was consumed by the route. If you need to find the action that created a relaxed or generated CIF, inspect its output connection instead.

For a platform-wide collection, do not run this loop manually in a notebook and hope it finishes. Use bounded concurrency and a local checkpoint so an interrupted job can resume. The complete build_cif_properties_dataset.py example in ouro-py handles those details and normalizes common response shapes.

Test the collection without creating an Ouro asset:

python examples/build_cif_properties_dataset.py --dry-run

Then create the dataset in an organization and team:

python examples/build_cif_properties_dataset.py \
  --org-id "$OURO_ORG_ID" \
  --team-id "$OURO_TEAM_ID"

Preserve provenance with references

The three ID columns should not be anonymous UUID strings. Declare their meaning when you create the dataset:

dataset = ouro.datasets.create(
    name="CIF calculated properties",
    description=(
        "Calculated properties extracted from successful route actions "
        "that used each CIF as an input."
    ),
    visibility="public",
    org_id=org_id,
    team_id=team_id,
    data=rows,
    refs={
        "file_id": "file",
        "action_id": "action",
        "route_id": "route",
    },
)

References make the table a provenance graph rather than a detached export. Ouro can validate that referenced objects exist, expose the columns as semantic references, and resolve IDs back to asset or action details.

After creation, verify the inferred schema:

for column in ouro.datasets.schema(str(dataset.id)):
    print(
        column["column_name"],
        column.get("semantic_type"),
        column.get("ref_kind"),
        column.get("asset_type"),
    )

When an application needs names and URLs alongside raw IDs, request resolved references on a paginated read:

page = ouro.datasets.query(
    str(dataset.id),
    limit=100,
    offset=0,
    resolve_refs=True,
)
 
rows = page["data"]
references = page["resolved_refs"]

From property history to a candidate list

The collected dataset is not yet a claim that every row is a good magnet. It is an evidence layer. The candidate-list step reshapes that evidence around each CIF, excludes rare-earth compositions, applies selection thresholds, and records which calculations are still missing.

For exploratory analysis, pivot numeric properties into one row per structure:

property_rows = ouro.datasets.query(dataset_id)
 
numeric = property_rows[property_rows["value_numeric"].notna()]
candidates = numeric.pivot_table(
    index=["file_id", "file_name"],
    columns="property_name",
    values="value_numeric",
    aggfunc="last",
).reset_index()

Before ranking, decide how to handle repeated calculations. aggfunc="last" is convenient for exploration, but production aggregation logic should choose results by an explicit rule such as approved route, model version, action timestamp, or lowest-energy relaxation. Keeping action_id and route_id in the source table makes that choice auditable.

The rare-earth-free permanent-magnet workflow then becomes:

1. Collect every accessible CIF.
2. Recover calculated properties from successful actions.
3. Exclude compositions containing rare-earth elements.
4. Rank structures with the magnetic, stability, cost, and anisotropy evidence available.
5. Mark missing evidence as the next work queue.
6. Append new action-backed results as routes finish.

The aggregate holds both the current candidate list and the gaps that determine the next action. A later screening run does not need to reconstruct context from posts, filenames, or agent memory; it can query the dataset, follow the references, and continue from the recorded results.

Design rules

• Keep source artifacts as files and computed values on actions.
• Store references to files, actions, and routes instead of copying provenance into text fields.
• Prefer long-form rows when routes return heterogeneous properties.
• Record failures and missing stages as state, not only successful final candidates.
• Keep the action-level evidence even when you publish a separate ranked or wide-form view.
• Treat the dataset as a queryable aggregate, not as a replacement for the underlying files and actions.

The same model works beyond materials science. A dataset can aggregate document extractions, image labels, model evaluations, due-diligence findings, or any other useful results produced by repeatable actions.