Output Format Reference

This page is the exhaustive schema reference for every file produced by cobre run. It documents column names, Arrow data types, nullability, JSON field structures, and binary format layouts for all 10 Parquet schemas, the two manifest types, the training metadata file, the five dictionary files, and the policy checkpoint format.

If you are new to Cobre output, start with Understanding Results first. That page explains what each file means conceptually and shows how to read results programmatically. This page is for readers who need the precise schema definition — for writing parsers, building dashboards, or implementing compatibility checks.

Output Directory Tree

A complete cobre run produces the following directory structure. Not every entity directory appears in every run: cobre run only writes directories for entity types present in the case. For example, a case with no pumping stations will not produce simulation/pumping_stations/.

<output_dir>/
  training/
    _manifest.json
    metadata.json
    convergence.parquet
    dictionaries/
      codes.json
      entities.csv
      variables.csv
      bounds.parquet
      state_dictionary.json
    timing/
      iterations.parquet
      mpi_ranks.parquet
  policy/
    cuts/
      stage_000.bin
      stage_001.bin
      ...
      stage_NNN.bin
    basis/
      stage_000.bin
      stage_001.bin
      ...
      stage_NNN.bin
    metadata.json
  simulation/
    _manifest.json
    costs/
      scenario_id=0000/
        data.parquet
      scenario_id=0001/
        data.parquet
      ...
    hydros/
      scenario_id=0000/data.parquet
      ...
    thermals/
      scenario_id=0000/data.parquet
      ...
    exchanges/
      scenario_id=0000/data.parquet
      ...
    buses/
      scenario_id=0000/data.parquet
      ...
    pumping_stations/
      scenario_id=0000/data.parquet
      ...
    contracts/
      scenario_id=0000/data.parquet
      ...
    non_controllables/
      scenario_id=0000/data.parquet
      ...
    inflow_lags/
      scenario_id=0000/data.parquet
      ...
    violations/
      generic/
        scenario_id=0000/data.parquet
        ...

Training Output

training/_manifest.json

The training manifest is written atomically at the end of the training run (and updated on each checkpoint if checkpointing is enabled). Consumers should read status before interpreting any other field.

JSON structure:

{
  "version": "2.0.0",
  "status": "complete",
  "started_at": "2026-01-17T08:00:00Z",
  "completed_at": "2026-01-17T12:30:00Z",
  "iterations": {
    "max_iterations": 200,
    "completed": 128,
    "converged_at": null
  },
  "convergence": {
    "achieved": false,
    "final_gap_percent": 0.45,
    "termination_reason": "iteration_limit"
  },
  "cuts": {
    "total_generated": 1250000,
    "total_active": 980000,
    "peak_active": 1100000
  },
  "checksum": null,
  "mpi_info": {
    "world_size": 1,
    "ranks_participated": 1
  }
}

Field reference:

training/metadata.json

The metadata file captures the configuration snapshot, problem dimensions, performance summary, data integrity hashes, and runtime environment for reproducibility and audit purposes. Fields marked “deferred” are null in the current release and will be populated in a future minor version.

Top-level structure:

{
  "version": "2.0.0",
  "run_info": { ... },
  "configuration_snapshot": { ... },
  "problem_dimensions": { ... },
  "performance_summary": null,
  "data_integrity": null,
  "environment": { ... }
}

run_info fields:

configuration_snapshot fields:

problem_dimensions fields:

performance_summary: Deferred. Always null in the current release. Will contain total_lp_solves, avg_lp_time_us, median_lp_time_us, p99_lp_time_us, and peak_memory_mb when implemented.

data_integrity: Deferred. Always null in the current release. Will contain SHA-256 hashes of input files, config, policy, and convergence data when implemented.

environment fields:

training/convergence.parquet

Per-iteration convergence log. One row per training iteration. 13 columns.

training/timing/iterations.parquet

Per-iteration wall-clock timing breakdown by phase. One row per training iteration. 10 columns. All columns are non-nullable.

training/timing/mpi_ranks.parquet

Per-iteration, per-rank timing statistics for distributed runs. One row per (iteration, rank) pair. 8 columns. All columns are non-nullable.

training/dictionaries/

Five self-documenting files that allow output Parquet files to be interpreted without reference to the original input case. All files are written atomically.

codes.json

Static mapping from integer codes to human-readable labels for all categorical fields used in Parquet output. The same mapping applies for the lifetime of a release (the version field tracks breaking changes).

{
  "version": "1.0",
  "generated_at": "2026-01-17T08:00:00Z",
  "operative_state": {
    "0": "deactivated",
    "1": "maintenance",
    "2": "operating",
    "3": "saturated"
  },
  "storage_binding": {
    "0": "none",
    "1": "below_minimum",
    "2": "above_maximum",
    "3": "both"
  },
  "contract_type": {
    "0": "import",
    "1": "export"
  },
  "entity_type": {
    "0": "hydro",
    "1": "thermal",
    "2": "bus",
    "3": "line",
    "4": "pumping_station",
    "5": "contract",
    "7": "non_controllable"
  },
  "bound_type": {
    "0": "storage_min",
    "1": "storage_max",
    "2": "turbined_min",
    "3": "turbined_max",
    "4": "outflow_min",
    "5": "outflow_max",
    "6": "generation_min",
    "7": "generation_max",
    "8": "flow_min",
    "9": "flow_max"
  }
}

entities.csv

One row per entity across all entity types. Columns:

Rows are ordered by entity_type_code ascending, then by entity_id ascending within each type.

variables.csv

One row per output column across all Parquet schemas. Documents every column name, its parent schema, and its unit of measure. Useful for building generic result readers that do not hard-code column names.

bounds.parquet

Per-entity, per-stage resolved LP variable bounds. Documents the actual numerical bounds used in each LP solve, after applying the three-tier penalty resolution (global / entity / stage overrides).

state_dictionary.json

Describes the state space structure used by the algorithm: which entities have state variables, how many state dimensions they contribute, and what units apply. Useful for interpreting cut coefficient vectors in the policy checkpoint.

{
  "version": "1.0",
  "state_dimension": 164,
  "storage_states": [
    { "hydro_id": 0, "dimension_index": 0, "unit": "hm3" },
    { "hydro_id": 1, "dimension_index": 1, "unit": "hm3" }
  ],
  "inflow_lag_states": [
    { "hydro_id": 0, "lag_index": 1, "dimension_index": 2, "unit": "m3s" }
  ]
}

Policy Checkpoint

policy/cuts/stage_NNN.bin

FlatBuffers binary file encoding all cuts for a single stage. One file per stage; file names are zero-padded to three digits (e.g. stage_000.bin, stage_012.bin).

The binary is not human-readable. The logical record structure for each cut contained in the file is:

The encoding uses the FlatBuffers runtime builder API (little-endian, no reflection, no generated code). Field order in the binary matches the declaration order above.

policy/basis/stage_NNN.bin

FlatBuffers binary file encoding the LP simplex basis checkpoint for a single stage. One file per stage. Used to warm-start LP solves when resuming a study.

The logical record structure is:

policy/metadata.json

Small JSON file describing the checkpoint at a high level. Human-readable and intended for compatibility checking on study resume.

Simulation Output

All simulation results use Hive partitioning: one data.parquet file per scenario stored in a scenario_id=NNNN/ subdirectory. See Hive Partitioning below for how to read these files.

simulation/costs/

Stage and block-level cost breakdown. One row per (stage, block) pair. 20 columns.

simulation/hydros/

Hydro plant dispatch results. One row per (stage, block, hydro) triplet. 28 columns.

simulation/thermals/

Thermal unit dispatch results. One row per (stage, block, thermal) triplet. 10 columns.

simulation/exchanges/

Transmission line flow results. One row per (stage, block, line) triplet. 11 columns.

simulation/buses/

Bus load balance results. One row per (stage, block, bus) triplet. 10 columns.

simulation/pumping_stations/

Pumping station results. One row per (stage, block, pumping station) triplet. 9 columns.

simulation/contracts/

Energy contract results. One row per (stage, block, contract) triplet. 8 columns.

simulation/non_controllables/

Non-controllable source results (wind, solar, run-of-river hydro without storage, etc.). One row per (stage, block, non-controllable) triplet. 10 columns.

simulation/inflow_lags/

Autoregressive inflow lag state variables. One row per (stage, hydro, lag) triplet. No block dimension — inflow lags are stage-level state variables. 4 columns. All columns are non-nullable.

simulation/violations/generic/

Generic user-defined constraint violations. One row per (stage, block, constraint) triplet where a violation occurred. 5 columns.

Hive Partitioning

All simulation Parquet output uses Hive partitioning: results for each scenario are stored in a directory named scenario_id=NNNN/ containing a single data.parquet file. The scenario_id column is encoded in the directory name, not as a column inside the Parquet file.

All major columnar data tools understand this layout and can read an entire simulation/<entity>/ directory as a single table with an automatically inferred scenario_id column:

# Polars — reads all scenarios at once, infers scenario_id from directory names
import polars as pl
df = pl.read_parquet("results/simulation/costs/")
print(df.head())

# Pandas with PyArrow backend
import pandas as pd
df = pd.read_parquet("results/simulation/costs/")

-- DuckDB — filter to a specific scenario at the storage layer
SELECT * FROM read_parquet('results/simulation/costs/**/*.parquet')
WHERE scenario_id = 0;

# R with the arrow package
library(arrow)
ds <- open_dataset("results/simulation/costs/")
dplyr::collect(dplyr::filter(ds, scenario_id == 0))

Scenario IDs are zero-based integers. The total number of scenarios is documented in simulation/_manifest.json under scenarios.total.

Manifest Files

Both training/_manifest.json and simulation/_manifest.json follow the same write protocol:

1. Serialize JSON to a temporary .json.tmp sibling file.
2. Atomically rename the .tmp file to the target path.

This ensures consumers never observe a partial manifest. If a manifest file exists, it contains a complete JSON document. If a run is interrupted before the final manifest write, the .tmp file may remain but the manifest itself will reflect the last successful checkpoint, not a partial write.

The status field is always the first indicator to check:

cobre report reads both manifests and training/metadata.json and prints a combined JSON summary to stdout. Use it in CI pipelines or shell scripts to inspect outcomes without parsing JSON directly:

# Extract the termination reason
cobre report results/ | jq '.training.convergence.termination_reason'

# Fail a CI job if the run did not complete
status=$(cobre report results/ | jq -r '.status')
[ "$status" = "complete" ] || exit 1