Rebuild Runbook

Use this runbook for deterministic refresh + progression regeneration.

Design Rationale

The rebuild pipeline is split into two distinct phases — db-refresh and generate-progressions — rather than a single monolithic command. This separation exists because the data flow has a clear materialization boundary: raw SRD and homebrew JSON is normalized and loaded into SQLite first (engine/ingest/refresh_sqlite.py), then progression artifacts are derived from that normalized store (engine/ingest/generate_progressions.py). Keeping the phases separate means you can re-ingest source data without regenerating progressions, or regenerate progressions from an already-current database — useful when iterating on progression logic without changing source content.

The API rebuild endpoint (POST /api/tools/rebuild) in server/init.py always runs db-refresh before generation, ensuring the database is current regardless of when the last CLI refresh happened. This "refresh-then-generate" invariant avoids stale-database bugs at the cost of slightly longer API rebuilds.

An alternative considered was a file-watcher that would automatically regenerate on source changes. This was rejected because progression generation touches the filesystem (writing JSON to progressions/) and updates progressions/manifest.json, making concurrent writes dangerous. Explicit, operator-initiated rebuilds keep the pipeline deterministic.

Assumptions & Constraints

SQLite is the intermediate store. The pipeline assumes data/5e-database.sqlite exists and is writable. If the file is missing, db-refresh creates it; if it is locked by another process, the refresh will fail.
Source directories are fixed. engine/ingest/refresh_sqlite.py hardcodes srd/2014/ and srd/2024/ plus homebrew/ as source roots. Adding a new edition requires updating SOURCE_DIRECTORIES in that module.
Manifest is the catalog of truth. After generation, progressions/manifest.json is the single artifact that the API and UI use to discover available progressions. If generation partially fails, the manifest may reference files that don't exist or omit files that do. Always verify the manifest after a rebuild.
Idempotent but not incremental. Running a full rebuild twice produces identical output, but the pipeline re-processes every entity each time. There is no diffing or caching layer — each run is a clean derivation from the SQLite state.

Conceptual Model

Think of the rebuild as a two-stage compiler:

Front-end (db-refresh): Parses raw JSON from srd/ and homebrew/, normalizes records through engine/ingest/normalizers.py, and writes them into typed SQLite tables. Homebrew records are read from canonical JSON via engine/ingest/homebrew_store.py. The database is the "intermediate representation."
Back-end (generate-progressions): Reads the normalized database, applies progression-building logic (feature ordering, spell grants, mechanic refs), and emits immutable JSON files plus the manifest. The progression files are the "compiled output."

The entity-scoped rebuild (scope: "entity") skips full generation and targets a single progression file, but it still runs a full db-refresh first when invoked via the API. This is a deliberate trade-off: correctness over speed, since a single entity's progression may depend on cross-entity data in the database (e.g., subclass references to parent class features).

The API response shape reflects that split cleanly: scope: "all" returns the manifest, while scope: "entity" returns the regenerated entity document. That lets automation distinguish catalog-wide rebuilds from focused re-materialization without inspecting the filesystem.

Full Rebuild

poetry run lorewright db-refresh
poetry run lorewright generate-progressions

Equivalent API call:

curl -X POST http://localhost:8000/api/tools/rebuild -H "Content-Type: application/json" -d "{\"scope\":\"all\"}"

Single Entity Rebuild

poetry run lorewright generate-progressions --entity-type class --edition 2014 --index bloodrager

Equivalent API payload:

{
  "scope": "entity",
  "entity_type": "class",
  "edition": "2014",
  "index": "bloodrager"
}

Post-Run Checks

Verify progressions/manifest.json changed as expected.
Spot-check output JSON under progressions/classes/ or progressions/subclasses/.
Run wiki and pipeline tests from Testing Strategy.