Skip to content

Feed the pipeline with locate + schema key_outputs, keep run identity explicit

Making single-run execution frictionless is a workstream separate from checkout. forest stays pipeline-agnostic: the Snakefile queries forest at runtime instead of hand-listing data paths. Resolution uses the existing locate command plus schema key_outputs (currently unpopulated in the built-in schemas) — the schema becomes the single source of truth for intra-unit layout (wsp, fcs_dir, sorts_dir, bcl_dir). No new forest path command is added. locate resolves local paths offline; locate remote gives remote URLs.

The residual per-run information — which input units make up a logical run — is irreducible domain knowledge and stays as a thin runs: entry in the workflow config (sorting_unit, sequencing_unit). Per-repo manifests are not built in v1: most downstream stages (e.g. bcl2airr's 13) are pipeline outputs with generated unit ids that do not fit a run→unit map; a manifest would only cover the two input stages. --id-driven sync is deferred; a minimal input-only manifest can be added later if that need is felt.

Consequences

  • Author key_outputs in facs-sort / illumina-bcl-run / 10x-* schemas.
  • The Snakefile calls forest -C <repo> locate <name> --stage <s> <unit> in input functions.
  • runs: shrinks from four hardcoded paths to ~two unit ids per run; no manifest infrastructure in v1.