14. Extract forest from biostore as a standalone data engine¶

Date: 2026-07-01

Status¶

Accepted

Context¶

biostore did two unrelated jobs in one package: a git-like data-management engine (workspaces, checkouts, remotes, bindings, push/pull/status/ls, sync state) and a schema/layout validator (data-type schemas, folder/file anatomy, detection). Coupling them meant the generic engine carried domain-specific knowledge it did not need, and neither concern could evolve or be used independently.

Decision¶

forest is the data-management half of biostore, extracted as a standalone, self-contained package. It manages arbitrary data trees and depends on no other project. The schema/layout-validation concern is out of scope for forest and lives elsewhere; forest neither imports it nor references it.

Concretely, forest:

keeps config, checkout, sync_state, rclone, paths, manifest, flow, and the CLI;
is a pure sync engine — it moves files and tracks their sync state but does not validate or interpret content. push/pull/status/diff/ls operate on all files in a unit (skipping OS junk); there is no content-based include/exclude, no pre-sync validation, and no schema-driven output resolution;
treats a stage's schema field as an opaque label it stores and round-trips but never acts on;
registers stages manually (forest add STAGE PATH) rather than auto-detecting data types.

On-disk names track the tool: .biostore/ → .forest/, and the checkout config biostore.yaml → forest.yaml. forest migrate performs the in-place rename.

Consequences¶

Forest is reusable for any data domain and installs on its own.
Content validation, detection, and schema-aware filtering are no longer part of forest; those are handled by a separate tool the user runs independently.
Existing biostore repos need a one-time forest migrate.