Skip to content

Recovering from failed syncs

What to do when forest push or forest pull fails, in whole or in part.

Symptoms

  • Per-unit transport failures print the rclone stderr:
ERROR multi/SI-TT-E3  Failed to copy: connection reset by peer
  • A missing local target prints:
FAIL multi/SI-TT-E3  target path does not exist: /path/to/unit
  • After 5 consecutive unit failures a push or pull run aborts early instead of hammering a dead remote:
Error: aborting after 5 consecutive failures.
  • A transfer that succeeded but could not be confirmed against the remote prints a verification failure (ADR 0019):
ERROR raw/plot-02  verification failed: survey/raw/plot-02/temps.csv: content hash mismatch

Nothing was recorded for that unit, so re-running the same command retransfers it. A mismatch that persists across re-runs means the remote is altering or truncating content — investigate the backend before raising retry knobs (verification failures never retry). Renamed/remapped keys use the strongest available SHA-256, SHA-1, or MD5 digest, falling back to size only when the backend exposes no usable hash.

  • A pull that would overwrite local changes forest has never synced refuses the unit (ADR 0018):
ERROR raw/plot-02  local file has uncommitted changes: notes.txt (use --force to overwrite)

Keep the local version by pushing it, or discard it with pull --force. Other clean units in the same run are handled independently.

  • A push that would overwrite remote files that changed out-of-band refuses the unit (ADR 0020):
ERROR raw/plot-02  remote changed since last sync: survey/raw/plot-02/temps.csv (use --force to overwrite)

A first push into an occupied prefix with no sync history refuses the same way (remote already has '<key>' with no recorded sync history). Inspect the remote with forest ls (or the backend's own tooling), then either pull --force to adopt the remote state or push --force to overwrite it deliberately.

Diagnose

  1. Re-run with --verbose for per-file detail and the remote banner.
  2. Turn on structured logs and re-run:
FOREST_LOG_FILE=/tmp/forest.jsonl FOREST_LOG_FORMAT=json forest push
  1. Every log record carries the invocation run_id. Grep it to follow one run end to end:
grep <run_id> /tmp/forest.jsonl

Retry activity shows up as transient rclone failure; retrying events (with exit_code, attempt, delay_s); a unit that exhausted retries logs rclone failed with the final exit_code and attempts.

Tuning knobs

Variable Default Effect
FOREST_TRANSFER_RETRIES 2 Extra attempts after the first failure; 0 disables
FOREST_RETRY_BASE_DELAY 0.5 Initial backoff seconds; doubles per attempt

Retries apply only to transient failures (src/forest/resilience.py): rclone exit code 5 ("temporary error") or stderr matching timeout / connection-reset / rate-limit patterns. Assertion-class failures — bad remote, missing path, auth — never retry; fix the cause instead of raising the knobs.

Recover

  1. Fix the cause (network, credentials via rclone config, disk space, missing local path).
  2. Re-run the same command. Push and pull are idempotent: a clean unit whose source content and remote path match the last recorded sync prints SKIP <stage>/<unit> already synced to <remote>, so only the failed units transfer again. Pull skips only when the backend supplies usable content hashes; otherwise it safely retransfers. Sync state is only recorded after a transfer verifies, so a failed unit always retransfers.
  3. An interrupted pull cannot leave partial files: each file downloads to a .forest-tmp temporary name and is atomically renamed into place, so the destination holds either the old bytes or the new bytes. Stray temps are cleaned up and are never picked up by a later push.
  4. Sync state (.forest/checkouts/<name>/sync_state.json) is written under a per-checkout sync_state.json.lock flock that spans each unit's whole check → transfer → verify → record span, so concurrent runs on the same checkout serialize per unit, an interrupted run cannot corrupt the state, and no manual lock cleanup is needed — the lock releases when the process exits.