Recovering from failed syncs¶
When forest push or forest pull fails or is interrupted (an error, Ctrl-C, kill -9, a power loss), fix the cause (network, credentials, disk space), then run the same command again, with no extra flags:
Forest records a unit only after its transfer is verified, so failed units transfer again and finished ones print SKIP <stage>/<unit> already synced to <remote>. rclone also skips files that already match.
A push stopped after its upload but before it recorded the unit (a lost network during verification, say) needs no --force either. The rerun keeps each remote file whose size and content hash already match the local file, uploads only the rest, verifies the whole unit and records it. A remote with no content hash cannot prove a match, so that case still refuses.
Pull is the mirror image. A local file that already holds the remote file's bytes is not a local change, since pulling would write the same bytes. So a pull stopped before it recorded reruns without --force, and a file you never synced that matches the remote is kept. One that differs still refuses. Pull also saves the hashes of the files it is about to download as pending writes, so a file the stopped pull wrote is replaced even if the remote changed again before the rerun. A file you edited after the stop matches neither and still refuses.
Editing a file between the stop and the rerun is fine too. Before each upload, Forest saves the content hashes of the bytes it is about to write in the checkout's sync_state.json (pending), and the record that follows drops them. A remote copy holding one of those hashes is Forest's own write, so the rerun replaces it with your edit. A remote file that matches neither the last sync, your local file, nor a pending write was changed by someone else and still refuses.
After 5 failed units in a row, a run stops with Error: aborting after 5 consecutive failures.
Check where things stand¶
status compares with the remote and transfers nothing. See the CLI reference for what each row means.
Refusals and --force¶
If the remote changed since your last sync, push refuses the unit:
$ forest push
ERROR raw/run1 remote changed since last sync: demo/raw/run1/a.txt (use --force to overwrite)
A first push into a remote prefix that already has files refuses with remote already has '<key>' with no recorded sync history.
So does a push after the stage was rebound to another folder or its remote location changed: the old history describes other bytes. The refusal adds last sync used <old path> -> <old remote>; now <new path> -> <new remote>. Use --force only when the new folder should replace the remote copy. A rebound folder whose bytes match the last sync skips and takes over that history.
If a local file changed since the last sync, pull refuses the unit:
Other units in the same run carry on. Pick the side to keep:
forest push --force # overwrite the remote with your local files
forest pull --force # overwrite your local files with the remote
--force skips the check and compares files by checksum instead of size and time. A forced push still skips synced units. A forced pull runs every unit, so a synced one prints PULL raw/run1 0 files, 0 B <- origin.
A line with verification failed means the transfer finished but Forest could not prove the copy matches. Nothing was recorded, so re-running retries it. If it keeps failing, look at the remote backend.
When the verification listing itself fails (a DNS or network error, say), Forest lists again after 30 seconds and then after 90 more before it gives up with cannot list the remote after 3 tries.
Symlinks and link files¶
Push stores each symlink as an rclone link file, <path>.rclonelink. A remote that already holds data at a link's path, such as full copies from an earlier push that followed links, can't also hold the link file: a pull could not tell which one is meant. Push refuses the unit, even with --force:
$ forest push
ERROR raw/run1 remote already holds data at link paths: demo/raw/run1/ref.fa
ERROR raw/run1 not pushed; Forest never deletes remote files: remove those remote keys, or --exclude the paths
The same refusal names a link file left on the remote where you now push real data (remote holds a link file where this push sends data). Forest never deletes remote files. Remove the stale keys with your own tools after checking them, then push again, or --exclude the paths to push the rest. Pull refuses the same mix with remote holds both a link file and data at its path.
Locks¶
Push and pull hold a lock on .forest/checkouts/<name>/sync_state.json.lock while each unit transfers. Two runs on the same checkout wait for each other. The OS releases the lock when the process exits, even after a crash, so there is nothing to delete.
Leftover *.forest-tmp files¶
Pull downloads each file to <name>.forest-tmp and renames it into place when it finishes, so an interrupted pull never leaves a half-written file under the real name. It can leave the .forest-tmp file, though. Forest ignores these files in push and status but never deletes them. Once no forest run is active, remove them yourself:
An interrupted upload to S3 can also leave an incomplete multipart upload behind. S3 does not list it as a file and Forest never sees it, but S3 stores its parts until they are aborted. A bucket lifecycle rule that aborts incomplete multipart uploads after a few days cleans these up.
Retries and logs¶
Temporary rclone failures (exit code 5, timeouts, connection resets, rate limits) retry automatically. FOREST_TRANSFER_RETRIES (default 2) and FOREST_RETRY_BASE_DELAY (default 0.5 seconds, doubling each attempt) tune this. Other failures, like a bad remote or bad credentials, never retry.
For a detailed log of a failing run:
The environment reference lists every variable.