Skip to content

Record content hashes in sync state, not just paths and sizes

Date: 2026-07-02

Forest's sync checksums covered only (relative_path, file_size) tuples. Any change that kept a file's size — an in-place edit of fixed-width data, silent corruption, remote tampering — was invisible: push reported SKIP … already synced and status reported synced forever. The research note docs/research/dud-rclone-lessons.md (§6–7) identified this as forest's single biggest integrity gap compared to dud, whose content-addressed storage makes byte changes structurally detectable.

Sync entries now record per-file SHA-256 content hashes. At transfer time (push, and pull after files land) each file's bytes are hashed exactly once via compute_content_checksum, which returns both the per-file map (stored as SyncEntry.file_hashes) and an aggregate checksum over the sorted relpath:size:sha256 lines (stored as SyncEntry.checksum). The push skip decision compares that content-sensitive aggregate, so a same-size edit now re-pushes instead of skipping. Pull entries also gained file_snapshot, so every recorded transfer carries the full per-file evidence later safety checks need.

status deliberately does not re-hash the world. Its default comparison for hash-bearing entries is the cheap {relpath: size} snapshot — same information as before, same speed, still offline. The new forest status --checksum flag opts into re-hashing local contents against the recorded file_hashes, flipping same-size edits to stale. Hashing happens where the integrity claim is made: push always hashes because it is the operation that asserts "already synced".

There is no state-file migration. file_hashes is an optional field; entries written by older versions validate unchanged (file_hashes=None) and are compared with the legacy size-based checksum, exactly like the file_snapshot precedent. The first push after upgrading re-pushes each unit once (the legacy checksum never matches a content checksum) and heals the entry; pushes are additive, so this is safe.

Consequences

  • Same-size local edits and corruption are detected by push (re-push, not SKIP) and by status --checksum (stale); default status keeps its fast size-based verdict and documents that boundary.
  • Pushing large units pays one SHA-256 pass over the files being pushed; status cost is unchanged unless --checksum is passed.
  • Every new sync entry (push and pull) carries file_snapshot and file_hashes, the evidence base for the pull dirty-file guard (ADR 0018) and force-with-lease push protection (ADR 0020).
  • One-time re-push of every unit recorded by a pre-0017 forest.
  • Contract locked in tests/test_push_core_lock.py (VAL-PUSH-041 inverted: same-size edit IS detected; VAL-PUSH-043/044) and tests/test_status_lock.py (VAL-STATUS-020/021); supersedes the old VAL-PUSH-041 "not detected" expectation.