05 — Duplicate Detection

From the Burrow™ specification set.

05 — Duplicate Detection

Categories

Exact Duplicate

Byte-identical content confirmed by full cryptographic hash.

Probable Duplicate

Strong metadata similarity but not fully hashed.

Similar/Versioned File

Name/content patterns suggest versions, but files differ.

Cross-Storage Duplicate

Exact content appears across different roots/providers/volumes.

Staged Algorithm

Stage 1 — Size Grouping

Only files with identical logical size can be exact duplicates.

Zero-byte files should be handled separately because they can create noisy groups.

Stage 2 — Fast Signature

For sufficiently large candidate files, compute a partial signature from deterministic file regions.

This is an optimization only, never final proof.

Stage 3 — Full Hash

Use BLAKE3 by default to confirm exact equality.

SHA-256 may be supported for manifests/interoperability.

Hash Cache

Store:

  • algorithm;
  • digest;
  • calculated time;
  • file identity/change evidence;
  • hash status.

Recalculate when content may have changed.

Cloud Files

Do not force-download cloud-only files solely for duplicate detection without explicit user action.

Such files can remain probable candidates until content becomes available.

Recommendation Rules

An exact duplicate group should show:

  • each path;
  • provider/volume;
  • logical and allocated size;
  • modification date;
  • cloud state;
  • backup/archive role;
  • hash verification;
  • potential recoverable local space.

The system must never automatically select a "winner" solely because one file is newer.

Similar Names

Patterns such as final, copy, (1), or version numbers are useful for discovery but never proof of duplication.

Directories

Directory duplication should initially be derived from contained files, not treated as a simple path/name match.

Reclaimable Space

Calculate reclaimable space per physical location. Removing a cloud placeholder may reclaim nearly nothing locally; removing a downloaded exact duplicate may reclaim allocated bytes.

v0.2 Acceptance

  • identical fixture files group correctly;
  • same-name/different-content files never become exact duplicates;
  • hash changes invalidate prior equality;
  • symlink behavior is safe;
  • cloud-unavailable content is not force-downloaded;
  • duplicate calculations remain deterministic.