81 lines
2.3 KiB
Markdown
81 lines
2.3 KiB
Markdown
# MidHash256
|
|
|
|
## Purpose
|
|
|
|
O(1) file identification hash - constant time regardless of file size. Designed for quick duplicate detection and large file identification before expensive full-hash operations.
|
|
|
|
## Multicodec
|
|
|
|
- **Code**: `0x1000` (custom, not in official multicodec registry)
|
|
- **CID codec**: `0x55` (raw)
|
|
- **Output key**: `cid_midhash256`
|
|
|
|
## Algorithm
|
|
|
|
```
|
|
1. Read file size (S)
|
|
2. If S <= 1MB:
|
|
sample = entire file content
|
|
3. If S > 1MB:
|
|
seek to (S - 1MB) / 2
|
|
sample = read 1MB from middle
|
|
4. Hash = SHA-256(S as u64 big-endian || sample)
|
|
```
|
|
|
|
### Pseudocode
|
|
|
|
```rust
|
|
fn midhash256(path: &Path) -> [u8; 32] {
|
|
let size = file_size(path);
|
|
let mut hasher = SHA256::new();
|
|
|
|
// Always include size as 8-byte big-endian prefix
|
|
hasher.update(size.to_be_bytes());
|
|
|
|
if size <= 1_048_576 {
|
|
// Small file: hash entire content
|
|
hasher.update(read_all(path));
|
|
} else {
|
|
// Large file: hash middle 1MB
|
|
let middle_start = (size - 1_048_576) / 2;
|
|
hasher.update(read_at(path, middle_start, 1_048_576));
|
|
}
|
|
|
|
hasher.finalize()
|
|
}
|
|
```
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Time complexity | O(1) for files > 1MB |
|
|
| Space complexity | O(1) - fixed 1MB buffer |
|
|
| Hash size | 32 bytes (SHA-256) |
|
|
| Collision resistance | Same as SHA-256 for middle content |
|
|
|
|
## Use Cases
|
|
|
|
1. **Quick duplicate detection** - Identify potential duplicates before computing full hashes
|
|
2. **Large file identification** - Identify terabyte files in ~20ms regardless of size
|
|
3. **Pre-filtering** - Filter candidates before expensive operations (full hash, content analysis)
|
|
4. **Change detection** - Quick check if middle content changed (useful for video files)
|
|
|
|
## Limitations
|
|
|
|
- Files differing only at edges (first/last 512KB for 2MB+ files) will have same midhash
|
|
- Not suitable as a standalone deduplication hash - use as pre-filter only
|
|
- Custom multicodec code may conflict with future official allocations
|
|
|
|
## CID Example
|
|
|
|
```
|
|
File: movie.mkv (4.7 GB)
|
|
MidHash256: 3a7b...c2d1 (32 bytes)
|
|
CID: bafkreibh4xvl5... (base32lower encoded CIDv1)
|
|
```
|
|
|
|
## Integration
|
|
|
|
The hash is computed separately from the standard single-pass algorithms since it requires seeking to the middle of the file. For files <= 1MB, it reads the entire file like other algorithms.
|