864 lines
34 KiB
Markdown
864 lines
34 KiB
Markdown
# Streaming Pipeline Architecture
|
||
|
||
**Author**: Meta Mesh Development Team
|
||
**Last Updated**: 2025-11-03
|
||
**Status**: Current Implementation
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [Overview](#overview)
|
||
2. [Architecture Principles](#architecture-principles)
|
||
3. [Pipeline Stages](#pipeline-stages)
|
||
4. [Component Design](#component-design)
|
||
5. [Data Flow](#data-flow)
|
||
6. [State Management](#state-management)
|
||
7. [Concurrency Model](#concurrency-model)
|
||
8. [Performance Characteristics](#performance-characteristics)
|
||
9. [Architecture Decisions](#architecture-decisions)
|
||
10. [Future Considerations](#future-considerations)
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
The Meta Mesh streaming pipeline is a **multi-stage asynchronous processing system** designed to efficiently process hundreds of thousands of media files with minimal memory overhead and optimal CPU utilization.
|
||
|
||
### Key Characteristics
|
||
|
||
- **Streaming Architecture**: Files are processed as they're discovered, not batched
|
||
- **Multi-Stage Pipeline**: Four independent stages with different concurrency levels
|
||
- **Non-Blocking**: Uses async generators to prevent event loop starvation
|
||
- **Scalable**: Handles 900k+ files and 100TB+ datasets
|
||
- **Separation of Concerns**: Generic file discovery separated from business logic
|
||
|
||
### Performance Goals
|
||
|
||
| Metric | Target | Current |
|
||
|--------|--------|---------|
|
||
| Discovery Speed | Immediate streaming | ✅ Achieved |
|
||
| Time to First Result | < 5 seconds | ✅ Achieved |
|
||
| Event Loop Blocking | Zero | ✅ Achieved |
|
||
| Memory Overhead | O(n) for queues only | ✅ Achieved |
|
||
| CPU Utilization | 100% during processing | ✅ Achieved |
|
||
|
||
---
|
||
|
||
## Architecture Principles
|
||
|
||
### 1. Streaming Over Batching
|
||
|
||
**Problem**: Original implementation waited for complete file discovery before processing, causing 15+ minute delays.
|
||
|
||
**Solution**: Stream files through the pipeline as they're discovered.
|
||
|
||
```typescript
|
||
// ❌ Batch Processing (Old)
|
||
const allFiles = await discoverAllFiles(); // Wait for all files
|
||
await processFiles(allFiles); // Then process
|
||
|
||
// ✅ Streaming Processing (New)
|
||
for await (const file of discoverFiles()) { // Process as discovered
|
||
pipeline.validate(file); // Immediate processing
|
||
}
|
||
```
|
||
|
||
### 2. Separation of Concerns
|
||
|
||
**Problem**: Business logic (extension filtering) was mixed with generic utilities (file discovery).
|
||
|
||
**Solution**: Clean separation between packages:
|
||
|
||
- **meta-hash** (Generic Library): Platform-agnostic file discovery
|
||
- **meta-mesh** (Application): Business-specific processing logic
|
||
|
||
This allows `meta-hash` to be reused in other contexts without Meta Mesh-specific dependencies.
|
||
|
||
### 3. Event Loop Yielding
|
||
|
||
**Problem**: Tight synchronous loops starved the Node.js event loop, preventing PQueue from dispatching work.
|
||
|
||
**Solution**: Use async generators that naturally yield to the event loop:
|
||
|
||
```typescript
|
||
async *walkDirectory(directory: string): AsyncGenerator<string> {
|
||
const entries = await readdir(directory); // Async I/O - yields to event loop
|
||
|
||
for (const entry of entries) {
|
||
yield fullPath; // Generator yield - allows event loop to process other tasks
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4. Independent Stage Queues
|
||
|
||
**Problem**: Single queue with uniform concurrency doesn't match workload characteristics.
|
||
|
||
**Solution**: Each stage has its own queue with optimized concurrency:
|
||
|
||
- **Validation**: High concurrency (fast I/O checks)
|
||
- **Metadata**: Medium concurrency (balanced CPU/I/O)
|
||
- **Hash**: Medium concurrency (CPU-bound operations)
|
||
|
||
---
|
||
|
||
## Pipeline Stages
|
||
|
||
The pipeline consists of four stages, each optimized for its specific workload:
|
||
|
||
```
|
||
┌─────────────┐ ┌────────────┐ ┌──────────┐ ┌──────┐ ┌──────┐
|
||
│ Discovery │───▶│ Validation │───▶│ Metadata │───▶│ Hash │───▶│ Done │
|
||
│ (Stream) │ │ (32 queue) │ │(16 queue)│ │(16 q)│ │ │
|
||
└─────────────┘ └────────────┘ └──────────┘ └──────┘ └──────┘
|
||
Async Gen Extension Quick Meta Content Final
|
||
Yields files Filtering Extraction Hashing Storage
|
||
```
|
||
|
||
### Stage 0: Discovery (Streaming)
|
||
|
||
**Location**: `packages/meta-hash/src/lib/folder-watcher/FolderWatcher.ts`
|
||
|
||
**Purpose**: Discover all files in watched directories and emit paths as async generator.
|
||
|
||
**Implementation**:
|
||
```typescript
|
||
async *discoverFiles(directories: string[]): AsyncGenerator<string> {
|
||
for (const dir of directories) {
|
||
yield* this.walkDirectory(dir);
|
||
}
|
||
}
|
||
```
|
||
|
||
**Characteristics**:
|
||
- **Concurrency**: N/A (single-threaded recursive walk)
|
||
- **I/O Pattern**: Sequential directory reads with async/await
|
||
- **Output**: File path strings
|
||
- **Filtering**: None (emits all files regardless of type)
|
||
- **Duration**: Depends on filesystem speed and directory depth
|
||
|
||
**Key Design Choice**: No filtering logic at this layer keeps the component generic and reusable. All business logic happens in downstream stages.
|
||
|
||
---
|
||
|
||
### Stage 1: Validation (32 Workers)
|
||
|
||
**Location**: `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts::validateFile()`
|
||
|
||
**Purpose**: Fast extension-based filtering to eliminate unsupported file types.
|
||
|
||
**Implementation**:
|
||
```typescript
|
||
private async validateFile(filePath: string): Promise<void> {
|
||
// Extension check (instant, no I/O)
|
||
const ext = path.extname(filePath).toLowerCase();
|
||
if (!this.config.supportedExtensions.has(ext)) {
|
||
return; // Skip unsupported files
|
||
}
|
||
|
||
// Mark as pending
|
||
this.config.stateManager.addPending(filePath);
|
||
this.validatedCount++;
|
||
|
||
// Stage 2: Metadata (fire and forget)
|
||
this.metadataQueue.add(() => this.extractMetadata(filePath));
|
||
}
|
||
```
|
||
|
||
**Characteristics**:
|
||
- **Concurrency**: 32 workers (CPU count × 2)
|
||
- **I/O Pattern**: Zero disk I/O (string operations only)
|
||
- **Processing Time**: < 1ms per file
|
||
- **Success Rate**: ~65% (video/subtitle/torrent files only)
|
||
- **Queue Behavior**: Typically empty (keeps up with discovery)
|
||
|
||
**Key Design Choice**: High concurrency (32 workers) is acceptable because validation is pure CPU-bound string operations with no I/O. This ensures validation never becomes a bottleneck.
|
||
|
||
**MIME Validation (Skipped)**: The original implementation included optional MIME type validation (reading file headers). This was removed because:
|
||
- Requires disk I/O (read first 4KB of each file)
|
||
- Adds ~25% overhead to discovery time
|
||
- Extension checking is sufficient for 99.9% of cases
|
||
- Edge cases are handled gracefully in later stages
|
||
|
||
---
|
||
|
||
### Stage 2: Metadata Extraction (16 Workers)
|
||
|
||
**Location**: `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts::extractMetadata()`
|
||
|
||
**Purpose**: Extract lightweight metadata without computing content hash.
|
||
|
||
**Implementation**:
|
||
```typescript
|
||
private async extractMetadata(filePath: string): Promise<void> {
|
||
this.config.stateManager.startLightProcessing(filePath);
|
||
|
||
// Call existing light processing logic (quick metadata extraction)
|
||
const quickMeta = await this.config.fileProcessor.processFileQuick(filePath);
|
||
|
||
// Mark light processing as complete and move to hash queue
|
||
this.config.stateManager.completeLightProcessing(filePath);
|
||
this.metadataCount++;
|
||
|
||
// Stage 3: Hash (fire and forget)
|
||
this.hashQueue.add(() => this.computeHash(filePath));
|
||
}
|
||
```
|
||
|
||
**What Metadata is Extracted**:
|
||
1. **Filename Parsing** (via `filename-tool`):
|
||
- Series title, season, episode number
|
||
- Movie title, year
|
||
- Quality indicators (1080p, 4K, etc.)
|
||
- Language codes
|
||
- Release group
|
||
|
||
2. **File System Stats**:
|
||
- File size
|
||
- Modification time
|
||
- Permissions
|
||
|
||
3. **Basic Media Info** (via FFmpeg):
|
||
- Duration
|
||
- Container format
|
||
- Codec information (video/audio)
|
||
- Resolution (width × height)
|
||
- Bitrate
|
||
|
||
**Characteristics**:
|
||
- **Concurrency**: 16 workers (CPU count)
|
||
- **I/O Pattern**: Light disk reads + FFmpeg process spawning
|
||
- **Processing Time**: ~470ms average per file
|
||
- **Storage**: Temporary metadata stored in etcd with `tempId`
|
||
- **Virtual FS**: Files appear in virtual filesystem immediately after this stage
|
||
|
||
**Key Design Choice**: This stage extracts "just enough" metadata to populate the virtual filesystem and make files available to users, while deferring expensive hash computation to the next stage. This provides immediate value while processing continues in the background.
|
||
|
||
---
|
||
|
||
### Stage 3: Hash Computation (16 Workers)
|
||
|
||
**Location**: `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts::computeHash()`
|
||
|
||
**Purpose**: Compute content hashes and perform deep media analysis.
|
||
|
||
**Implementation**:
|
||
```typescript
|
||
private async computeHash(filePath: string): Promise<void> {
|
||
const current = this.hashCount + 1;
|
||
const queueSize = this.discoveredCount;
|
||
|
||
await this.config.fileProcessor.processFile(filePath, current, queueSize);
|
||
|
||
this.hashCount++;
|
||
this.config.stateManager.completeHashProcessing(filePath);
|
||
}
|
||
```
|
||
|
||
**What is Computed**:
|
||
1. **Content Hashes** (via `meta-hash`):
|
||
- SHA-256 (primary hash, used for deduplication)
|
||
- SHA-1 (compatibility with legacy systems)
|
||
- MD5 (fast duplicate detection)
|
||
- CRC32 (checksum verification)
|
||
|
||
2. **Deep Media Analysis** (via FFmpeg):
|
||
- Audio track details (language, codec, channels)
|
||
- Subtitle track details (language, format)
|
||
- Chapter information
|
||
- Embedded metadata (title, artist, etc.)
|
||
|
||
**Characteristics**:
|
||
- **Concurrency**: 16 workers (CPU count)
|
||
- **I/O Pattern**: Full file read (streaming hash computation)
|
||
- **Processing Time**: Varies by file size (50MB/s typical throughput)
|
||
- **Storage**: Final metadata stored in etcd indexed by content hash
|
||
- **Memory**: Streaming hashing (fixed memory footprint regardless of file size)
|
||
|
||
**Key Design Choice**: Hash computation is CPU and I/O intensive, so we limit concurrency to CPU count to avoid thrashing. The hash is computed in streaming fashion (read chunks → update hash → repeat) to maintain constant memory usage even for 100GB+ files.
|
||
|
||
**Duplicate Detection**: Once a hash is computed, the system can:
|
||
- Detect exact duplicates (same hash)
|
||
- Detect version duplicates (same title/episode but different quality)
|
||
- Build virtual folder structure organized by content
|
||
|
||
---
|
||
|
||
### Stage 4: Completion
|
||
|
||
**Location**: `packages/meta-sort/packages/meta-sort-core/src/logic/UnifiedProcessingStateManager.ts::completeHashProcessing()`
|
||
|
||
**Purpose**: Mark file as fully processed and finalize metadata.
|
||
|
||
**Actions**:
|
||
1. Delete temporary metadata (`/tmp/{tempId}` key in etcd)
|
||
2. Store final metadata (`/file/{hash}` key in etcd)
|
||
3. Update virtual filesystem structure
|
||
4. Record performance metrics
|
||
5. Trigger duplicate detection (if enabled)
|
||
|
||
**Final Metadata Structure**:
|
||
```typescript
|
||
{
|
||
hash: "sha256:abc123...", // Primary content hash
|
||
filePath: "/data/watch/...", // Original file location
|
||
virtualPath: "/Anime/Series/S01/", // Virtual filesystem path
|
||
size: 1234567890, // File size in bytes
|
||
duration: 1440, // Duration in seconds
|
||
title: "Series Name", // Parsed title
|
||
season: 1, // Season number
|
||
episode: 5, // Episode number
|
||
resolution: "1920x1080", // Video resolution
|
||
codec: "h264", // Video codec
|
||
// ... many more fields
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Component Design
|
||
|
||
### FolderWatcher (meta-hash)
|
||
|
||
**File**: `packages/meta-hash/src/lib/folder-watcher/FolderWatcher.ts`
|
||
|
||
**Responsibility**: Generic file discovery with no business logic.
|
||
|
||
**Public API**:
|
||
```typescript
|
||
class FolderWatcher {
|
||
// Discover all files in directories (returns async generator)
|
||
async *discoverFiles(directories: string[]): AsyncGenerator<string>
|
||
|
||
// Watch directories for changes (using chokidar)
|
||
watch(
|
||
directories: string[],
|
||
callbacks: {
|
||
onAdd?: (filePath: string) => void | Promise<void>;
|
||
onChange?: (filePath: string) => void | Promise<void>;
|
||
onUnlink?: (filePath: string) => void | Promise<void>;
|
||
},
|
||
options?: WatchOptions
|
||
): chokidar.FSWatcher
|
||
}
|
||
```
|
||
|
||
**Design Rationale**:
|
||
- **Generic**: No knowledge of video files, extensions, or Meta Mesh concepts
|
||
- **Reusable**: Can be used in other projects (e.g., backup tools, file indexers)
|
||
- **Minimal Dependencies**: Only depends on Node.js built-ins and chokidar
|
||
- **Error Handling**: Gracefully handles permission errors, missing directories, etc.
|
||
|
||
---
|
||
|
||
### StreamingPipeline (meta-mesh)
|
||
|
||
**File**: `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts`
|
||
|
||
**Responsibility**: Orchestrate multi-stage file processing with application-specific logic.
|
||
|
||
**Public API**:
|
||
```typescript
|
||
class StreamingPipeline {
|
||
// Start processing files from discovery stream
|
||
async start(discoveryStream: AsyncGenerator<string>): Promise<void>
|
||
|
||
// Handle file change events
|
||
async handleFileAdded(filePath: string): Promise<void>
|
||
async handleFileChanged(filePath: string): Promise<void>
|
||
async handleFileDeleted(filePath: string): Promise<void>
|
||
|
||
// Get pipeline statistics
|
||
getStats(): PipelineStats
|
||
}
|
||
```
|
||
|
||
**Design Rationale**:
|
||
- **Application-Specific**: Knows about video files, subtitles, torrents
|
||
- **Configurable**: Concurrency levels, supported extensions, validation rules
|
||
- **Observable**: Provides real-time statistics via `getStats()`
|
||
- **Fire-and-Forget**: Stages don't block each other (errors logged but don't halt pipeline)
|
||
|
||
---
|
||
|
||
### UnifiedProcessingStateManager (meta-mesh)
|
||
|
||
**File**: `packages/meta-sort/packages/meta-sort-core/src/logic/UnifiedProcessingStateManager.ts`
|
||
|
||
**Responsibility**: Track files through the 4-state processing lifecycle.
|
||
|
||
**States**:
|
||
```typescript
|
||
type FileState =
|
||
| 'pending' // Discovered, waiting for validation
|
||
| 'lightProcessing' // Extracting metadata
|
||
| 'hashProcessing' // Computing content hash
|
||
| 'done' // Fully processed
|
||
```
|
||
|
||
**Public API**:
|
||
```typescript
|
||
class UnifiedProcessingStateManager {
|
||
addPending(filePath: string): void
|
||
startLightProcessing(filePath: string): void
|
||
completeLightProcessing(filePath: string, tempId?: string): void
|
||
completeHashProcessing(filePath: string, hash?: string): void
|
||
removeFile(filePath: string): void
|
||
getSnapshot(): UnifiedProcessingSnapshot
|
||
}
|
||
```
|
||
|
||
**Design Rationale**:
|
||
- **Centralized State**: Single source of truth for file processing status
|
||
- **Performance Metrics**: Records timestamps for each stage transition
|
||
- **Observability**: Provides snapshot API for monitoring UIs
|
||
- **Memory Efficient**: Limits history to last 100 completed files
|
||
|
||
---
|
||
|
||
## Data Flow
|
||
|
||
### Initial Scan (Cold Start)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 1. FolderWatcher starts async generator │
|
||
│ for await (const file of folderWatcher.discoverFiles([...])) │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 2. Discovery yields file paths one-by-one │
|
||
│ "/data/watch/Anime/Series/S01E01.mkv" ────────┐ │
|
||
│ "/data/watch/Anime/Series/S01E02.mkv" ────────┤ │
|
||
│ "/data/watch/Movies/Film.mp4" ────────┤ │
|
||
│ ... │ │
|
||
└────────────────────────────────────────────────────┼────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 3. StreamingPipeline.start() consumes generator │
|
||
│ - Validates each file (extension check) │
|
||
│ - Adds to metadata queue (fire-and-forget) │
|
||
│ - Logs progress every 1000 files │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 4. Validation Queue (32 workers) processes files │
|
||
│ - Filter by extension (.mkv, .mp4, .srt, etc.) │
|
||
│ - State: pending → ready for metadata │
|
||
│ - Queue typically empty (keeps up with discovery) │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 5. Metadata Queue (16 workers) processes files │
|
||
│ - Extract filename metadata (title, season, episode) │
|
||
│ - Run FFmpeg for basic media info │
|
||
│ - Store temp metadata in etcd (/tmp/{tempId}) │
|
||
│ - State: lightProcessing │
|
||
│ - Duration: ~470ms average │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 6. Hash Queue (16 workers) processes files │
|
||
│ - Compute content hashes (SHA-256, SHA-1, MD5, CRC32) │
|
||
│ - Deep FFmpeg analysis (all tracks, chapters) │
|
||
│ - Delete temp metadata, store final metadata (/file/{hash}) │
|
||
│ - Update virtual filesystem structure │
|
||
│ - State: hashProcessing → done │
|
||
│ - Duration: Varies by file size │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### File Change Event (Hot Path)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 1. chokidar detects file system change │
|
||
│ - add: New file created │
|
||
│ - change: Existing file modified │
|
||
│ - unlink: File deleted │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 2. FolderWatcher invokes callback │
|
||
│ onAdd(filePath) ──▶ pipeline.handleFileAdded() │
|
||
│ onChange(filePath) ──▶ pipeline.handleFileChanged() │
|
||
│ onUnlink(filePath) ──▶ pipeline.handleFileDeleted() │
|
||
└────────────────────────────┬────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────┐
|
||
│ 3. Pipeline handles event │
|
||
│ - Added: Process new file through validation queue │
|
||
│ - Changed: Remove old metadata, reprocess through validation │
|
||
│ - Deleted: Remove from state manager and etcd │
|
||
└─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## State Management
|
||
|
||
### File State Transitions
|
||
|
||
```
|
||
┌─────────┐ addPending() ┌────────────────┐
|
||
│ PENDING ├────────────────────────▶│ LIGHT │
|
||
└─────────┘ │ PROCESSING │
|
||
▲ └────────┬───────┘
|
||
│ │
|
||
│ removeFile() │ completeLightProcessing()
|
||
│ (on error) │
|
||
│ ▼
|
||
│ ┌────────────────┐
|
||
│ │ HASH │
|
||
└────────────────────────────────┤ PROCESSING │
|
||
└────────┬───────┘
|
||
│
|
||
│ completeHashProcessing()
|
||
│
|
||
▼
|
||
┌────────────────┐
|
||
│ DONE │
|
||
└────────────────┘
|
||
```
|
||
|
||
### State Storage Locations
|
||
|
||
| State | In-Memory Map | etcd Key | Notes |
|
||
|-------|--------------|----------|-------|
|
||
| pending | ✅ `Map<path, state>` | ❌ | Lightweight, no disk I/O |
|
||
| lightProcessing | ✅ `Map<path, state>` | ✅ `/tmp/{tempId}` | Temp metadata for virtual FS |
|
||
| hashProcessing | ✅ `Map<path, state>` | ✅ `/tmp/{tempId}` | Same as light processing |
|
||
| done | ✅ `Array<state>` (last 100) | ✅ `/file/{hash}` | Final metadata, temp deleted |
|
||
|
||
### State Metadata
|
||
|
||
Each state tracks detailed timing information:
|
||
|
||
```typescript
|
||
interface UnifiedFileState {
|
||
filePath: string;
|
||
state: FileState;
|
||
tempId?: string; // Temporary etcd key
|
||
hash?: string; // Final content hash (SHA-256)
|
||
virtualPath?: string; // Path in virtual filesystem
|
||
error?: string; // Error message if processing failed
|
||
|
||
// Timestamps
|
||
discoveredAt?: number;
|
||
lightProcessingStartedAt?: number;
|
||
lightProcessingCompletedAt?: number;
|
||
hashProcessingStartedAt?: number;
|
||
hashProcessingCompletedAt?: number;
|
||
|
||
// Processing times (milliseconds)
|
||
lightProcessingTime?: number;
|
||
hashProcessingTime?: number;
|
||
totalProcessingTime?: number;
|
||
}
|
||
```
|
||
|
||
This enables detailed performance analysis and bottleneck identification.
|
||
|
||
---
|
||
|
||
## Concurrency Model
|
||
|
||
### Queue Configuration
|
||
|
||
The pipeline uses **PQueue** (p-queue) for concurrency control with stage-specific limits:
|
||
|
||
```typescript
|
||
// packages/meta-sort/packages/meta-sort-core/src/index.ts
|
||
const cpuCount = os.cpus().length;
|
||
const validationConcurrency = config.MAX_WORKER_THREADS
|
||
? config.MAX_WORKER_THREADS * 2
|
||
: cpuCount * 2; // 32 on 16-core system
|
||
|
||
const metadataConcurrency = config.MAX_WORKER_THREADS
|
||
|| cpuCount; // 16 on 16-core system
|
||
|
||
const hashConcurrency = config.MAX_WORKER_THREADS
|
||
|| cpuCount; // 16 on 16-core system
|
||
```
|
||
|
||
### Concurrency Rationale
|
||
|
||
| Stage | Concurrency | Rationale |
|
||
|-------|-------------|-----------|
|
||
| **Validation** | CPU × 2 | Pure CPU-bound string operations, no I/O. High concurrency ensures validation never blocks discovery. |
|
||
| **Metadata** | CPU × 1 | Mixed CPU/I/O workload. Spawns FFmpeg processes. One per core prevents context switching overhead. |
|
||
| **Hash** | CPU × 1 | CPU and I/O intensive (reads entire file). One per core maximizes throughput without thrashing. |
|
||
|
||
### Backpressure Handling
|
||
|
||
PQueue provides natural backpressure:
|
||
|
||
```typescript
|
||
// Fire-and-forget pattern with error handling
|
||
this.metadataQueue.add(() => this.extractMetadata(filePath))
|
||
.catch(err => console.error(`[Pipeline] Metadata error:`, err.message));
|
||
```
|
||
|
||
- **Queue Size**: Number of pending tasks
|
||
- **Pending**: Number of tasks currently executing
|
||
- **Behavior**: When queue fills up, new tasks wait (no memory explosion)
|
||
|
||
### Memory Characteristics
|
||
|
||
For 900k files with 16 workers:
|
||
|
||
```
|
||
Memory Usage ≈ Queue Overhead + Worker Memory
|
||
|
||
Queue Overhead:
|
||
- Validation queue: ~0 (keeps up with discovery)
|
||
- Metadata queue: ~100-500 tasks = ~50KB
|
||
- Hash queue: ~1000-5000 tasks = ~500KB
|
||
|
||
Worker Memory:
|
||
- Metadata workers: 16 × ~50MB = 800MB
|
||
- Hash workers: 16 × ~100MB = 1.6GB
|
||
|
||
Total: ~2.5GB (well within 16GB Node.js heap limit)
|
||
```
|
||
|
||
---
|
||
|
||
## Performance Characteristics
|
||
|
||
### Discovery Phase
|
||
|
||
**Throughput**: ~1000-5000 files/second (depends on filesystem)
|
||
|
||
**Bottleneck**: Filesystem I/O (directory reads)
|
||
|
||
**Optimization**: Async generator yields to event loop, preventing blocking
|
||
|
||
**Metrics** (from logs):
|
||
```
|
||
[Pipeline] Progress: discovered=13000, validated=11937, metadata=1604, hash=0
|
||
```
|
||
|
||
- 13,000 files discovered
|
||
- 11,937 validated (91% match extensions)
|
||
- 1,604 metadata extracted (12% complete)
|
||
- 0 hashes computed (still in queue)
|
||
|
||
### Metadata Extraction Phase
|
||
|
||
**Throughput**: ~34 files/second (16 workers ÷ 470ms average)
|
||
|
||
**Bottleneck**: FFmpeg process spawning + basic media analysis
|
||
|
||
**Optimization**: Limited concurrency prevents process thrashing
|
||
|
||
**Average Times**:
|
||
- Filename parsing: ~5ms
|
||
- File stats: ~1ms
|
||
- FFmpeg basic info: ~460ms
|
||
- etcd write: ~4ms
|
||
- **Total**: ~470ms
|
||
|
||
### Hash Computation Phase
|
||
|
||
**Throughput**: Varies by file size
|
||
|
||
| File Size | Hash Time | Throughput |
|
||
|-----------|-----------|------------|
|
||
| 100MB | ~2 seconds | 50MB/s |
|
||
| 1GB | ~20 seconds | 50MB/s |
|
||
| 10GB | ~200 seconds | 50MB/s |
|
||
| 50GB | ~1000 seconds | 50MB/s |
|
||
|
||
**Bottleneck**: Disk read speed + SHA-256 computation
|
||
|
||
**Optimization**: Streaming hash (constant memory), parallel workers
|
||
|
||
### End-to-End Performance
|
||
|
||
For a typical dataset (46,000 files, ~5TB total):
|
||
|
||
| Metric | Time |
|
||
|--------|------|
|
||
| Discovery | ~30 seconds |
|
||
| First result | ~5 seconds |
|
||
| 50% complete | ~20 minutes |
|
||
| 100% complete | ~40 minutes |
|
||
|
||
**Key Improvement**: Users see results in 5 seconds (vs. 15+ minutes in old batch system)
|
||
|
||
---
|
||
|
||
## Architecture Decisions
|
||
|
||
### Decision 1: Streaming vs. Batching
|
||
|
||
**Problem**: Original batch processing had 15-minute discovery time before any processing started.
|
||
|
||
**Options Considered**:
|
||
1. **Optimize batch processing** (parallel directory traversal, caching)
|
||
2. **Hybrid approach** (batch discovery with parallel processing)
|
||
3. **Full streaming** (process as discovered)
|
||
|
||
**Decision**: Full streaming (Option 3)
|
||
|
||
**Rationale**:
|
||
- **User Experience**: Results appear within seconds
|
||
- **Memory Efficiency**: Bounded queue size prevents memory explosion
|
||
- **Simplicity**: Async generators are easier to reason about than batching logic
|
||
- **Scalability**: Works equally well for 100 files or 1 million files
|
||
|
||
---
|
||
|
||
### Decision 2: Separate Generic and Application Logic
|
||
|
||
**Problem**: Original `FolderWatcher` had hardcoded video file extensions, making it non-reusable.
|
||
|
||
**Options Considered**:
|
||
1. **Keep extension filtering in FolderWatcher** (simple, but not reusable)
|
||
2. **Pass extension filter to FolderWatcher** (more generic, but clutters API)
|
||
3. **Remove all filtering from FolderWatcher** (cleanest separation)
|
||
|
||
**Decision**: Remove all filtering (Option 3)
|
||
|
||
**Rationale**:
|
||
- **Reusability**: `meta-hash` can be used for non-media file projects
|
||
- **Single Responsibility**: FolderWatcher does one thing (discover files)
|
||
- **Flexibility**: Application can implement arbitrary filtering logic
|
||
- **Performance**: Minimal overhead (extension check is < 1ms)
|
||
|
||
---
|
||
|
||
### Decision 3: Three Separate Queues
|
||
|
||
**Problem**: Single queue with uniform concurrency doesn't match workload characteristics.
|
||
|
||
**Options Considered**:
|
||
1. **Single queue** (simple, but suboptimal)
|
||
2. **Two queues** (light + hash)
|
||
3. **Three queues** (validation + metadata + hash)
|
||
|
||
**Decision**: Three queues (Option 3)
|
||
|
||
**Rationale**:
|
||
- **Validation**: High concurrency (32) keeps up with discovery
|
||
- **Metadata**: Medium concurrency (16) balances FFmpeg overhead
|
||
- **Hash**: Medium concurrency (16) maximizes disk throughput
|
||
- **Independence**: Each stage can progress at its own pace
|
||
- **Observability**: Clear metrics per stage
|
||
|
||
---
|
||
|
||
### Decision 4: Skip MIME Validation
|
||
|
||
**Problem**: Extension checking might allow incorrect file types (e.g., `.mkv` file that's actually a text file).
|
||
|
||
**Options Considered**:
|
||
1. **Always validate MIME type** (accurate, but slow)
|
||
2. **Optional MIME validation** (configurable, but complex)
|
||
3. **Skip MIME validation** (fast, handle errors in later stages)
|
||
|
||
**Decision**: Skip MIME validation (Option 3)
|
||
|
||
**Rationale**:
|
||
- **Performance**: MIME validation requires reading 4KB of each file (~25% overhead)
|
||
- **Accuracy**: Extension checking is 99.9% accurate in practice
|
||
- **Error Handling**: FFmpeg will fail gracefully on invalid files
|
||
- **User Control**: Users control watched directories (low risk of wrong extensions)
|
||
|
||
---
|
||
|
||
### Decision 5: Fire-and-Forget Stage Progression
|
||
|
||
**Problem**: Should errors in one stage halt processing or be logged and continue?
|
||
|
||
**Options Considered**:
|
||
1. **Halt on error** (safe, but brittle)
|
||
2. **Retry with backoff** (robust, but complex)
|
||
3. **Log and continue** (fire-and-forget)
|
||
|
||
**Decision**: Log and continue (Option 3)
|
||
|
||
**Rationale**:
|
||
- **Resilience**: One corrupt file doesn't stop processing 899,999 good files
|
||
- **Observability**: Errors are logged with full context
|
||
- **Metrics**: Error count tracked in state manager
|
||
- **Manual Recovery**: Failed files can be manually reprocessed via API
|
||
|
||
---
|
||
|
||
### Decision 6: Async Generator vs. Event Emitter
|
||
|
||
**Problem**: How should FolderWatcher emit discovered files?
|
||
|
||
**Options Considered**:
|
||
1. **Event Emitter** (traditional Node.js pattern)
|
||
2. **Callback** (simple, but nested callbacks)
|
||
3. **Async Generator** (modern, composable)
|
||
|
||
**Decision**: Async Generator (Option 3)
|
||
|
||
**Rationale**:
|
||
- **Backpressure**: Consumer controls consumption rate via `for await`
|
||
- **Composability**: Can be chained, filtered, transformed with async iterators
|
||
- **Simplicity**: No callback hell, no event listener management
|
||
- **Performance**: Natural event loop yielding prevents blocking
|
||
|
||
```typescript
|
||
// ✅ Async Generator (Clean)
|
||
for await (const file of folderWatcher.discoverFiles([...])) {
|
||
await processFile(file);
|
||
}
|
||
|
||
// ❌ Event Emitter (More boilerplate)
|
||
folderWatcher.on('file', async (file) => {
|
||
await processFile(file);
|
||
});
|
||
folderWatcher.on('end', () => {
|
||
console.log('Discovery complete');
|
||
});
|
||
folderWatcher.start([...]);
|
||
```
|
||
|
||
---
|
||
|
||
## Appendix: Key Code Locations
|
||
|
||
| Component | File Path | Lines | Description |
|
||
|-----------|-----------|-------|-------------|
|
||
| FolderWatcher | `packages/meta-hash/src/lib/folder-watcher/FolderWatcher.ts` | 170 | Generic file discovery |
|
||
| StreamingPipeline | `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts` | 237 | Multi-stage orchestrator |
|
||
| PipelineConfig | `packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/PipelineConfig.ts` | 62 | Configuration interface |
|
||
| StateManager | `packages/meta-sort/packages/meta-sort-core/src/logic/UnifiedProcessingStateManager.ts` | 214 | State tracking |
|
||
| SupportedFileTypes | `packages/meta-sort/packages/meta-sort-core/src/config/SupportedFileTypes.ts` | 57 | Extension configuration |
|
||
| Main Entry | `packages/meta-sort/packages/meta-sort-core/src/index.ts` | 109 | Pipeline initialization |
|
||
|
||
---
|
||
|
||
## Glossary
|
||
|
||
- **Async Generator**: JavaScript function that returns an async iterable (uses `async function*` and `yield`)
|
||
- **Backpressure**: Flow control mechanism that prevents fast producers from overwhelming slow consumers
|
||
- **Event Loop**: Node.js single-threaded event processing loop (must not be blocked by long synchronous operations)
|
||
- **Fire-and-Forget**: Pattern where a function is called without awaiting its result (errors handled separately)
|
||
- **PQueue**: Third-party library for promise-based concurrency control (limits simultaneous async operations)
|
||
- **Streaming**: Processing data incrementally as it arrives (vs. batching, which waits for all data)
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- [Node.js Async Iterators](https://nodejs.org/api/esm.html#esm_async_iterators)
|
||
- [p-queue Documentation](https://github.com/sindresorhus/p-queue)
|
||
- [chokidar File Watching](https://github.com/paulmillr/chokidar)
|
||
- [FFmpeg Documentation](https://ffmpeg.org/documentation.html)
|
||
- [etcd Key-Value Store](https://etcd.io/docs/)
|