metastar/meta-sort
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Pierre
2026-01-11 17:26:49 +01:00
commit 888790fd11
240 changed files with 48691 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+841
View File
@@ -0,0 +1,841 @@
# meta-sort
A standalone file sorting and metadata extraction service for the MetaMesh ecosystem. meta-sort monitors watch folders, extracts metadata from media files, stores it in a distributed key-value store, and manages remote drive mounting for shared access.
## Overview
meta-sort is a **process-write** service that serves as the primary data ingestion point for the MetaMesh platform. It:
1. **Watches folders** for new media files (video, audio, subtitles)
2. **Extracts metadata** using FFmpeg and filename parsing
3. **Computes content identifiers** using midhash256 (fast) and SHA-256 (background)
4. **Stores metadata** in a distributed KV store (Redis or etcd)
5. **Manages remote mounts** (NFS, SMB, cloud storage) for shared access
---
## Meta-Core Architecture
MetaMesh uses **two shared volumes** for operation:
### 1. META_CORE_VOLUME - Infrastructure
The infrastructure volume stores all service coordination data:
| Context | Path | Description |
|---------|------|-------------|
| **Inside Container** | `/meta-core` | Mounted shared volume |
| **Production (Host)** | `/DATA/Apps/meta-core` | External shared drive mount point |
| **Development** | `./data/meta-core` | Local development directory |
```
META_CORE_VOLUME (/meta-core or /DATA/Apps/meta-core)
├── db/ # KV Database storage
│ ├── redis/ # Redis persistence (RDB + AOF)
│ │ ├── dump.rdb # Redis snapshot
│ │ └── appendonly.aof # Redis append-only log
│ └── etcd/ # etcd data (alternative)
│ └── member/
│
├── locks/ # Distributed locking
│ └── kv-leader.lock # Leader election lock file
│ # Contains: {"host":"service-name","api":"http://host:port"}
│
├── services/ # Service discovery registry
│ ├── meta-sort.json # {"name":"meta-sort","api":"http://...","status":"running","pid":123}
│ ├── meta-fuse.json # {"name":"meta-fuse","api":"http://...","status":"running","pid":456}
│ ├── meta-stremio.json # {"name":"meta-stremio","api":"http://...","status":"running","pid":789}
│ └── meta-orbit.json # {"name":"meta-orbit","api":"http://...","status":"running","pid":321}
│
└── config/ # Shared configuration
├── remote-storage.conf # Remote mount definitions
└── smb-credentials.txt # SMB credentials (chmod 600)
```
### 2. FILES_VOLUME - Shared Media Drive
The **FILES_VOLUME** (canonical name: `/files`) is the shared drive for all media files. This is where all actual media content resides, and file paths in the KV database reference this volume.
| Context | Path | Description |
|---------|------|-------------|
| **Inside Container** | `/files` | Mounted shared media volume |
| **Production (Host)** | Varies (NFS/SMB/local) | External shared drive |
| **Development** | `./data/files` | Local development directory |
```
FILES_VOLUME (/files)
├── media1/ # Local folder mounted from host
│ ├── Movies/
│ │ └── Inception (2010)/
│ │ └── Inception.mkv
│ └── TV Shows/
│ └── Breaking Bad/
│ └── Season 01/
│
├── media2/ # Another local media folder
│ └── Anime/
│ └── Steins Gate/
│
├── smb/ # SMB/CIFS remote mount
│ └── NAS-Media/
│ └── Movies/
│
├── nfs/ # NFS remote mount
│ └── server-media/
│
└── gdrive/ # rclone remote mount (Google Drive)
└── Cloud-Media/
```
### File Path References
All file paths stored in the KV database are **relative to FILES_VOLUME**:
```
KV Key: /file/{midhash256}/filePath
KV Value: "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
Full path construction:
Container: /files + /media1/Movies/... = /files/media1/Movies/Inception (2010)/Inception.mkv
Any host: $FILES_VOLUME + /media1/Movies/... = $FILES_VOLUME/media1/Movies/Inception (2010)/Inception.mkv
```
This allows services to be deployed on different hosts while sharing the same volume. The FILES_VOLUME can be:
- A local folder bind mount
- An NFS share mounted on the host
- An SMB/CIFS share
- A FUSE mount (rclone, sshfs, etc.)
### Mount Management
meta-sort manages remote mounts **inside** the FILES_VOLUME for shared access:
```bash
# meta-sort mounts remote storage INTO /files
/files/smb/nas-share ← SMB mount managed by meta-sort
/files/nfs/server ← NFS mount managed by meta-sort
/files/gdrive/ ← rclone mount managed by meta-sort
# Other folders are bind mounts from host
/files/media1 ← Host folder mounted into container
/files/media2 ← Another host folder
```
All services access files through `/files/*`, regardless of whether the underlying storage is local, NFS, SMB, or cloud.
---
## Leader Election and Lock Mechanism
### How It Works
MetaMesh uses a **filesystem-based distributed consensus** mechanism using `flock(2)`. The shared filesystem acts as the consensus layer.
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Leader Election via flock │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Step 1: Service Startup │
│ ┌─────────────┐ │
│ │ Service │ ──▶ Try to acquire flock on /meta-core/locks/kv-leader.lock│
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ flock() Result │ │
│ │ ┌─────────────────────┐ ┌──────────────────────────────────┐ │ │
│ │ │ Lock Acquired │ │ Lock Blocked │ │ │
│ │ │ ────────────── │ │ ───────────── │ │ │
│ │ │ │ │ │ │ │
│ │ │ BECOME LEADER │ │ BECOME FOLLOWER │ │ │
│ │ │ 1. Spawn Redis │ │ 1. Read lock file │ │ │
│ │ │ 2. Write API │ │ 2. Extract leader API │ │ │
│ │ │ endpoint to │ │ 3. Connect to leader's │ │ │
│ │ │ lock file │ │ Redis │ │ │
│ │ │ 3. Accept clients │ │ 4. Start reconnect loop │ │ │
│ │ └─────────────────────┘ └──────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### Lock File Format
The lock file at `/meta-core/locks/kv-leader.lock` contains JSON:
```json
{
"host": "meta-sort-instance-1",
"api": "redis://10.0.1.50:6379",
"http": "http://10.0.1.50:3000",
"timestamp": 1703808000000,
"pid": 12345
}
```
### flock Implementation
```typescript
import { open, flock } from 'fs';
import { promisify } from 'util';
const LOCK_FILE = '/meta-core/locks/kv-leader.lock';
async function tryBecomeLeader(): Promise<boolean> {
const fd = await promisify(open)(LOCK_FILE, 'w');
try {
// LOCK_EX = exclusive lock, LOCK_NB = non-blocking
await promisify(flock)(fd, 'exnb'); // Throws if cannot acquire
// We are the leader - spawn KV database and write our endpoint
await spawnRedis();
await fs.writeFile(LOCK_FILE, JSON.stringify({
host: hostname(),
api: `redis://${myIP}:6379`,
http: `http://${myIP}:3000`,
timestamp: Date.now(),
pid: process.pid
}));
return true; // We are leader
} catch (err) {
if (err.code === 'EAGAIN' || err.code === 'EWOULDBLOCK') {
// Lock held by another process - become follower
return false;
}
throw err;
}
}
```
### Failure Detection and Recovery
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Failure Detection Loop │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Follower Service: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ while (true) { │ │
│ │ try { │ │
│ │ await redis.ping(); // Health check │ │
│ │ await sleep(5000); // Wait 5 seconds │ │
│ │ } catch (connectionError) { │ │
│ │ // Leader might be down │ │
│ │ await tryBecomeLeader(); // Attempt to take over │ │
│ │ if (isLeader) break; │ │
│ │ await reconnectToNewLeader(); // Or connect to new leader │ │
│ │ } │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ When Leader Dies: │
│ 1. flock is automatically released by OS │
│ 2. First follower to call flock() wins │
│ 3. New leader spawns Redis, writes new endpoint │
│ 4. Other followers detect connection failure, read new endpoint │
│ │
└──────────────────────────────────────────────────────────────────────────┘
```
---
## Service Discovery
### Registration Mechanism
Each service registers itself in `/meta-core/services/{service-name}.json`:
```json
{
"name": "meta-sort",
"version": "1.0.0",
"api": "http://10.0.1.50:3000",
"status": "running",
"pid": 12345,
"hostname": "meta-sort-instance-1",
"startedAt": "2024-12-28T10:00:00Z",
"lastHeartbeat": "2024-12-28T12:30:00Z",
"capabilities": ["write", "mount", "monitor"],
"endpoints": {
"health": "/health",
"api": "/api",
"metrics": "/api/metrics"
}
}
```
### Service Lifecycle
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Service Lifecycle │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Startup: │
│ 1. Create/update /meta-core/services/{name}.json │
│ 2. Set status = "starting" │
│ 3. Initialize (connect to KV, etc.) │
│ 4. Set status = "running" │
│ 5. Start heartbeat loop (update lastHeartbeat every 30s) │
│ │
│ Running: │
│ - Heartbeat updates lastHeartbeat timestamp │
│ - Other services read this file to discover endpoints │
│ - stale detection: if (now - lastHeartbeat > 60s) → service dead │
│ │
│ Shutdown: │
│ 1. Set status = "stopping" │
│ 2. Graceful shutdown (finish current work) │
│ 3. Delete /meta-core/services/{name}.json OR set status = "stopped" │
│ │
└──────────────────────────────────────────────────────────────────────────┘
```
### Discovering Other Services
```typescript
async function discoverService(name: string): Promise<ServiceInfo | null> {
const servicePath = `/meta-core/services/${name}.json`;
try {
const content = await fs.readFile(servicePath, 'utf-8');
const service = JSON.parse(content);
// Check if service is alive (heartbeat within last 60 seconds)
const lastHeartbeat = new Date(service.lastHeartbeat).getTime();
const isAlive = Date.now() - lastHeartbeat < 60_000;
if (!isAlive) {
console.warn(`Service ${name} appears stale (last heartbeat: ${service.lastHeartbeat})`);
return null;
}
return service;
} catch (err) {
if (err.code === 'ENOENT') {
return null; // Service not registered
}
throw err;
}
}
```
---
## MetaMesh Ecosystem Architecture
### Service Hierarchy
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ MetaMesh Ecosystem │
│ │
│ Shared Volumes: │
│ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐ │
│ │ META_CORE_VOLUME │ │ FILES_VOLUME │ │
│ │ (/meta-core) │ │ (/files) │ │
│ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │
│ │ │ db/ │ │ locks/ │ │svc/ │ │ │ │media1/ │ │ smb/ │ │ nfs/ │ │ │
│ │ │(Redis) │ │(flock) │ │(JSON)│ │ │ │(local) │ │(mount) │ │(mnt) │ │ │
│ │ └────┬───┘ └───┬────┘ └──┬───┘ │ │ └───┬────┘ └───┬────┘ └──┬───┘ │ │
│ └───────┼─────────┼─────────┼─────┘ └──────┼──────────┼─────────┼─────┘ │
│ │ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐│
│ │ ││
│ │ [1] meta-sort PROCESS-WRITE ││
│ │ ├── Watches /files/* for new media files ││
│ │ ├── Extracts metadata → writes to KV (Redis in /meta-core/db) ││
│ │ ├── Manages remote mounts INTO /files (NFS, SMB, rclone) ││
│ │ ├── Leader election (spawns Redis if leader) ││
│ │ └── Serves monitoring UI (meta-sort-editor) ││
│ │ ││
│ │ [2] meta-fuse PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Builds virtual filesystem from metadata ││
│ │ ├── Serves FUSE mount + WebDAV ││
│ │ └── File paths resolve to /files/* ││
│ │ ││
│ │ [3] meta-stremio PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Stremio addon server (manifest, catalog, streams) ││
│ │ ├── HLS transcoding via FFmpeg ││
│ │ └── Video files from /files/* ││
│ │ ││
│ │ [4] meta-orbit SHARING-READ-WRITE ││
│ │ ├── P2P metadata synchronization (libp2p + OrbitDB) ││
│ │ ├── Decentralized search across network ││
│ │ ├── Reads from local KV, shares to network ││
│ │ └── Receives from network, writes to local KV ││
│ │ ││
│ └───────────────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### Data Flow
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ Data Flow │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ File Discovery: │
│ │
│ /files/media1/Movies/Inception (2010)/Inception.mkv │
│ │ │
│ ▼ │
│ [meta-sort] ──▶ Extract metadata ──▶ Write to Redis (in /meta-core/db) │
│ │ │ │
│ │ Key: file:{midhash256} │ │
│ │ filePath: "media1/Movies/Inception (2010)/Inception.mkv" │
│ │ title: "Inception" │
│ │ year: "2010" │
│ │ duration: 8880 │
│ │ │ │
│ ▼ ▼ │
│ │
│ File Access: │
│ │
│ [meta-fuse] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Virtual FS: /mnt/virtual/Movies/Inception (2010)/Inception.mkv │
│ │ ↓ │
│ │ symlink to /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-stremio] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Stream: GET /stremio/stream/{midhash256}/playlist.m3u8 │
│ │ ↓ │
│ │ transcode /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-orbit] ◀──▶ Sync with P2P network │
│ │ │
│ ▼ │
│ Share metadata to peers / Receive metadata from peers │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Deployment
### Docker Compose (Production)
```yaml
version: '3.8'
services:
meta-sort:
image: metazla/meta-sort:latest
volumes:
# Infrastructure volume (KV DB, locks, services)
- /DATA/Apps/meta-core:/meta-core:rw
# Shared media volume
- /DATA/Apps/meta-core/files:/files:rw,shared
# OR mount multiple sources into /files
# - /mnt/nas/media:/files/nas:ro
# - /mnt/local/media:/files/local:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
- WATCH_PATHS=/files
- REDIS_DATA_DIR=/meta-core/db/redis
privileged: true # Required for rclone FUSE mounts
meta-fuse:
image: metazla/meta-fuse:latest
volumes:
# Infrastructure (read-only for followers)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
# FUSE mount output
- /mnt/metamesh:/mnt/metamesh:rw,shared
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
privileged: true # Required for FUSE
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse
meta-stremio:
image: metazla/meta-stremio:latest
volumes:
# Infrastructure (read-only)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
ports:
- "7000:7000"
meta-orbit:
image: metazla/meta-orbit:latest
volumes:
# Infrastructure (read-write for P2P sync)
- /DATA/Apps/meta-core:/meta-core:rw
environment:
- META_CORE_PATH=/meta-core
ports:
- "4001:4001" # libp2p
```
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `META_CORE_PATH` | `/meta-core` | Path to META_CORE_VOLUME (infrastructure) |
| `FILES_PATH` | `/files` | Path to FILES_VOLUME (shared media) |
| `REDIS_URL` | Auto-discover | Redis connection (from leader lock file) |
| `WATCH_PATHS` | `$FILES_PATH` | Folders to watch for media |
| `SERVICE_NAME` | Container hostname | Unique service identifier |
---
## File Processing Pipeline
meta-sort uses a **plugin-based architecture** with a two-queue task scheduler:
```
┌──────────────┐ ┌────────────────────────────────────────────────────┐
│ Discovery │ │ Task Scheduler │
│ (Stream) │───▶│ ┌─────────────────┐ ┌──────────────────────────┐ │
└──────────────┘ │ │ Fast Queue │ │ Background Queue │ │
Watch folder │ │ (16 workers) │ │ (4 workers) │ │
+ manual scan │ │ <1s tasks │ │ >=1s tasks │ │
│ └─────────────────┘ └──────────────────────────┘ │
└────────────────────────────────────────────────────┘
```
Tasks are automatically classified and scheduled based on plugin dependencies:
```
Discovery → file-info → ffmpeg → filename-parser → jellyfin-nfo → tmdb
↓
anime-detector ← language ← subtitle ← torrent ← full-hash
```
---
## Plugin System
Plugins are the core processing units that extract and enrich metadata. Each plugin:
- Has a `manifest.yml` defining metadata schema, config, and dependencies
- Runs in either **fast** (high concurrency) or **background** (low concurrency) queue
- Can depend on other plugins to ensure correct execution order
### Built-in Plugins
| Plugin | Queue | Priority | Description |
|--------|-------|----------|-------------|
| **file-info** | fast | 10 | Basic file info (type, MIME, size, midhash256) |
| **ffmpeg** | fast | 15 | Video/audio/subtitle stream metadata via FFprobe |
| **filename-parser** | fast | 20 | Parse title, season, episode, year, quality from filename |
| **jellyfin-nfo** | fast | 25 | Parse Jellyfin/Kodi NFO metadata files |
| **tmdb** | background | 30 | Fetch metadata from The Movie Database API |
| **anime-detector** | fast | 35 | Detect anime content from keywords and audio tracks |
| **language** | fast | 40 | Aggregate languages from all streams (ISO 639-3) |
| **subtitle** | fast | 45 | Find sibling subtitle files and detect language |
| **torrent** | fast | 50 | Parse .torrent file metadata |
| **full-hash** | background | 100 | Compute SHA-256 hash of entire file |
### Plugin Manifest Example
```yaml
id: ffmpeg
name: FFmpeg Metadata
version: 1.0.0
description: Extracts video/audio/subtitle stream metadata using FFprobe
priority: 15
color: "#4CAF50"
defaultQueue: fast
dependencies:
- file-info
config: {}
schema:
fileinfo/duration:
label: Duration
type: string
readonly: true
fileinfo/streamdetails/video:
label: Video Streams
type: json
readonly: true
pattern: "fileinfo/streamdetails/video/{n}/*"
```
### Task Queue Architecture
- **Fast Queue** (16 workers): High-throughput for quick operations (<1s)
- File info extraction, FFprobe, filename parsing, NFO reading
- **Background Queue** (4 workers): Low-concurrency for slow operations (>=1s)
- TMDB API calls, full SHA-256 hashing
- Automatically pauses when fast queue has work, resumes when idle
### Plugin Management
Plugins can be enabled/disabled and configured via:
- **Dashboard**: http://localhost:8180/ → Plugins tab
- **Config file**: `plugins.json` in cache directory
---
## Processing States
1. **Pending** - File discovered in watch folder
2. **Processing** - Plugins running (fast queue first, then background)
3. **Done** - All plugins completed successfully
---
## Package Structure
```
meta-sort/
├── packages/
│ ├── meta-sort-core/ # Core processing service (@meta-sort/core)
│ │ ├── plugins/ # Built-in plugins
│ │ │ ├── file-info/ # Basic file information
│ │ │ ├── ffmpeg/ # FFprobe metadata extraction
│ │ │ ├── filename-parser/# Filename parsing
│ │ │ ├── jellyfin-nfo/ # NFO file parsing
│ │ │ ├── tmdb/ # TMDB API integration
│ │ │ ├── anime-detector/ # Anime content detection
│ │ │ ├── language/ # Language aggregation
│ │ │ ├── subtitle/ # Subtitle file finder
│ │ │ ├── torrent/ # Torrent file parsing
│ │ │ └── full-hash/ # SHA-256 full file hashing
│ │ ├── src/
│ │ │ ├── api/ # Fastify REST API (UnifiedAPIServer)
│ │ │ ├── config/ # Environment configuration
│ │ │ ├── jellyfin/ # Jellyfin integration
│ │ │ ├── kv/ # KV manager with leader election (Redis)
│ │ │ ├── logic/ # File processing logic
│ │ │ ├── metrics/ # Performance tracking
│ │ │ ├── plugin-engine/ # Plugin loader and task scheduler
│ │ │ ├── types/ # TypeScript types
│ │ │ ├── utils/ # Utility functions
│ │ │ └── index.ts # Entry point
│ │ └── package.json
│ │
│ ├── meta-sort-editor/ # Metadata editor UI (@meta-sort/editor)
│ │ └── ... # React/Vite app
│ │
│ ├── meta-sort-ui/ # Monitoring dashboard (@meta-sort/ui)
│ │ └── ... # React/Vite app
│ │
│ └── async-utils/ # Async utility library
│ └── ... # MultiQueue, PromiseQueue, etc.
│
├── docker/
│ ├── nginx.conf # Reverse proxy config
│ ├── redis.conf # Redis configuration
│ └── supervisord.conf # Process management
│
├── Dockerfile
├── docker-compose.yml
├── package.json # Workspace root
├── pnpm-workspace.yaml
└── README.md
```
### Key Components (meta-sort-core)
| Component | File | Purpose |
|-----------|------|---------|
| Entry Point | `src/index.ts` | Initializes KV manager, plugin engine, API server, file watching |
| File Processor | `src/logic/WatchedFileProcessor.ts` | File discovery and plugin task scheduling |
| Plugin Loader | `src/plugin-engine/PluginLoader.ts` | Loads plugins from manifest.yml, manages plugin lifecycle |
| Task Scheduler | `src/plugin-engine/TaskScheduler.ts` | Two-queue scheduler (fast/background) with dependency resolution |
| KV Manager | `src/kv/KVManager.ts` | Leader election via flock, Redis spawning, service discovery |
| Unified API | `src/api/UnifiedAPIServer.ts` | Fastify REST API for monitoring and control |
| Performance Metrics | `src/metrics/PerformanceMetrics.ts` | Tracks processing throughput and timing |
| Config | `src/config/EnvConfig.ts` | Environment variable parsing |
---
## Core Features
### 1. Folder Watching
- **Real-time monitoring** using chokidar for file system events
- **Streaming discovery** via async generators for efficient memory usage
- **Configurable watch paths** supporting local and remote mounted directories
- **File type filtering** (video, audio, subtitle, torrent files)
### 2. Plugin-Based Metadata Extraction
Metadata extraction is handled by plugins that can be enabled/disabled and configured:
- **file-info** - Basic file info, MIME type, midhash256 content identifier
- **ffmpeg** - Duration, codec, resolution, bitrate, audio tracks, subtitles via FFprobe
- **filename-parser** - Extract title, year, season, episode, quality from filenames
- **jellyfin-nfo** - Parse NFO sidecar files (Jellyfin/Kodi format)
- **tmdb** - Fetch rich metadata from The Movie Database API
- **anime-detector** - Detect anime content from keywords and audio tracks
- **language** - Aggregate languages from all streams (ISO 639-3)
- **subtitle** - Find sibling subtitle files and detect language
- **torrent** - Parse .torrent file metadata
- **full-hash** - Compute SHA-256 hash of entire file (background)
### 3. KV Storage
- **Nested key architecture** for property-level granularity
- **Leader election** via flock for high availability
- **Service discovery** via shared filesystem
- **CRDT-compatible** for distributed synchronization
```
Key Structure:
/file/{midhash256}/title → "Inception"
/file/{midhash256}/year → "2010"
/file/{midhash256}/video/codec → "h265"
/file/{midhash256}/audio/0/language → "en"
/file/{midhash256}/filePath → "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
```
### 4. Remote Mount Management
- **NFS shares** - Network file systems
- **SMB/CIFS** - Windows network shares with credentials
- **rclone** - Cloud storage (Google Drive, Dropbox, S3, etc.)
- **Automatic mounting** on startup
- **Mounts created inside FILES_VOLUME** for shared access by all services
Mounts are created inside `/files/` (FILES_VOLUME) so all services can access them:
```bash
# Example mount structure
/files/ # FILES_VOLUME
├── media1/ # Local folder bind mount from host
├── media2/ # Another local folder
├── smb/ # SMB/CIFS remote mounts
│ └── nas-share/ # SMB: //server/share
├── nfs/ # NFS remote mounts
│ └── server-media/ # NFS: nas.local:/volume1/media
└── gdrive/ # rclone: gdrive:Media
```
---
## Configuration
### Environment Variables
```bash
# Volume paths
META_CORE_PATH=/meta-core # Infrastructure volume
FILES_PATH=/files # Shared media volume
# Watch paths (comma-separated, relative to FILES_PATH or absolute)
WATCH_PATHS=/files
# Processing configuration
MAX_WORKER_THREADS=16 # Defaults to CPU count
LIGHT_PROCESSING_TIMEOUT_MS=120000
HASH_PROCESSING_TIMEOUT_MS=7200000
# Remote mount configuration
REMOTE_STORAGE_CONFIG=/meta-core/config/remote-storage.conf
```
### Remote Storage Configuration
Create `/meta-core/config/remote-storage.conf`:
```conf
# NFS mount (into FILES_VOLUME)
nfs://nas.local:/volume1/media /files/nfs/nas-media nfs defaults,rw 0 0
# SMB mount (credentials in separate file)
//server/share /files/smb/nas-share cifs credentials=/meta-core/config/smb-creds.txt,uid=1000 0 0
# rclone mount
gdrive: /files/gdrive rclone vfs-cache-mode=full 0 0
```
---
## API
### REST Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/health` | GET | Service health check |
| `/api/scan/trigger` | POST | Trigger manual folder scan |
| `/api/processing/status` | GET | Current pipeline state |
| `/api/metrics` | GET | Performance metrics |
| `/api/mounts` | GET | List mounted remote storage |
| `/api/mounts` | POST | Add new remote mount |
| `/api/mounts/{id}` | DELETE | Remove remote mount |
### Processing Status Response
```json
{
"totalPending": 150,
"totalLightProcessing": 16,
"totalHashProcessing": 8,
"totalDone": 45000,
"totalFilesInVFS": 45150,
"throughput": {
"filesPerSecond": 34,
"avgLightProcessingMs": 470,
"avgHashProcessingMs": 15000
}
}
```
---
## Usage
### Development
```bash
# Install dependencies
pnpm install
# Build all packages
pnpm run build
# Run in development mode
pnpm run dev
```
### Docker
```bash
# Build and run with Docker Compose
docker-compose up -d
# Check health
curl http://localhost/health
curl http://localhost/api/processing/status
```
---
## License
MIT License - see LICENSE file for details.