meta-sort
A standalone file sorting and metadata extraction service for the MetaMesh ecosystem. meta-sort monitors watch folders, extracts metadata from media files, stores it in a distributed key-value store, and manages remote drive mounting for shared access.
Overview
meta-sort is a process-write service that serves as the primary data ingestion point for the MetaMesh platform. It:
- Watches folders for new media files (video, audio, subtitles)
- Extracts metadata using FFmpeg and filename parsing
- Computes content identifiers using midhash256 (fast) and SHA-256 (background)
- Stores metadata in a distributed KV store (Redis or etcd)
- Manages remote mounts (NFS, SMB, cloud storage) for shared access
Meta-Core Architecture
MetaMesh uses two shared volumes for operation:
1. META_CORE_VOLUME - Infrastructure
The infrastructure volume stores all service coordination data:
| Context | Path | Description |
|---|---|---|
| Inside Container | /meta-core |
Mounted shared volume |
| Production (Host) | /DATA/Apps/meta-core |
External shared drive mount point |
| Development | ./data/meta-core |
Local development directory |
META_CORE_VOLUME (/meta-core or /DATA/Apps/meta-core)
├── db/ # KV Database storage
│ ├── redis/ # Redis persistence (RDB + AOF)
│ │ ├── dump.rdb # Redis snapshot
│ │ └── appendonly.aof # Redis append-only log
│ └── etcd/ # etcd data (alternative)
│ └── member/
│
├── locks/ # Distributed locking
│ └── kv-leader.lock # Leader election lock file
│ # Contains: {"host":"service-name","api":"http://host:port"}
│
├── services/ # Service discovery registry
│ ├── meta-sort.json # {"name":"meta-sort","api":"http://...","status":"running","pid":123}
│ ├── meta-fuse.json # {"name":"meta-fuse","api":"http://...","status":"running","pid":456}
│ ├── meta-stremio.json # {"name":"meta-stremio","api":"http://...","status":"running","pid":789}
│ └── meta-orbit.json # {"name":"meta-orbit","api":"http://...","status":"running","pid":321}
│
└── config/ # Shared configuration
├── remote-storage.conf # Remote mount definitions
└── smb-credentials.txt # SMB credentials (chmod 600)
2. FILES_VOLUME - Shared Media Drive
The FILES_VOLUME (canonical name: /files) is the shared drive for all media files. This is where all actual media content resides, and file paths in the KV database reference this volume.
| Context | Path | Description |
|---|---|---|
| Inside Container | /files |
Mounted shared media volume |
| Production (Host) | Varies (NFS/SMB/local) | External shared drive |
| Development | ./data/files |
Local development directory |
FILES_VOLUME (/files)
├── watch/ # Host media folder (read-only bind mount)
│ ├── Movies/
│ │ └── Inception (2010)/
│ │ └── Inception.mkv
│ └── TV Shows/
│ └── Breaking Bad/
│
├── test/ # Test media folder (development)
│
├── plugin/ # Plugin output files (read-write)
│ └── {plugin-id}/ # Per-plugin output directory
│
└── corn/ # SMB/rclone mounts (mounted inside container)
├── nas-share/ # SMB mount via rclone
└── gdrive/ # Google Drive mount via rclone
WebDAV File Access
meta-sort exposes /files via WebDAV at the /webdav/ endpoint. This allows:
- Container plugins to access files without direct volume mounts
- meta-fuse and meta-stremio to read files from meta-sort
- Access to dynamic mounts (SMB, rclone) that can't be pre-mounted in other containers
Container Plugin ──► WebDAV (http://meta-sort/webdav/) ──► /files/
GET, PUT, DELETE, MKCOL supported
Environment variable: PLUGIN_WEBDAV_URL=http://meta-sort-dev/webdav
File Path References
All file paths stored in the KV database are relative to FILES_VOLUME:
KV Key: /file/{midhash256}/filePath
KV Value: "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
Full path construction:
Container: /files + /media1/Movies/... = /files/media1/Movies/Inception (2010)/Inception.mkv
Any host: $FILES_VOLUME + /media1/Movies/... = $FILES_VOLUME/media1/Movies/Inception (2010)/Inception.mkv
This allows services to be deployed on different hosts while sharing the same volume. The FILES_VOLUME can be:
- A local folder bind mount
- An NFS share mounted on the host
- An SMB/CIFS share
- A FUSE mount (rclone, sshfs, etc.)
Mount Management
meta-sort manages remote mounts inside the FILES_VOLUME for shared access:
# meta-sort mounts remote storage INTO /files
/files/smb/nas-share ← SMB mount managed by meta-sort
/files/nfs/server ← NFS mount managed by meta-sort
/files/gdrive/ ← rclone mount managed by meta-sort
# Other folders are bind mounts from host
/files/media1 ← Host folder mounted into container
/files/media2 ← Another host folder
All services access files through /files/*, regardless of whether the underlying storage is local, NFS, SMB, or cloud.
Leader Election and Lock Mechanism
How It Works
MetaMesh uses a filesystem-based distributed consensus mechanism using flock(2). The shared filesystem acts as the consensus layer.
┌─────────────────────────────────────────────────────────────────────────────┐
│ Leader Election via flock │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Step 1: Service Startup │
│ ┌─────────────┐ │
│ │ Service │ ──▶ Try to acquire flock on /meta-core/locks/kv-leader.lock│
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ flock() Result │ │
│ │ ┌─────────────────────┐ ┌──────────────────────────────────┐ │ │
│ │ │ Lock Acquired │ │ Lock Blocked │ │ │
│ │ │ ────────────── │ │ ───────────── │ │ │
│ │ │ │ │ │ │ │
│ │ │ BECOME LEADER │ │ BECOME FOLLOWER │ │ │
│ │ │ 1. Spawn Redis │ │ 1. Read lock file │ │ │
│ │ │ 2. Write API │ │ 2. Extract leader API │ │ │
│ │ │ endpoint to │ │ 3. Connect to leader's │ │ │
│ │ │ lock file │ │ Redis │ │ │
│ │ │ 3. Accept clients │ │ 4. Start reconnect loop │ │ │
│ │ └─────────────────────┘ └──────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Lock File Format
The lock file at /meta-core/locks/kv-leader.lock contains JSON:
{
"host": "meta-sort-instance-1",
"api": "redis://10.0.1.50:6379",
"http": "http://10.0.1.50:3000",
"timestamp": 1703808000000,
"pid": 12345
}
flock Implementation
import { open, flock } from 'fs';
import { promisify } from 'util';
const LOCK_FILE = '/meta-core/locks/kv-leader.lock';
async function tryBecomeLeader(): Promise<boolean> {
const fd = await promisify(open)(LOCK_FILE, 'w');
try {
// LOCK_EX = exclusive lock, LOCK_NB = non-blocking
await promisify(flock)(fd, 'exnb'); // Throws if cannot acquire
// We are the leader - spawn KV database and write our endpoint
await spawnRedis();
await fs.writeFile(LOCK_FILE, JSON.stringify({
host: hostname(),
api: `redis://${myIP}:6379`,
http: `http://${myIP}:3000`,
timestamp: Date.now(),
pid: process.pid
}));
return true; // We are leader
} catch (err) {
if (err.code === 'EAGAIN' || err.code === 'EWOULDBLOCK') {
// Lock held by another process - become follower
return false;
}
throw err;
}
}
Failure Detection and Recovery
┌──────────────────────────────────────────────────────────────────────────┐
│ Failure Detection Loop │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Follower Service: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ while (true) { │ │
│ │ try { │ │
│ │ await redis.ping(); // Health check │ │
│ │ await sleep(5000); // Wait 5 seconds │ │
│ │ } catch (connectionError) { │ │
│ │ // Leader might be down │ │
│ │ await tryBecomeLeader(); // Attempt to take over │ │
│ │ if (isLeader) break; │ │
│ │ await reconnectToNewLeader(); // Or connect to new leader │ │
│ │ } │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ When Leader Dies: │
│ 1. flock is automatically released by OS │
│ 2. First follower to call flock() wins │
│ 3. New leader spawns Redis, writes new endpoint │
│ 4. Other followers detect connection failure, read new endpoint │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Service Discovery
Registration Mechanism
Each service registers itself in /meta-core/services/{service-name}.json:
{
"name": "meta-sort",
"version": "1.0.0",
"api": "http://10.0.1.50:3000",
"status": "running",
"pid": 12345,
"hostname": "meta-sort-instance-1",
"startedAt": "2024-12-28T10:00:00Z",
"lastHeartbeat": "2024-12-28T12:30:00Z",
"capabilities": ["write", "mount", "monitor"],
"endpoints": {
"health": "/health",
"api": "/api",
"metrics": "/api/metrics"
}
}
Service Lifecycle
┌──────────────────────────────────────────────────────────────────────────┐
│ Service Lifecycle │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Startup: │
│ 1. Create/update /meta-core/services/{name}.json │
│ 2. Set status = "starting" │
│ 3. Initialize (connect to KV, etc.) │
│ 4. Set status = "running" │
│ 5. Start heartbeat loop (update lastHeartbeat every 30s) │
│ │
│ Running: │
│ - Heartbeat updates lastHeartbeat timestamp │
│ - Other services read this file to discover endpoints │
│ - stale detection: if (now - lastHeartbeat > 60s) → service dead │
│ │
│ Shutdown: │
│ 1. Set status = "stopping" │
│ 2. Graceful shutdown (finish current work) │
│ 3. Delete /meta-core/services/{name}.json OR set status = "stopped" │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Discovering Other Services
async function discoverService(name: string): Promise<ServiceInfo | null> {
const servicePath = `/meta-core/services/${name}.json`;
try {
const content = await fs.readFile(servicePath, 'utf-8');
const service = JSON.parse(content);
// Check if service is alive (heartbeat within last 60 seconds)
const lastHeartbeat = new Date(service.lastHeartbeat).getTime();
const isAlive = Date.now() - lastHeartbeat < 60_000;
if (!isAlive) {
console.warn(`Service ${name} appears stale (last heartbeat: ${service.lastHeartbeat})`);
return null;
}
return service;
} catch (err) {
if (err.code === 'ENOENT') {
return null; // Service not registered
}
throw err;
}
}
MetaMesh Ecosystem Architecture
Service Hierarchy
┌─────────────────────────────────────────────────────────────────────────────┐
│ MetaMesh Ecosystem │
│ │
│ Shared Volumes: │
│ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐ │
│ │ META_CORE_VOLUME │ │ FILES_VOLUME │ │
│ │ (/meta-core) │ │ (/files) │ │
│ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │
│ │ │ db/ │ │ locks/ │ │svc/ │ │ │ │media1/ │ │ smb/ │ │ nfs/ │ │ │
│ │ │(Redis) │ │(flock) │ │(JSON)│ │ │ │(local) │ │(mount) │ │(mnt) │ │ │
│ │ └────┬───┘ └───┬────┘ └──┬───┘ │ │ └───┬────┘ └───┬────┘ └──┬───┘ │ │
│ └───────┼─────────┼─────────┼─────┘ └──────┼──────────┼─────────┼─────┘ │
│ │ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐│
│ │ ││
│ │ [1] meta-sort PROCESS-WRITE ││
│ │ ├── Watches /files/* for new media files ││
│ │ ├── Extracts metadata → writes to KV (Redis in /meta-core/db) ││
│ │ ├── Manages remote mounts INTO /files (NFS, SMB, rclone) ││
│ │ └── Leader election (spawns Redis if leader) ││
│ │ ││
│ │ [2] meta-fuse PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Builds virtual filesystem from metadata ││
│ │ ├── Serves FUSE mount + WebDAV ││
│ │ └── File paths resolve to /files/* ││
│ │ ││
│ │ [3] meta-stremio PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Stremio addon server (manifest, catalog, streams) ││
│ │ ├── HLS transcoding via FFmpeg ││
│ │ └── Video files from /files/* ││
│ │ ││
│ │ [4] meta-orbit SHARING-READ-WRITE ││
│ │ ├── P2P metadata synchronization (libp2p + OrbitDB) ││
│ │ ├── Decentralized search across network ││
│ │ ├── Reads from local KV, shares to network ││
│ │ └── Receives from network, writes to local KV ││
│ │ ││
│ └───────────────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Data Flow
┌─────────────────────────────────────────────────────────────────────────────┐
│ Data Flow │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ File Discovery: │
│ │
│ /files/media1/Movies/Inception (2010)/Inception.mkv │
│ │ │
│ ▼ │
│ [meta-sort] ──▶ Extract metadata ──▶ Write to Redis (in /meta-core/db) │
│ │ │ │
│ │ Key: file:{midhash256} │ │
│ │ filePath: "media1/Movies/Inception (2010)/Inception.mkv" │
│ │ title: "Inception" │
│ │ year: "2010" │
│ │ duration: 8880 │
│ │ │ │
│ ▼ ▼ │
│ │
│ File Access: │
│ │
│ [meta-fuse] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Virtual FS: /mnt/virtual/Movies/Inception (2010)/Inception.mkv │
│ │ ↓ │
│ │ symlink to /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-stremio] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Stream: GET /stremio/stream/{midhash256}/playlist.m3u8 │
│ │ ↓ │
│ │ transcode /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-orbit] ◀──▶ Sync with P2P network │
│ │ │
│ ▼ │
│ Share metadata to peers / Receive metadata from peers │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Deployment
Docker Compose (Production)
version: '3.8'
services:
meta-sort:
image: metazla/meta-sort:latest
volumes:
# Infrastructure volume (KV DB, locks, services)
- /DATA/Apps/meta-core:/meta-core:rw
# Shared media volume
- /DATA/Apps/meta-core/files:/files:rw,shared
# OR mount multiple sources into /files
# - /mnt/nas/media:/files/nas:ro
# - /mnt/local/media:/files/local:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
- WATCH_PATHS=/files
- REDIS_DATA_DIR=/meta-core/db/redis
privileged: true # Required for rclone FUSE mounts
meta-fuse:
image: metazla/meta-fuse:latest
volumes:
# Infrastructure (read-only for followers)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
# FUSE mount output
- /mnt/metamesh:/mnt/metamesh:rw,shared
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
privileged: true # Required for FUSE
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse
meta-stremio:
image: metazla/meta-stremio:latest
volumes:
# Infrastructure (read-only)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
ports:
- "7000:7000"
meta-orbit:
image: metazla/meta-orbit:latest
volumes:
# Infrastructure (read-write for P2P sync)
- /DATA/Apps/meta-core:/meta-core:rw
environment:
- META_CORE_PATH=/meta-core
ports:
- "4001:4001" # libp2p
Environment Variables
| Variable | Default | Description |
|---|---|---|
META_CORE_PATH |
/meta-core |
Path to META_CORE_VOLUME (infrastructure) |
FILES_PATH |
/files |
Path to FILES_VOLUME (shared media) |
REDIS_URL |
Auto-discover | Redis connection (from leader lock file) |
WATCH_FOLDER_LIST |
$FILES_PATH |
Comma-separated folders to watch for media |
SERVICE_NAME |
Container hostname | Unique service identifier |
BASE_URL |
http://localhost:8180 |
External URL for this service |
Container Plugins:
| Variable | Default | Description |
|---|---|---|
CONTAINER_PLUGINS_CONFIG |
/app/plugins.yml |
Path to plugins configuration |
DOCKER_SOCKET_PATH |
/var/run/docker.sock |
Docker socket for container management |
CONTAINER_CALLBACK_URL |
http://meta-sort:80 |
URL plugins use to call back |
CONTAINER_NETWORK |
meta-network |
Docker network for plugin containers |
PLUGIN_WEBDAV_URL |
- | WebDAV URL for plugin file access (e.g., http://meta-sort-dev/webdav) |
PLUGIN_STACK_NAME |
- | Docker Compose project name for grouping containers |
File Processing Pipeline
meta-sort uses a plugin-based architecture with a two-queue task scheduler:
┌──────────────┐ ┌────────────────────────────────────────────────────┐
│ Discovery │ │ Task Scheduler │
│ (Stream) │───▶│ ┌─────────────────┐ ┌──────────────────────────┐ │
└──────────────┘ │ │ Fast Queue │ │ Background Queue │ │
Watch folder │ │ (16 workers) │ │ (4 workers) │ │
+ manual scan │ │ <1s tasks │ │ >=1s tasks │ │
│ └─────────────────┘ └──────────────────────────┘ │
└────────────────────────────────────────────────────┘
Tasks are automatically classified and scheduled based on plugin dependencies:
Discovery → file-info → ffmpeg → filename-parser → jellyfin-nfo → tmdb
↓
anime-detector ← language ← subtitle ← torrent ← full-hash
Plugin System
Plugins are the core processing units that extract and enrich metadata. Each plugin:
- Has a
manifest.ymldefining metadata schema, config, and dependencies - Runs in either fast (high concurrency) or background (low concurrency) queue
- Can depend on other plugins to ensure correct execution order
Container Plugins
All plugins run as Docker containers for isolation and flexibility. Plugins are configured in plugins.yml and communicate via HTTP API.
| Plugin | Language | Queue | Priority | Description |
|---|---|---|---|---|
| file-info | TypeScript | fast | 10 | Basic file info (type, MIME, size, midhash256) |
| ffmpeg | TypeScript | fast | 15 | Video/audio/subtitle stream metadata via FFprobe |
| filename-parser | TypeScript | fast | 20 | Parse title, season, episode, year, quality from filename |
| jellyfin-nfo | TypeScript | fast | 25 | Parse Jellyfin/Kodi NFO metadata files |
| tmdb | TypeScript | background | 30 | Fetch metadata from The Movie Database API |
| anime-detector | TypeScript | fast | 35 | Detect anime content from keywords and audio tracks |
| language | TypeScript | fast | 40 | Aggregate languages from all streams (ISO 639-3) |
| subtitle | TypeScript | fast | 45 | Find sibling subtitle files and detect language |
| torrent | TypeScript | fast | 50 | Parse .torrent file metadata |
| full-hash | TypeScript | background | 100 | Compute SHA-256/SHA-1/MD5 hashes (JS implementation) |
| fast-full-hash | Rust | background | 100 | High-performance multi-algorithm hashing (Rust/Axum) |
Plugin File Access
Plugins access files via WebDAV served by meta-sort nginx. This allows plugins to:
- Access files without requiring direct volume mounts
- See SMB/rclone mounts that are mounted dynamically inside meta-sort
- Read and write to the plugin output directory at
/files/plugin/
Plugin Manifest Example
id: ffmpeg
name: FFmpeg Metadata
version: 1.0.0
description: Extracts video/audio/subtitle stream metadata using FFprobe
priority: 15
color: "#4CAF50"
defaultQueue: fast
dependencies:
- file-info
config: {}
schema:
fileinfo/duration:
label: Duration
type: string
readonly: true
fileinfo/streamdetails/video:
label: Video Streams
type: json
readonly: true
pattern: "fileinfo/streamdetails/video/{n}/*"
Task Queue Architecture
- Fast Queue (16 workers): High-throughput for quick operations (<1s)
- File info extraction, FFprobe, filename parsing, NFO reading
- Background Queue (4 workers): Low-concurrency for slow operations (>=1s)
- TMDB API calls, full SHA-256 hashing
- Automatically pauses when fast queue has work, resumes when idle
Plugin Management
Plugins can be enabled/disabled and configured via:
- Dashboard: http://localhost:8180/ → Plugins tab
- Config file:
plugins.jsonin cache directory
Processing States
- Pending - File discovered in watch folder
- Processing - Plugins running (fast queue first, then background)
- Done - All plugins completed successfully
Package Structure
meta-sort/
├── packages/
│ ├── meta-sort-core/ # Core processing service (@meta-sort/core)
│ │ ├── plugins/ # Built-in plugins
│ │ │ ├── file-info/ # Basic file information
│ │ │ ├── ffmpeg/ # FFprobe metadata extraction
│ │ │ ├── filename-parser/# Filename parsing
│ │ │ ├── jellyfin-nfo/ # NFO file parsing
│ │ │ ├── tmdb/ # TMDB API integration
│ │ │ ├── anime-detector/ # Anime content detection
│ │ │ ├── language/ # Language aggregation
│ │ │ ├── subtitle/ # Subtitle file finder
│ │ │ ├── torrent/ # Torrent file parsing
│ │ │ └── full-hash/ # SHA-256 full file hashing
│ │ ├── src/
│ │ │ ├── api/ # Fastify REST API (UnifiedAPIServer)
│ │ │ ├── config/ # Environment configuration
│ │ │ ├── jellyfin/ # Jellyfin integration
│ │ │ ├── kv/ # KV manager with leader election (Redis)
│ │ │ ├── logic/ # File processing logic
│ │ │ ├── metrics/ # Performance tracking
│ │ │ ├── plugin-engine/ # Plugin loader and task scheduler
│ │ │ ├── types/ # TypeScript types
│ │ │ ├── utils/ # Utility functions
│ │ │ └── index.ts # Entry point
│ │ └── package.json
│ │
│ ├── meta-sort-ui/ # Monitoring dashboard (@meta-sort/ui)
│ │ └── ... # React/Vite app
│ │
│ └── async-utils/ # Async utility library
│ └── ... # MultiQueue, PromiseQueue, etc.
│
├── docker/
│ ├── nginx.conf # Reverse proxy config
│ ├── redis.conf # Redis configuration
│ └── supervisord.conf # Process management
│
├── Dockerfile
├── docker-compose.yml
├── package.json # Workspace root
├── pnpm-workspace.yaml
└── README.md
Key Components (meta-sort-core)
| Component | File | Purpose |
|---|---|---|
| Entry Point | src/index.ts |
Initializes KV manager, plugin engine, API server, file watching |
| File Processor | src/logic/WatchedFileProcessor.ts |
File discovery and plugin task scheduling |
| Plugin Loader | src/plugin-engine/PluginLoader.ts |
Loads plugins from manifest.yml, manages plugin lifecycle |
| Task Scheduler | src/plugin-engine/TaskScheduler.ts |
Two-queue scheduler (fast/background) with dependency resolution |
| KV Manager | src/kv/KVManager.ts |
Leader election via flock, Redis spawning, service discovery |
| Unified API | src/api/UnifiedAPIServer.ts |
Fastify REST API for monitoring and control |
| Performance Metrics | src/metrics/PerformanceMetrics.ts |
Tracks processing throughput and timing |
| Config | src/config/EnvConfig.ts |
Environment variable parsing |
Core Features
1. Folder Watching
- Real-time monitoring using chokidar for file system events
- Streaming discovery via async generators for efficient memory usage
- Configurable watch paths supporting local and remote mounted directories
- File type filtering (video, audio, subtitle, torrent files)
2. Plugin-Based Metadata Extraction
Metadata extraction is handled by plugins that can be enabled/disabled and configured:
- file-info - Basic file info, MIME type, midhash256 content identifier
- ffmpeg - Duration, codec, resolution, bitrate, audio tracks, subtitles via FFprobe
- filename-parser - Extract title, year, season, episode, quality from filenames
- jellyfin-nfo - Parse NFO sidecar files (Jellyfin/Kodi format)
- tmdb - Fetch rich metadata from The Movie Database API
- anime-detector - Detect anime content from keywords and audio tracks
- language - Aggregate languages from all streams (ISO 639-3)
- subtitle - Find sibling subtitle files and detect language
- torrent - Parse .torrent file metadata
- full-hash - Compute SHA-256 hash of entire file (background)
3. KV Storage
- Nested key architecture for property-level granularity
- Leader election via flock for high availability
- Service discovery via shared filesystem
- CRDT-compatible for distributed synchronization
Key Structure:
/file/{midhash256}/title → "Inception"
/file/{midhash256}/year → "2010"
/file/{midhash256}/video/codec → "h265"
/file/{midhash256}/audio/0/language → "en"
/file/{midhash256}/filePath → "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
4. Remote Mount Management
- NFS shares - Network file systems
- SMB/CIFS - Windows network shares with credentials
- rclone - Cloud storage (Google Drive, Dropbox, S3, etc.)
- Automatic mounting on startup
- Mounts created inside FILES_VOLUME for shared access by all services
Mounts are created inside /files/ (FILES_VOLUME) so all services can access them:
# Example mount structure
/files/ # FILES_VOLUME
├── media1/ # Local folder bind mount from host
├── media2/ # Another local folder
├── smb/ # SMB/CIFS remote mounts
│ └── nas-share/ # SMB: //server/share
├── nfs/ # NFS remote mounts
│ └── server-media/ # NFS: nas.local:/volume1/media
└── gdrive/ # rclone: gdrive:Media
Configuration
Environment Variables
# Volume paths
META_CORE_PATH=/meta-core # Infrastructure volume
FILES_PATH=/files # Shared media volume
# Watch paths (comma-separated, relative to FILES_PATH or absolute)
WATCH_PATHS=/files
# Processing configuration
MAX_WORKER_THREADS=16 # Defaults to CPU count
LIGHT_PROCESSING_TIMEOUT_MS=120000
HASH_PROCESSING_TIMEOUT_MS=7200000
# Remote mount configuration
REMOTE_STORAGE_CONFIG=/meta-core/config/remote-storage.conf
Remote Storage Configuration
Create /meta-core/config/remote-storage.conf:
# NFS mount (into FILES_VOLUME)
nfs://nas.local:/volume1/media /files/nfs/nas-media nfs defaults,rw 0 0
# SMB mount (credentials in separate file)
//server/share /files/smb/nas-share cifs credentials=/meta-core/config/smb-creds.txt,uid=1000 0 0
# rclone mount
gdrive: /files/gdrive rclone vfs-cache-mode=full 0 0
API
REST Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/health |
GET | Service health check |
/api/scan/trigger |
POST | Trigger manual folder scan |
/api/processing/status |
GET | Current pipeline state |
/api/metrics |
GET | Performance metrics |
/api/mounts |
GET | List mounted remote storage |
/api/mounts |
POST | Add new remote mount |
/api/mounts/{id} |
DELETE | Remove remote mount |
Processing Status Response
{
"totalPending": 150,
"totalLightProcessing": 16,
"totalHashProcessing": 8,
"totalDone": 45000,
"totalFilesInVFS": 45150,
"throughput": {
"filesPerSecond": 34,
"avgLightProcessingMs": 470,
"avgHashProcessingMs": 15000
}
}
Usage
Development
# Install dependencies
pnpm install
# Build all packages
pnpm run build
# Run in development mode
pnpm run dev
Docker
# Build and run with Docker Compose
docker-compose up -d
# Check health
curl http://localhost/health
curl http://localhost/api/processing/status
License
MIT License - see LICENSE file for details.