meta-sort
A standalone file sorting and metadata extraction service for the MetaMesh ecosystem. meta-sort monitors watch folders, extracts metadata from media files, stores it in a distributed key-value store, and manages remote drive mounting for shared access.
Overview
meta-sort is a process-write service that serves as the primary data ingestion point for the MetaMesh platform. It:
- Watches folders for new media files (video, audio, subtitles)
- Extracts metadata using FFmpeg and filename parsing
- Computes content identifiers using midhash256 (fast) and SHA-256 (background)
- Stores metadata in a distributed KV store (Redis or etcd)
- Manages remote mounts (NFS, SMB, cloud storage) for shared access
Meta-Core Architecture
MetaMesh uses two shared volumes for operation:
1. META_CORE_VOLUME - Infrastructure
The infrastructure volume stores all service coordination data:
| Context | Path | Description |
|---|---|---|
| Inside Container | /meta-core |
Mounted shared volume |
| Production (Host) | /DATA/Apps/meta-core |
External shared drive mount point |
| Development | ./data/meta-core |
Local development directory |
META_CORE_VOLUME (/meta-core or /DATA/Apps/meta-core)
├── db/ # KV Database storage
│ ├── redis/ # Redis persistence (RDB + AOF)
│ │ ├── dump.rdb # Redis snapshot
│ │ └── appendonly.aof # Redis append-only log
│ └── etcd/ # etcd data (alternative)
│ └── member/
│
├── locks/ # Distributed locking
│ └── kv-leader.lock # Leader election lock file
│ # Contains: {"host":"service-name","api":"http://host:port"}
│
├── services/ # Service discovery registry
│ ├── meta-sort.json # {"name":"meta-sort","api":"http://...","status":"running","pid":123}
│ ├── meta-fuse.json # {"name":"meta-fuse","api":"http://...","status":"running","pid":456}
│ ├── meta-stremio.json # {"name":"meta-stremio","api":"http://...","status":"running","pid":789}
│ └── meta-orbit.json # {"name":"meta-orbit","api":"http://...","status":"running","pid":321}
│
└── config/ # Shared configuration
├── remote-storage.conf # Remote mount definitions
└── smb-credentials.txt # SMB credentials (chmod 600)
2. FILES_VOLUME - Shared Media Drive
The FILES_VOLUME (canonical name: /files) is the shared drive for all media files. This is where all actual media content resides, and file paths in the KV database reference this volume.
| Context | Path | Description |
|---|---|---|
| Inside Container | /files |
Mounted shared media volume |
| Production (Host) | Varies (NFS/SMB/local) | External shared drive |
| Development | ./data/files |
Local development directory |
FILES_VOLUME (/files)
├── media1/ # Local folder mounted from host
│ ├── Movies/
│ │ └── Inception (2010)/
│ │ └── Inception.mkv
│ └── TV Shows/
│ └── Breaking Bad/
│ └── Season 01/
│
├── media2/ # Another local media folder
│ └── Anime/
│ └── Steins Gate/
│
├── smb/ # SMB/CIFS remote mount
│ └── NAS-Media/
│ └── Movies/
│
├── nfs/ # NFS remote mount
│ └── server-media/
│
└── gdrive/ # rclone remote mount (Google Drive)
└── Cloud-Media/
File Path References
All file paths stored in the KV database are relative to FILES_VOLUME:
KV Key: /file/{midhash256}/filePath
KV Value: "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
Full path construction:
Container: /files + /media1/Movies/... = /files/media1/Movies/Inception (2010)/Inception.mkv
Any host: $FILES_VOLUME + /media1/Movies/... = $FILES_VOLUME/media1/Movies/Inception (2010)/Inception.mkv
This allows services to be deployed on different hosts while sharing the same volume. The FILES_VOLUME can be:
- A local folder bind mount
- An NFS share mounted on the host
- An SMB/CIFS share
- A FUSE mount (rclone, sshfs, etc.)
Mount Management
meta-sort manages remote mounts inside the FILES_VOLUME for shared access:
# meta-sort mounts remote storage INTO /files
/files/smb/nas-share ← SMB mount managed by meta-sort
/files/nfs/server ← NFS mount managed by meta-sort
/files/gdrive/ ← rclone mount managed by meta-sort
# Other folders are bind mounts from host
/files/media1 ← Host folder mounted into container
/files/media2 ← Another host folder
All services access files through /files/*, regardless of whether the underlying storage is local, NFS, SMB, or cloud.
Leader Election and Lock Mechanism
How It Works
MetaMesh uses a filesystem-based distributed consensus mechanism using flock(2). The shared filesystem acts as the consensus layer.
┌─────────────────────────────────────────────────────────────────────────────┐
│ Leader Election via flock │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Step 1: Service Startup │
│ ┌─────────────┐ │
│ │ Service │ ──▶ Try to acquire flock on /meta-core/locks/kv-leader.lock│
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ flock() Result │ │
│ │ ┌─────────────────────┐ ┌──────────────────────────────────┐ │ │
│ │ │ Lock Acquired │ │ Lock Blocked │ │ │
│ │ │ ────────────── │ │ ───────────── │ │ │
│ │ │ │ │ │ │ │
│ │ │ BECOME LEADER │ │ BECOME FOLLOWER │ │ │
│ │ │ 1. Spawn Redis │ │ 1. Read lock file │ │ │
│ │ │ 2. Write API │ │ 2. Extract leader API │ │ │
│ │ │ endpoint to │ │ 3. Connect to leader's │ │ │
│ │ │ lock file │ │ Redis │ │ │
│ │ │ 3. Accept clients │ │ 4. Start reconnect loop │ │ │
│ │ └─────────────────────┘ └──────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Lock File Format
The lock file at /meta-core/locks/kv-leader.lock contains JSON:
{
"host": "meta-sort-instance-1",
"api": "redis://10.0.1.50:6379",
"http": "http://10.0.1.50:3000",
"timestamp": 1703808000000,
"pid": 12345
}
flock Implementation
import { open, flock } from 'fs';
import { promisify } from 'util';
const LOCK_FILE = '/meta-core/locks/kv-leader.lock';
async function tryBecomeLeader(): Promise<boolean> {
const fd = await promisify(open)(LOCK_FILE, 'w');
try {
// LOCK_EX = exclusive lock, LOCK_NB = non-blocking
await promisify(flock)(fd, 'exnb'); // Throws if cannot acquire
// We are the leader - spawn KV database and write our endpoint
await spawnRedis();
await fs.writeFile(LOCK_FILE, JSON.stringify({
host: hostname(),
api: `redis://${myIP}:6379`,
http: `http://${myIP}:3000`,
timestamp: Date.now(),
pid: process.pid
}));
return true; // We are leader
} catch (err) {
if (err.code === 'EAGAIN' || err.code === 'EWOULDBLOCK') {
// Lock held by another process - become follower
return false;
}
throw err;
}
}
Failure Detection and Recovery
┌──────────────────────────────────────────────────────────────────────────┐
│ Failure Detection Loop │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Follower Service: │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ while (true) { │ │
│ │ try { │ │
│ │ await redis.ping(); // Health check │ │
│ │ await sleep(5000); // Wait 5 seconds │ │
│ │ } catch (connectionError) { │ │
│ │ // Leader might be down │ │
│ │ await tryBecomeLeader(); // Attempt to take over │ │
│ │ if (isLeader) break; │ │
│ │ await reconnectToNewLeader(); // Or connect to new leader │ │
│ │ } │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ When Leader Dies: │
│ 1. flock is automatically released by OS │
│ 2. First follower to call flock() wins │
│ 3. New leader spawns Redis, writes new endpoint │
│ 4. Other followers detect connection failure, read new endpoint │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Service Discovery
Registration Mechanism
Each service registers itself in /meta-core/services/{service-name}.json:
{
"name": "meta-sort",
"version": "1.0.0",
"api": "http://10.0.1.50:3000",
"status": "running",
"pid": 12345,
"hostname": "meta-sort-instance-1",
"startedAt": "2024-12-28T10:00:00Z",
"lastHeartbeat": "2024-12-28T12:30:00Z",
"capabilities": ["write", "mount", "monitor"],
"endpoints": {
"health": "/health",
"api": "/api",
"metrics": "/api/metrics"
}
}
Service Lifecycle
┌──────────────────────────────────────────────────────────────────────────┐
│ Service Lifecycle │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Startup: │
│ 1. Create/update /meta-core/services/{name}.json │
│ 2. Set status = "starting" │
│ 3. Initialize (connect to KV, etc.) │
│ 4. Set status = "running" │
│ 5. Start heartbeat loop (update lastHeartbeat every 30s) │
│ │
│ Running: │
│ - Heartbeat updates lastHeartbeat timestamp │
│ - Other services read this file to discover endpoints │
│ - stale detection: if (now - lastHeartbeat > 60s) → service dead │
│ │
│ Shutdown: │
│ 1. Set status = "stopping" │
│ 2. Graceful shutdown (finish current work) │
│ 3. Delete /meta-core/services/{name}.json OR set status = "stopped" │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Discovering Other Services
async function discoverService(name: string): Promise<ServiceInfo | null> {
const servicePath = `/meta-core/services/${name}.json`;
try {
const content = await fs.readFile(servicePath, 'utf-8');
const service = JSON.parse(content);
// Check if service is alive (heartbeat within last 60 seconds)
const lastHeartbeat = new Date(service.lastHeartbeat).getTime();
const isAlive = Date.now() - lastHeartbeat < 60_000;
if (!isAlive) {
console.warn(`Service ${name} appears stale (last heartbeat: ${service.lastHeartbeat})`);
return null;
}
return service;
} catch (err) {
if (err.code === 'ENOENT') {
return null; // Service not registered
}
throw err;
}
}
MetaMesh Ecosystem Architecture
Service Hierarchy
┌─────────────────────────────────────────────────────────────────────────────┐
│ MetaMesh Ecosystem │
│ │
│ Shared Volumes: │
│ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐ │
│ │ META_CORE_VOLUME │ │ FILES_VOLUME │ │
│ │ (/meta-core) │ │ (/files) │ │
│ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │ ┌────────┐ ┌────────┐ ┌──────┐ │ │
│ │ │ db/ │ │ locks/ │ │svc/ │ │ │ │media1/ │ │ smb/ │ │ nfs/ │ │ │
│ │ │(Redis) │ │(flock) │ │(JSON)│ │ │ │(local) │ │(mount) │ │(mnt) │ │ │
│ │ └────┬───┘ └───┬────┘ └──┬───┘ │ │ └───┬────┘ └───┬────┘ └──┬───┘ │ │
│ └───────┼─────────┼─────────┼─────┘ └──────┼──────────┼─────────┼─────┘ │
│ │ │ │ │ │ │ │
│ ▼ ▼ ▼ ▼ ▼ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐│
│ │ ││
│ │ [1] meta-sort PROCESS-WRITE ││
│ │ ├── Watches /files/* for new media files ││
│ │ ├── Extracts metadata → writes to KV (Redis in /meta-core/db) ││
│ │ ├── Manages remote mounts INTO /files (NFS, SMB, rclone) ││
│ │ ├── Leader election (spawns Redis if leader) ││
│ │ └── Serves monitoring UI (meta-sort-editor) ││
│ │ ││
│ │ [2] meta-fuse PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Builds virtual filesystem from metadata ││
│ │ ├── Serves FUSE mount + WebDAV ││
│ │ └── File paths resolve to /files/* ││
│ │ ││
│ │ [3] meta-stremio PROCESS-READ ││
│ │ ├── Reads metadata from KV (Redis) ││
│ │ ├── Stremio addon server (manifest, catalog, streams) ││
│ │ ├── HLS transcoding via FFmpeg ││
│ │ └── Video files from /files/* ││
│ │ ││
│ │ [4] meta-orbit SHARING-READ-WRITE ││
│ │ ├── P2P metadata synchronization (libp2p + OrbitDB) ││
│ │ ├── Decentralized search across network ││
│ │ ├── Reads from local KV, shares to network ││
│ │ └── Receives from network, writes to local KV ││
│ │ ││
│ └───────────────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Data Flow
┌─────────────────────────────────────────────────────────────────────────────┐
│ Data Flow │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ File Discovery: │
│ │
│ /files/media1/Movies/Inception (2010)/Inception.mkv │
│ │ │
│ ▼ │
│ [meta-sort] ──▶ Extract metadata ──▶ Write to Redis (in /meta-core/db) │
│ │ │ │
│ │ Key: file:{midhash256} │ │
│ │ filePath: "media1/Movies/Inception (2010)/Inception.mkv" │
│ │ title: "Inception" │
│ │ year: "2010" │
│ │ duration: 8880 │
│ │ │ │
│ ▼ ▼ │
│ │
│ File Access: │
│ │
│ [meta-fuse] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Virtual FS: /mnt/virtual/Movies/Inception (2010)/Inception.mkv │
│ │ ↓ │
│ │ symlink to /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-stremio] ◀── Read from Redis │
│ │ │
│ ▼ │
│ Stream: GET /stremio/stream/{midhash256}/playlist.m3u8 │
│ │ ↓ │
│ │ transcode /files/media1/Movies/Inception (2010)/... │
│ │ │
│ [meta-orbit] ◀──▶ Sync with P2P network │
│ │ │
│ ▼ │
│ Share metadata to peers / Receive metadata from peers │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Deployment
Docker Compose (Production)
version: '3.8'
services:
meta-sort:
image: metazla/meta-sort:latest
volumes:
# Infrastructure volume (KV DB, locks, services)
- /DATA/Apps/meta-core:/meta-core:rw
# Shared media volume
- /DATA/Apps/meta-core/files:/files:rw,shared
# OR mount multiple sources into /files
# - /mnt/nas/media:/files/nas:ro
# - /mnt/local/media:/files/local:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
- WATCH_PATHS=/files
- REDIS_DATA_DIR=/meta-core/db/redis
privileged: true # Required for rclone FUSE mounts
meta-fuse:
image: metazla/meta-fuse:latest
volumes:
# Infrastructure (read-only for followers)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
# FUSE mount output
- /mnt/metamesh:/mnt/metamesh:rw,shared
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
privileged: true # Required for FUSE
cap_add:
- SYS_ADMIN
devices:
- /dev/fuse
meta-stremio:
image: metazla/meta-stremio:latest
volumes:
# Infrastructure (read-only)
- /DATA/Apps/meta-core:/meta-core:ro
# Shared media volume (read-only)
- /DATA/Apps/meta-core/files:/files:ro
environment:
- META_CORE_PATH=/meta-core
- FILES_PATH=/files
ports:
- "7000:7000"
meta-orbit:
image: metazla/meta-orbit:latest
volumes:
# Infrastructure (read-write for P2P sync)
- /DATA/Apps/meta-core:/meta-core:rw
environment:
- META_CORE_PATH=/meta-core
ports:
- "4001:4001" # libp2p
Environment Variables
| Variable | Default | Description |
|---|---|---|
META_CORE_PATH |
/meta-core |
Path to META_CORE_VOLUME (infrastructure) |
FILES_PATH |
/files |
Path to FILES_VOLUME (shared media) |
REDIS_URL |
Auto-discover | Redis connection (from leader lock file) |
WATCH_PATHS |
$FILES_PATH |
Folders to watch for media |
SERVICE_NAME |
Container hostname | Unique service identifier |
File Processing Pipeline
meta-sort uses a plugin-based architecture with a two-queue task scheduler:
┌──────────────┐ ┌────────────────────────────────────────────────────┐
│ Discovery │ │ Task Scheduler │
│ (Stream) │───▶│ ┌─────────────────┐ ┌──────────────────────────┐ │
└──────────────┘ │ │ Fast Queue │ │ Background Queue │ │
Watch folder │ │ (16 workers) │ │ (4 workers) │ │
+ manual scan │ │ <1s tasks │ │ >=1s tasks │ │
│ └─────────────────┘ └──────────────────────────┘ │
└────────────────────────────────────────────────────┘
Tasks are automatically classified and scheduled based on plugin dependencies:
Discovery → file-info → ffmpeg → filename-parser → jellyfin-nfo → tmdb
↓
anime-detector ← language ← subtitle ← torrent ← full-hash
Plugin System
Plugins are the core processing units that extract and enrich metadata. Each plugin:
- Has a
manifest.ymldefining metadata schema, config, and dependencies - Runs in either fast (high concurrency) or background (low concurrency) queue
- Can depend on other plugins to ensure correct execution order
Built-in Plugins
| Plugin | Queue | Priority | Description |
|---|---|---|---|
| file-info | fast | 10 | Basic file info (type, MIME, size, midhash256) |
| ffmpeg | fast | 15 | Video/audio/subtitle stream metadata via FFprobe |
| filename-parser | fast | 20 | Parse title, season, episode, year, quality from filename |
| jellyfin-nfo | fast | 25 | Parse Jellyfin/Kodi NFO metadata files |
| tmdb | background | 30 | Fetch metadata from The Movie Database API |
| anime-detector | fast | 35 | Detect anime content from keywords and audio tracks |
| language | fast | 40 | Aggregate languages from all streams (ISO 639-3) |
| subtitle | fast | 45 | Find sibling subtitle files and detect language |
| torrent | fast | 50 | Parse .torrent file metadata |
| full-hash | background | 100 | Compute SHA-256 hash of entire file |
Plugin Manifest Example
id: ffmpeg
name: FFmpeg Metadata
version: 1.0.0
description: Extracts video/audio/subtitle stream metadata using FFprobe
priority: 15
color: "#4CAF50"
defaultQueue: fast
dependencies:
- file-info
config: {}
schema:
fileinfo/duration:
label: Duration
type: string
readonly: true
fileinfo/streamdetails/video:
label: Video Streams
type: json
readonly: true
pattern: "fileinfo/streamdetails/video/{n}/*"
Task Queue Architecture
- Fast Queue (16 workers): High-throughput for quick operations (<1s)
- File info extraction, FFprobe, filename parsing, NFO reading
- Background Queue (4 workers): Low-concurrency for slow operations (>=1s)
- TMDB API calls, full SHA-256 hashing
- Automatically pauses when fast queue has work, resumes when idle
Plugin Management
Plugins can be enabled/disabled and configured via:
- Dashboard: http://localhost:8180/ → Plugins tab
- Config file:
plugins.jsonin cache directory
Processing States
- Pending - File discovered in watch folder
- Processing - Plugins running (fast queue first, then background)
- Done - All plugins completed successfully
Package Structure
meta-sort/
├── packages/
│ ├── meta-sort-core/ # Core processing service (@meta-sort/core)
│ │ ├── plugins/ # Built-in plugins
│ │ │ ├── file-info/ # Basic file information
│ │ │ ├── ffmpeg/ # FFprobe metadata extraction
│ │ │ ├── filename-parser/# Filename parsing
│ │ │ ├── jellyfin-nfo/ # NFO file parsing
│ │ │ ├── tmdb/ # TMDB API integration
│ │ │ ├── anime-detector/ # Anime content detection
│ │ │ ├── language/ # Language aggregation
│ │ │ ├── subtitle/ # Subtitle file finder
│ │ │ ├── torrent/ # Torrent file parsing
│ │ │ └── full-hash/ # SHA-256 full file hashing
│ │ ├── src/
│ │ │ ├── api/ # Fastify REST API (UnifiedAPIServer)
│ │ │ ├── config/ # Environment configuration
│ │ │ ├── jellyfin/ # Jellyfin integration
│ │ │ ├── kv/ # KV manager with leader election (Redis)
│ │ │ ├── logic/ # File processing logic
│ │ │ ├── metrics/ # Performance tracking
│ │ │ ├── plugin-engine/ # Plugin loader and task scheduler
│ │ │ ├── types/ # TypeScript types
│ │ │ ├── utils/ # Utility functions
│ │ │ └── index.ts # Entry point
│ │ └── package.json
│ │
│ ├── meta-sort-editor/ # Metadata editor UI (@meta-sort/editor)
│ │ └── ... # React/Vite app
│ │
│ ├── meta-sort-ui/ # Monitoring dashboard (@meta-sort/ui)
│ │ └── ... # React/Vite app
│ │
│ └── async-utils/ # Async utility library
│ └── ... # MultiQueue, PromiseQueue, etc.
│
├── docker/
│ ├── nginx.conf # Reverse proxy config
│ ├── redis.conf # Redis configuration
│ └── supervisord.conf # Process management
│
├── Dockerfile
├── docker-compose.yml
├── package.json # Workspace root
├── pnpm-workspace.yaml
└── README.md
Key Components (meta-sort-core)
| Component | File | Purpose |
|---|---|---|
| Entry Point | src/index.ts |
Initializes KV manager, plugin engine, API server, file watching |
| File Processor | src/logic/WatchedFileProcessor.ts |
File discovery and plugin task scheduling |
| Plugin Loader | src/plugin-engine/PluginLoader.ts |
Loads plugins from manifest.yml, manages plugin lifecycle |
| Task Scheduler | src/plugin-engine/TaskScheduler.ts |
Two-queue scheduler (fast/background) with dependency resolution |
| KV Manager | src/kv/KVManager.ts |
Leader election via flock, Redis spawning, service discovery |
| Unified API | src/api/UnifiedAPIServer.ts |
Fastify REST API for monitoring and control |
| Performance Metrics | src/metrics/PerformanceMetrics.ts |
Tracks processing throughput and timing |
| Config | src/config/EnvConfig.ts |
Environment variable parsing |
Core Features
1. Folder Watching
- Real-time monitoring using chokidar for file system events
- Streaming discovery via async generators for efficient memory usage
- Configurable watch paths supporting local and remote mounted directories
- File type filtering (video, audio, subtitle, torrent files)
2. Plugin-Based Metadata Extraction
Metadata extraction is handled by plugins that can be enabled/disabled and configured:
- file-info - Basic file info, MIME type, midhash256 content identifier
- ffmpeg - Duration, codec, resolution, bitrate, audio tracks, subtitles via FFprobe
- filename-parser - Extract title, year, season, episode, quality from filenames
- jellyfin-nfo - Parse NFO sidecar files (Jellyfin/Kodi format)
- tmdb - Fetch rich metadata from The Movie Database API
- anime-detector - Detect anime content from keywords and audio tracks
- language - Aggregate languages from all streams (ISO 639-3)
- subtitle - Find sibling subtitle files and detect language
- torrent - Parse .torrent file metadata
- full-hash - Compute SHA-256 hash of entire file (background)
3. KV Storage
- Nested key architecture for property-level granularity
- Leader election via flock for high availability
- Service discovery via shared filesystem
- CRDT-compatible for distributed synchronization
Key Structure:
/file/{midhash256}/title → "Inception"
/file/{midhash256}/year → "2010"
/file/{midhash256}/video/codec → "h265"
/file/{midhash256}/audio/0/language → "en"
/file/{midhash256}/filePath → "media1/Movies/Inception (2010)/Inception.mkv"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Relative to FILES_VOLUME (/files)
4. Remote Mount Management
- NFS shares - Network file systems
- SMB/CIFS - Windows network shares with credentials
- rclone - Cloud storage (Google Drive, Dropbox, S3, etc.)
- Automatic mounting on startup
- Mounts created inside FILES_VOLUME for shared access by all services
Mounts are created inside /files/ (FILES_VOLUME) so all services can access them:
# Example mount structure
/files/ # FILES_VOLUME
├── media1/ # Local folder bind mount from host
├── media2/ # Another local folder
├── smb/ # SMB/CIFS remote mounts
│ └── nas-share/ # SMB: //server/share
├── nfs/ # NFS remote mounts
│ └── server-media/ # NFS: nas.local:/volume1/media
└── gdrive/ # rclone: gdrive:Media
Configuration
Environment Variables
# Volume paths
META_CORE_PATH=/meta-core # Infrastructure volume
FILES_PATH=/files # Shared media volume
# Watch paths (comma-separated, relative to FILES_PATH or absolute)
WATCH_PATHS=/files
# Processing configuration
MAX_WORKER_THREADS=16 # Defaults to CPU count
LIGHT_PROCESSING_TIMEOUT_MS=120000
HASH_PROCESSING_TIMEOUT_MS=7200000
# Remote mount configuration
REMOTE_STORAGE_CONFIG=/meta-core/config/remote-storage.conf
Remote Storage Configuration
Create /meta-core/config/remote-storage.conf:
# NFS mount (into FILES_VOLUME)
nfs://nas.local:/volume1/media /files/nfs/nas-media nfs defaults,rw 0 0
# SMB mount (credentials in separate file)
//server/share /files/smb/nas-share cifs credentials=/meta-core/config/smb-creds.txt,uid=1000 0 0
# rclone mount
gdrive: /files/gdrive rclone vfs-cache-mode=full 0 0
API
REST Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/health |
GET | Service health check |
/api/scan/trigger |
POST | Trigger manual folder scan |
/api/processing/status |
GET | Current pipeline state |
/api/metrics |
GET | Performance metrics |
/api/mounts |
GET | List mounted remote storage |
/api/mounts |
POST | Add new remote mount |
/api/mounts/{id} |
DELETE | Remove remote mount |
Processing Status Response
{
"totalPending": 150,
"totalLightProcessing": 16,
"totalHashProcessing": 8,
"totalDone": 45000,
"totalFilesInVFS": 45150,
"throughput": {
"filesPerSecond": 34,
"avgLightProcessingMs": 470,
"avgHashProcessingMs": 15000
}
}
Usage
Development
# Install dependencies
pnpm install
# Build all packages
pnpm run build
# Run in development mode
pnpm run dev
Docker
# Build and run with Docker Compose
docker-compose up -d
# Check health
curl http://localhost/health
curl http://localhost/api/processing/status
License
MIT License - see LICENSE file for details.