improve doc and bug fix

This commit is contained in:
Pierre
2026-05-25 23:48:42 +02:00
parent f77bb71b44
commit 31bb1245da
5 changed files with 600 additions and 2353 deletions
+342 -807
View File
File diff suppressed because it is too large Load Diff
+5 -5
View File
@@ -587,7 +587,7 @@ Deduplication logic remains identical. Fast-hash serves the same purpose as full
The Stremio addon retrieves files by metadata queries (title, season, episode), not by hash. The hash is used internally for metadata storage. Changing to fast-hash is transparent to the addon. The Stremio addon retrieves files by metadata queries (title, season, episode), not by hash. The hash is used internally for metadata storage. Changing to fast-hash is transparent to the addon.
### Meta-Orbit (Distributed Synchronization) ### meta-share (Distributed Synchronization)
**Impact:** Hash serves as content identifier for P2P synchronization. **Impact:** Hash serves as content identifier for P2P synchronization.
@@ -716,7 +716,7 @@ MetaMesh implements a two-phase processing pipeline:
**Phase 2: Hash Processing (Full Hashes - Background)** **Phase 2: Hash Processing (Full Hashes - Background)**
- Compute full-file hashes: SHA-256, SHA-1, MD5, CRC32 (slow, minutes) - Compute full-file hashes: SHA-256, SHA-1, MD5, CRC32 (slow, minutes)
- Update etcd with additional hash metadata - Update etcd with additional hash metadata
- Required for Meta-Orbit distributed synchronization - Required for meta-share distributed synchronization
### Why Both Are Needed ### Why Both Are Needed
@@ -726,7 +726,7 @@ MetaMesh implements a two-phase processing pipeline:
- **Fast deduplication**: Quickly identify potential duplicates - **Fast deduplication**: Quickly identify potential duplicates
**Full Hash Purpose:** **Full Hash Purpose:**
- **Meta-Orbit integration**: Distributed P2P synchronization requires cryptographically secure hashes - **meta-share integration**: Distributed P2P synchronization requires cryptographically secure hashes
- **External system compatibility**: Torrent verification, IPFS CIDs - **External system compatibility**: Torrent verification, IPFS CIDs
- **Collision resolution**: If midhash256 collision detected, full SHA-256 provides definitive answer - **Collision resolution**: If midhash256 collision detected, full SHA-256 provides definitive answer
@@ -744,7 +744,7 @@ Light Processing (32 workers, parallel)
Hash Processing (8 workers, background) Hash Processing (8 workers, background)
- Compute SHA-256, SHA-1, MD5, CRC32 (15+ minutes for 50GB) - Compute SHA-256, SHA-1, MD5, CRC32 (15+ minutes for 50GB)
- Update etcd with full hashes - Update etcd with full hashes
- Enable Meta-Orbit synchronization ✓ - Enable meta-share synchronization ✓
``` ```
### Collision Detection and Handling ### Collision Detection and Handling
@@ -778,4 +778,4 @@ The trade-offs—weaker cryptographic guarantees, potential collisions for adver
This architecture aligns with Meta-Mesh's design philosophy: **optimize for the common case** (large media files, organic content, personal libraries) while maintaining **practicality and performance** over theoretical perfection. This architecture aligns with Meta-Mesh's design philosophy: **optimize for the common case** (large media files, organic content, personal libraries) while maintaining **practicality and performance** over theoretical perfection.
Meta-Mesh leverages midhash256 for **immediate file availability** while computing full SHA-256 hashes **in the background** for Meta-Orbit distributed synchronization and external system compatibility. This dual-hash approach provides both speed and security. Meta-Mesh leverages midhash256 for **immediate file availability** while computing full SHA-256 hashes **in the background** for meta-share distributed synchronization and external system compatibility. This dual-hash approach provides both speed and security.
+27 -754
View File
@@ -1,762 +1,35 @@
# Meta-Sort Plugin System Architecture # Plugin System (Deprecated — See Container Plugins)
This document describes the plugin system architecture for meta-sort, enabling extensible metadata extraction through dynamically loaded plugins. > **This document is a historical pointer.** The in-process plugin system it
> originally described — `plugins/<id>/{manifest.yml,index.ts}` directories
> loaded via dynamic `import()` inside meta-sort-core — has been removed.
> All plugins now run as Docker containers and communicate over HTTP.
## Overview For the current design, read:
The plugin system provides a modular architecture for metadata extraction. Plugins are self-contained units that: - **[containerized-plugin-architecture.md](./containerized-plugin-architecture.md)** — full container plugin design (HTTP contract, lifecycle, callbacks, KV access).
- **[plugin-task-queue-architecture.md](./plugin-task-queue-architecture.md)** — how plugin work is dispatched across the fast / background queues.
- Root **`CLAUDE.md`** (`Plugin Development` and `Runtime Plugin Configuration` sections) — operational quickstart.
- Plugin sources at **`packages/plugins/metamesh-plugin-*`** (separate submodules) — `file-info`, `ffmpeg`, `filename-parser`, `jellyfin-nfo`, `tmdb`, `anime-detector`, `language`, `subtitle`, `subtitle-extractor`, `torrent`, `fullhash`.
- Are discovered and loaded from a plugins directory at startup ## What survived from the old design
- Can be activated/deactivated at runtime without restart
- Process files sequentially in dependency order
- Store metadata in a flat key-value format
- Have access to their own persistent cache folder
## Directory Structure The `Plugin` / `PluginContext` / `PluginLifecycleContext` / `PluginKVStore`
interfaces in `src/plugin-engine/types.ts` are still exported, but the
container-only `PluginManager` (`src/plugin-engine/PluginManager.ts`) never
constructs an in-process `Plugin` instance — manifests, lifecycle, and
execution are owned by `ContainerManager` and `ContainerPluginScheduler`
under `src/container-plugins/`. Treat the `Plugin`-interface types as
vestigial and do not write new code against them.
``` ## Why the change
packages/meta-sort/packages/meta-sort-core/
├── plugins/ # All plugins (shipped + custom)
│ ├── file-info/
│ │ ├── manifest.yml # Plugin metadata & config schema
│ │ └── index.ts # Plugin implementation
│ ├── ffmpeg/
│ ├── filename-parser/
│ ├── anime-detector/
│ ├── jellyfin-nfo/
│ ├── subtitle/
│ ├── torrent/
│ ├── language/
│ ├── tmdb/
│ └── debug/
├── plugins.json # Runtime state (active/inactive + configs)
└── src/
└── plugin/
├── types.ts # Type definitions
├── PluginLoader.ts # Manifest parsing, dynamic import
├── PluginContext.ts # Context factory
├── PluginManager.ts # Lifecycle management
└── PluginExecutor.ts # Dependency-ordered execution
```
Cache folders are stored separately under `CACHE_FOLDER_PATH` (default `/data/cache/plugins/`): | Limitation of in-process plugins | Container model |
|----------------------------------|-----------------|
| Heavy native deps (FFmpeg, Whisper, ML libs) inflated the meta-sort image | Each plugin ships its own image |
| All plugins had to be TypeScript / Node | Any language with an HTTP server |
| A crashing plugin took down meta-sort | Plugin crashes are isolated; meta-sort restarts the container |
| Plugin updates required redeploying meta-sort | Plugins are versioned and pulled independently |
``` See `containerized-plugin-architecture.md` for the migration rationale and
/data/cache/plugins/ the HTTP contract container plugins must implement.
├── ffmpeg/
│ ├── {cid1}.json
│ └── {cid2}.json
├── tmdb/
│ ├── {cid1}.json
│ └── {cid2}.json
└── ...
```
## Plugin Manifest
Each plugin must have a `manifest.yml` file defining its metadata, dependencies, and configuration schema.
```yaml
# manifest.yml
id: tmdb # Unique identifier (matches folder name)
name: TMDB Enricher # Human-readable name
version: 1.0.0 # Semantic version
description: Enriches metadata from The Movie Database API
# Plugins that must execute BEFORE this one
dependencies:
- filename-parser # Needs parsed title/year
# Configuration schema (rendered in dashboard)
config:
apiKey:
type: string
label: API Key
required: true
secret: true # Masked in UI
language:
type: string
label: Language
default: en-US
includeAdult:
type: boolean
label: Include Adult Content
default: false
# Metadata schema for editor UI (optional)
# Describes keys this plugin writes
schema:
tmdb/id:
label: TMDB ID
type: string
readonly: true
tmdb/title:
label: Title
type: string
tmdb/overview:
label: Overview
type: text
tmdb/poster:
label: Poster URL
type: string
format: url
tmdb/genre/{n}:
label: Genre
type: string
indexed: true # Indicates indexed keys (0, 1, 2...)
```
### Manifest Fields
| Field | Required | Description |
|-------|----------|-------------|
| `id` | Yes | Unique plugin identifier, must match folder name |
| `name` | Yes | Display name for dashboard |
| `version` | Yes | Semantic version string |
| `description` | No | Brief description |
| `dependencies` | No | Array of plugin IDs that must run before this one |
| `config` | No | Configuration schema for dashboard UI |
| `schema` | No | Metadata keys written by this plugin (for editor) |
### Config Field Types
```yaml
config:
fieldName:
type: string | number | boolean | select
label: Display Label
required: true | false # Default: false
default: <value> # Default value
secret: true | false # Mask in UI (default: false)
options: # For type: select
- value: opt1
label: Option 1
- value: opt2
label: Option 2
```
## Plugin Interface
Plugins export a default object implementing the `Plugin` interface.
```typescript
// types.ts
export interface Plugin {
/**
* Called once when the plugin is loaded at startup.
* Use for one-time initialization.
*/
onLoad?(ctx: PluginLifecycleContext): Promise<void>;
/**
* Called when the plugin is activated.
* Triggered at startup (if previously active) or from dashboard.
*/
onActivate?(ctx: PluginLifecycleContext): Promise<void>;
/**
* Called when the plugin is deactivated from the dashboard.
* Use for cleanup, releasing resources.
*/
onDeactivate?(ctx: PluginLifecycleContext): Promise<void>;
/**
* Called when the plugin is unloaded at shutdown.
*/
onUnload?(ctx: PluginLifecycleContext): Promise<void>;
/**
* Process a single file. Called for each file in the library.
* Must be implemented by all plugins.
*/
process(ctx: PluginContext): Promise<void>;
}
export interface PluginLifecycleContext {
/** Plugin configuration from plugins.json */
config: Record<string, unknown>;
/** Absolute path to plugin's cache directory */
cacheDir: string;
/** Scoped logger */
log: PluginLogger;
}
```
## Plugin Context
The `PluginContext` is injected into the `process()` function, providing access to file metadata and utilities.
```typescript
export interface PluginContext {
/**
* Key-value storage for the current file's metadata.
* Keys use path notation: "video/codec", "audio/0/language"
*/
kv: PluginKVStore;
/** Absolute path to the file being processed */
filePath: string;
/** Plugin configuration values */
config: Record<string, unknown>;
/** Absolute path to plugin's cache directory */
cacheDir: string;
/** Cache utilities for persistent storage */
cache: PluginCache;
/** Scoped logger (prefixed with plugin name) */
log: PluginLogger;
}
```
### KV Store Interface
```typescript
export interface PluginKVStore {
/**
* Get a value by key.
* @param key Path-style key, e.g., "video/codec" or "audio/0/language"
*/
get(key: string): string | undefined;
/**
* Set a value. Only string values are supported.
* @param key Path-style key
* @param value String value
*/
set(key: string, value: string): void;
/**
* Delete a key.
*/
delete(key: string): void;
/**
* Get all keys, optionally filtered by prefix.
* @param prefix Optional prefix filter, e.g., "audio/" returns ["audio/0/codec", "audio/0/language", ...]
*/
keys(prefix?: string): string[];
/**
* Get all key-value pairs, optionally filtered by prefix.
*/
entries(prefix?: string): Array<[string, string]>;
}
```
### Cache Interface
```typescript
export interface PluginCache {
/**
* Get full path for a cache file.
* @param filename Relative filename within cache directory
*/
getPath(filename: string): string;
/**
* Read and parse a JSON cache file.
* @returns Parsed data or null if not found
*/
readJson<T>(filename: string): Promise<T | null>;
/**
* Write data as JSON to cache file.
*/
writeJson(filename: string, data: unknown): Promise<void>;
/**
* Check if a cache file exists.
*/
exists(filename: string): Promise<boolean>;
/**
* Delete a cache file.
*/
delete(filename: string): Promise<void>;
/**
* Clear all files in the plugin's cache directory.
*/
clear(): Promise<void>;
}
```
### Logger Interface
```typescript
export interface PluginLogger {
debug(message: string, ...args: unknown[]): void;
info(message: string, ...args: unknown[]): void;
warn(message: string, ...args: unknown[]): void;
error(message: string, ...args: unknown[]): void;
}
```
## Plugin Implementation Example
```typescript
// plugins/tmdb/index.ts
import type { Plugin, PluginContext, PluginLifecycleContext } from '../../src/plugin/types';
interface TmdbConfig {
apiKey: string;
language: string;
includeAdult: boolean;
}
const plugin: Plugin = {
async onLoad(ctx: PluginLifecycleContext) {
ctx.log.info('TMDB plugin loaded');
},
async onActivate(ctx: PluginLifecycleContext) {
const config = ctx.config as TmdbConfig;
if (!config.apiKey) {
ctx.log.warn('No API key configured, plugin will skip processing');
}
},
async onDeactivate(ctx: PluginLifecycleContext) {
ctx.log.info('TMDB plugin deactivated');
},
async process(ctx: PluginContext) {
const config = ctx.config as TmdbConfig;
// Skip if not configured
if (!config.apiKey) return;
// Get data from previous plugins
const title = ctx.kv.get('filename/title');
const year = ctx.kv.get('filename/year');
const videoType = ctx.kv.get('video/type');
if (!title) return;
// Check cache first (keyed by title+year)
const cacheKey = `${sanitize(title)}-${year || 'unknown'}.json`;
const cached = await ctx.cache.readJson<TmdbResult>(cacheKey);
if (cached) {
applyResult(ctx.kv, cached);
return;
}
// Fetch from TMDB API
try {
const result = await fetchTmdb(title, year, videoType, config);
if (result) {
await ctx.cache.writeJson(cacheKey, result);
applyResult(ctx.kv, result);
}
} catch (error) {
ctx.log.error('TMDB fetch failed', error);
}
}
};
function applyResult(kv: PluginContext['kv'], result: TmdbResult) {
kv.set('tmdb/id', String(result.id));
kv.set('tmdb/title', result.title);
kv.set('tmdb/overview', result.overview);
kv.set('tmdb/poster', result.poster_path);
kv.set('tmdb/backdrop', result.backdrop_path);
kv.set('tmdb/rating', String(result.vote_average));
kv.set('tmdb/year', result.release_date?.slice(0, 4) || '');
// Genres as indexed keys
result.genres?.forEach((genre, i) => {
kv.set(`tmdb/genre/${i}`, genre.name);
});
}
export default plugin;
```
## Metadata Key Format
All metadata is stored as flat key-value pairs with string values only. Keys use path notation with `/` as separator.
### Key Naming Convention
```
{plugin-id}/{field} # Simple field
{plugin-id}/{group}/{field} # Grouped field
{plugin-id}/{group}/{index}/{field} # Indexed field (for lists)
```
### Examples
```
# File info plugin
file/name = "Movie.2024.1080p.mkv"
file/size = "4294967296"
file/type = "video"
file/mime = "video/x-matroska"
# FFmpeg plugin
video/codec = "h264"
video/width = "1920"
video/height = "1080"
video/duration = "7200.5"
video/bitrate = "8000000"
audio/0/codec = "aac"
audio/0/language = "eng"
audio/0/channels = "6"
audio/1/codec = "dts"
audio/1/language = "fra"
audio/1/channels = "6"
subtitle/0/language = "eng"
subtitle/0/format = "subrip"
subtitle/1/language = "fra"
subtitle/1/format = "subrip"
# Filename parser plugin
filename/title = "Movie Title"
filename/year = "2024"
filename/quality = "1080p"
filename/source = "bluray"
# TMDB plugin
tmdb/id = "12345"
tmdb/title = "Movie Title"
tmdb/overview = "A movie about..."
tmdb/poster = "/path/to/poster.jpg"
tmdb/genre/0 = "Action"
tmdb/genre/1 = "Adventure"
```
### Key Constraints
- Keys must be non-empty strings
- Keys use `/` as path separator
- Values must be strings (numbers converted via `String()`)
- No nested objects or arrays (use indexed keys instead)
- Keys are case-sensitive
- Recommended: use lowercase with hyphens for multi-word keys
## Plugin Manager
The `PluginManager` handles plugin lifecycle and orchestration.
```typescript
export class PluginManager extends EventEmitter {
/**
* Scan plugins directory and load manifests.
* Called once at startup.
*/
async scanPlugins(): Promise<PluginInfo[]>;
/**
* Load all plugins and activate those marked active in state.
* Called once at startup after scanning.
*/
async loadAll(): Promise<void>;
/**
* Activate a plugin (hot, no restart required).
* Calls plugin.onActivate(), rebuilds execution order, saves state.
* Emits 'plugin:activated' event.
*/
async activate(pluginId: string): Promise<void>;
/**
* Deactivate a plugin (hot, no restart required).
* Calls plugin.onDeactivate(), rebuilds execution order, saves state.
* Emits 'plugin:deactivated' event.
*/
async deactivate(pluginId: string): Promise<void>;
/**
* Update plugin configuration.
* Saves to plugins.json.
* Emits 'plugin:config-changed' event.
*/
async updateConfig(pluginId: string, config: Record<string, unknown>): Promise<void>;
/**
* Clear a plugin's cache directory.
*/
async clearCache(pluginId: string): Promise<void>;
/**
* Get all discovered plugins.
*/
getPlugins(): PluginInfo[];
/**
* Get only active plugins.
*/
getActivePlugins(): PluginInfo[];
/**
* Get execution order (dependency-sorted active plugin IDs).
*/
getExecutionOrder(): string[];
/**
* Process a file through all active plugins.
*/
async processFile(filePath: string, kv: KVStore): Promise<ProcessingResult>;
/**
* Shutdown: deactivate and unload all plugins.
*/
async shutdown(): Promise<void>;
}
```
### Events
| Event | Payload | Description |
|-------|---------|-------------|
| `plugin:activated` | `{ pluginId: string }` | Plugin was activated |
| `plugin:deactivated` | `{ pluginId: string }` | Plugin was deactivated |
| `plugin:config-changed` | `{ pluginId: string, config: object }` | Plugin config updated |
| `plugin:error` | `{ pluginId: string, error: Error }` | Plugin error occurred |
### Plugin Info
```typescript
export interface PluginInfo {
id: string;
name: string;
version: string;
description?: string;
dependencies: string[];
active: boolean;
config: Record<string, unknown>;
configSchema?: ConfigSchema;
metadataSchema?: MetadataSchema;
status: 'loaded' | 'error' | 'unloaded';
error?: string;
}
```
## State Management
Plugin state is persisted in `plugins.json` located in the meta-sort-core package root.
```json
{
"plugins": {
"file-info": {
"active": true,
"config": {}
},
"ffmpeg": {
"active": true,
"config": {}
},
"tmdb": {
"active": true,
"config": {
"apiKey": "your-api-key",
"language": "en-US",
"includeAdult": false
}
},
"custom-plugin": {
"active": false,
"config": {}
}
}
}
```
### State Behavior
- **New plugins**: Discovered plugins not in state file are added with `active: false`
- **Removed plugins**: Plugins in state but not on disk are kept (preserves config for reinstall)
- **Config merge**: Default values from manifest are merged with saved config
- **Persistence**: State is saved immediately on any change
## Execution Pipeline
### Startup Sequence
```
1. PluginManager.scanPlugins()
└── Scan plugins/ directory
└── Parse each manifest.yml
└── Validate manifest structure
2. PluginManager.loadAll()
└── Load plugins.json state
└── For each discovered plugin:
├── Dynamic import(plugin/index.js)
├── Validate Plugin interface
├── Call plugin.onLoad()
└── If active in state:
└── Call plugin.onActivate()
└── Resolve execution order (topological sort)
```
### File Processing Sequence
```
1. Create KVStore for file
2. For each active plugin (in dependency order):
├── Create PluginContext
├── Start timing
├── try:
│ └── await plugin.process(ctx)
└── catch:
└── Log error, continue to next plugin
3. Persist KV to Redis
```
### Activation Sequence (Hot)
```
1. PluginManager.activate(pluginId)
├── Validate plugin exists and is loaded
├── Check dependencies are active
├── Call plugin.onActivate()
├── Mark as active in memory
├── Rebuild execution order
├── Save plugins.json
└── Emit 'plugin:activated' event
2. Caller receives event
└── Trigger library rescan
```
### Deactivation Sequence (Hot)
```
1. PluginManager.deactivate(pluginId)
├── Check no active plugins depend on this one
├── Call plugin.onDeactivate()
├── Mark as inactive in memory
├── Rebuild execution order
├── Save plugins.json
└── Emit 'plugin:deactivated' event
2. Caller receives event
└── Trigger library rescan
Note: Metadata written by the plugin remains in Redis.
A separate "clean metadata" feature can remove orphaned keys.
```
## Dependency Resolution
Plugins declare dependencies in their manifest. The system performs topological sort to determine execution order.
### Rules
1. A plugin cannot be activated if any of its dependencies are inactive
2. A plugin cannot be deactivated if any active plugin depends on it
3. Circular dependencies are detected and rejected at load time
4. Missing dependencies are logged as warnings; plugin can still load but may fail
### Example
```yaml
# filename-parser depends on file-info
# tmdb depends on filename-parser
# anime-detector depends on filename-parser
Resolved order: file-info → filename-parser → [tmdb, anime-detector]
```
Plugins at the same dependency level may execute in any order (or parallel in future).
## REST API
The plugin system exposes REST endpoints for dashboard integration.
### Endpoints
```
GET /api/plugins # List all plugins
GET /api/plugins/:id # Get plugin details
POST /api/plugins/:id/activate # Activate plugin
POST /api/plugins/:id/deactivate # Deactivate plugin
PUT /api/plugins/:id/config # Update plugin config
DELETE /api/plugins/:id/cache # Clear plugin cache
GET /api/plugins/:id/schema # Get metadata schema (for editor)
```
### Response Format
```typescript
// GET /api/plugins
interface PluginsResponse {
plugins: PluginInfo[];
executionOrder: string[];
}
// POST /api/plugins/:id/activate
interface ActivateResponse {
success: boolean;
error?: string; # e.g., "Dependency 'x' is not active"
}
// PUT /api/plugins/:id/config
interface ConfigUpdateRequest {
config: Record<string, unknown>;
}
```
## Error Handling
### Plugin Load Errors
- Invalid manifest: Plugin marked as `status: 'error'`, logged, skipped
- Import failure: Plugin marked as `status: 'error'`, logged, skipped
- Missing dependencies: Warning logged, plugin can load but activation may fail
### Plugin Execution Errors
- Errors in `process()` are caught, logged, and execution continues to next plugin
- The file's metadata from previous plugins is preserved
- Error count is tracked per plugin for monitoring
### Lifecycle Hook Errors
- Errors in `onActivate()` prevent activation, error returned to caller
- Errors in `onDeactivate()` are logged but deactivation proceeds
- Errors in `onLoad()`/`onUnload()` are logged, plugin marked as error state
## Future Considerations
### Metadata Cleanup
A future feature will allow cleaning metadata keys written by a specific plugin:
```typescript
// Future API
await pluginManager.cleanMetadata(pluginId);
// Removes all keys matching "{pluginId}/*" from Redis
```
### Parallel Execution
Plugins at the same dependency level could execute in parallel:
```typescript
// Future: parallel execution of independent plugins
const level1 = ['file-info'];
const level2 = ['ffmpeg', 'filename-parser']; // parallel
const level3 = ['tmdb', 'anime-detector']; // parallel
```
### Plugin Hot Reload
Reloading a plugin's code without full restart (for development):
```typescript
// Future API
await pluginManager.reload(pluginId);
// Re-imports plugin code, preserves state
```
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -35,7 +35,7 @@ lazyValue().then(value => console.log(value));
### Listener Cleaner ### Listener Cleaner
```typescript ```typescript
import { ListenerCleaner } from "worph/async-utils"; import { ListenerCleaner } from "@worph/async-utils";
const cleaner = new ListenerCleaner(); const cleaner = new ListenerCleaner();
cleaner.add(() => console.log("Cleanup action")); cleaner.add(() => console.log("Cleanup action"));