improve doc and bug fix
This commit is contained in:
@@ -587,7 +587,7 @@ Deduplication logic remains identical. Fast-hash serves the same purpose as full
|
|||||||
|
|
||||||
The Stremio addon retrieves files by metadata queries (title, season, episode), not by hash. The hash is used internally for metadata storage. Changing to fast-hash is transparent to the addon.
|
The Stremio addon retrieves files by metadata queries (title, season, episode), not by hash. The hash is used internally for metadata storage. Changing to fast-hash is transparent to the addon.
|
||||||
|
|
||||||
### Meta-Orbit (Distributed Synchronization)
|
### meta-share (Distributed Synchronization)
|
||||||
|
|
||||||
**Impact:** Hash serves as content identifier for P2P synchronization.
|
**Impact:** Hash serves as content identifier for P2P synchronization.
|
||||||
|
|
||||||
@@ -716,7 +716,7 @@ MetaMesh implements a two-phase processing pipeline:
|
|||||||
**Phase 2: Hash Processing (Full Hashes - Background)**
|
**Phase 2: Hash Processing (Full Hashes - Background)**
|
||||||
- Compute full-file hashes: SHA-256, SHA-1, MD5, CRC32 (slow, minutes)
|
- Compute full-file hashes: SHA-256, SHA-1, MD5, CRC32 (slow, minutes)
|
||||||
- Update etcd with additional hash metadata
|
- Update etcd with additional hash metadata
|
||||||
- Required for Meta-Orbit distributed synchronization
|
- Required for meta-share distributed synchronization
|
||||||
|
|
||||||
### Why Both Are Needed
|
### Why Both Are Needed
|
||||||
|
|
||||||
@@ -726,7 +726,7 @@ MetaMesh implements a two-phase processing pipeline:
|
|||||||
- **Fast deduplication**: Quickly identify potential duplicates
|
- **Fast deduplication**: Quickly identify potential duplicates
|
||||||
|
|
||||||
**Full Hash Purpose:**
|
**Full Hash Purpose:**
|
||||||
- **Meta-Orbit integration**: Distributed P2P synchronization requires cryptographically secure hashes
|
- **meta-share integration**: Distributed P2P synchronization requires cryptographically secure hashes
|
||||||
- **External system compatibility**: Torrent verification, IPFS CIDs
|
- **External system compatibility**: Torrent verification, IPFS CIDs
|
||||||
- **Collision resolution**: If midhash256 collision detected, full SHA-256 provides definitive answer
|
- **Collision resolution**: If midhash256 collision detected, full SHA-256 provides definitive answer
|
||||||
|
|
||||||
@@ -744,7 +744,7 @@ Light Processing (32 workers, parallel)
|
|||||||
Hash Processing (8 workers, background)
|
Hash Processing (8 workers, background)
|
||||||
- Compute SHA-256, SHA-1, MD5, CRC32 (15+ minutes for 50GB)
|
- Compute SHA-256, SHA-1, MD5, CRC32 (15+ minutes for 50GB)
|
||||||
- Update etcd with full hashes
|
- Update etcd with full hashes
|
||||||
- Enable Meta-Orbit synchronization ✓
|
- Enable meta-share synchronization ✓
|
||||||
```
|
```
|
||||||
|
|
||||||
### Collision Detection and Handling
|
### Collision Detection and Handling
|
||||||
@@ -778,4 +778,4 @@ The trade-offs—weaker cryptographic guarantees, potential collisions for adver
|
|||||||
|
|
||||||
This architecture aligns with Meta-Mesh's design philosophy: **optimize for the common case** (large media files, organic content, personal libraries) while maintaining **practicality and performance** over theoretical perfection.
|
This architecture aligns with Meta-Mesh's design philosophy: **optimize for the common case** (large media files, organic content, personal libraries) while maintaining **practicality and performance** over theoretical perfection.
|
||||||
|
|
||||||
Meta-Mesh leverages midhash256 for **immediate file availability** while computing full SHA-256 hashes **in the background** for Meta-Orbit distributed synchronization and external system compatibility. This dual-hash approach provides both speed and security.
|
Meta-Mesh leverages midhash256 for **immediate file availability** while computing full SHA-256 hashes **in the background** for meta-share distributed synchronization and external system compatibility. This dual-hash approach provides both speed and security.
|
||||||
|
|||||||
+27
-754
@@ -1,762 +1,35 @@
|
|||||||
# Meta-Sort Plugin System Architecture
|
# Plugin System (Deprecated — See Container Plugins)
|
||||||
|
|
||||||
This document describes the plugin system architecture for meta-sort, enabling extensible metadata extraction through dynamically loaded plugins.
|
> **This document is a historical pointer.** The in-process plugin system it
|
||||||
|
> originally described — `plugins/<id>/{manifest.yml,index.ts}` directories
|
||||||
|
> loaded via dynamic `import()` inside meta-sort-core — has been removed.
|
||||||
|
> All plugins now run as Docker containers and communicate over HTTP.
|
||||||
|
|
||||||
## Overview
|
For the current design, read:
|
||||||
|
|
||||||
The plugin system provides a modular architecture for metadata extraction. Plugins are self-contained units that:
|
- **[containerized-plugin-architecture.md](./containerized-plugin-architecture.md)** — full container plugin design (HTTP contract, lifecycle, callbacks, KV access).
|
||||||
|
- **[plugin-task-queue-architecture.md](./plugin-task-queue-architecture.md)** — how plugin work is dispatched across the fast / background queues.
|
||||||
|
- Root **`CLAUDE.md`** (`Plugin Development` and `Runtime Plugin Configuration` sections) — operational quickstart.
|
||||||
|
- Plugin sources at **`packages/plugins/metamesh-plugin-*`** (separate submodules) — `file-info`, `ffmpeg`, `filename-parser`, `jellyfin-nfo`, `tmdb`, `anime-detector`, `language`, `subtitle`, `subtitle-extractor`, `torrent`, `fullhash`.
|
||||||
|
|
||||||
- Are discovered and loaded from a plugins directory at startup
|
## What survived from the old design
|
||||||
- Can be activated/deactivated at runtime without restart
|
|
||||||
- Process files sequentially in dependency order
|
|
||||||
- Store metadata in a flat key-value format
|
|
||||||
- Have access to their own persistent cache folder
|
|
||||||
|
|
||||||
## Directory Structure
|
The `Plugin` / `PluginContext` / `PluginLifecycleContext` / `PluginKVStore`
|
||||||
|
interfaces in `src/plugin-engine/types.ts` are still exported, but the
|
||||||
|
container-only `PluginManager` (`src/plugin-engine/PluginManager.ts`) never
|
||||||
|
constructs an in-process `Plugin` instance — manifests, lifecycle, and
|
||||||
|
execution are owned by `ContainerManager` and `ContainerPluginScheduler`
|
||||||
|
under `src/container-plugins/`. Treat the `Plugin`-interface types as
|
||||||
|
vestigial and do not write new code against them.
|
||||||
|
|
||||||
```
|
## Why the change
|
||||||
packages/meta-sort/packages/meta-sort-core/
|
|
||||||
├── plugins/ # All plugins (shipped + custom)
|
|
||||||
│ ├── file-info/
|
|
||||||
│ │ ├── manifest.yml # Plugin metadata & config schema
|
|
||||||
│ │ └── index.ts # Plugin implementation
|
|
||||||
│ ├── ffmpeg/
|
|
||||||
│ ├── filename-parser/
|
|
||||||
│ ├── anime-detector/
|
|
||||||
│ ├── jellyfin-nfo/
|
|
||||||
│ ├── subtitle/
|
|
||||||
│ ├── torrent/
|
|
||||||
│ ├── language/
|
|
||||||
│ ├── tmdb/
|
|
||||||
│ └── debug/
|
|
||||||
├── plugins.json # Runtime state (active/inactive + configs)
|
|
||||||
└── src/
|
|
||||||
└── plugin/
|
|
||||||
├── types.ts # Type definitions
|
|
||||||
├── PluginLoader.ts # Manifest parsing, dynamic import
|
|
||||||
├── PluginContext.ts # Context factory
|
|
||||||
├── PluginManager.ts # Lifecycle management
|
|
||||||
└── PluginExecutor.ts # Dependency-ordered execution
|
|
||||||
```
|
|
||||||
|
|
||||||
Cache folders are stored separately under `CACHE_FOLDER_PATH` (default `/data/cache/plugins/`):
|
| Limitation of in-process plugins | Container model |
|
||||||
|
|----------------------------------|-----------------|
|
||||||
|
| Heavy native deps (FFmpeg, Whisper, ML libs) inflated the meta-sort image | Each plugin ships its own image |
|
||||||
|
| All plugins had to be TypeScript / Node | Any language with an HTTP server |
|
||||||
|
| A crashing plugin took down meta-sort | Plugin crashes are isolated; meta-sort restarts the container |
|
||||||
|
| Plugin updates required redeploying meta-sort | Plugins are versioned and pulled independently |
|
||||||
|
|
||||||
```
|
See `containerized-plugin-architecture.md` for the migration rationale and
|
||||||
/data/cache/plugins/
|
the HTTP contract container plugins must implement.
|
||||||
├── ffmpeg/
|
|
||||||
│ ├── {cid1}.json
|
|
||||||
│ └── {cid2}.json
|
|
||||||
├── tmdb/
|
|
||||||
│ ├── {cid1}.json
|
|
||||||
│ └── {cid2}.json
|
|
||||||
└── ...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Plugin Manifest
|
|
||||||
|
|
||||||
Each plugin must have a `manifest.yml` file defining its metadata, dependencies, and configuration schema.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# manifest.yml
|
|
||||||
id: tmdb # Unique identifier (matches folder name)
|
|
||||||
name: TMDB Enricher # Human-readable name
|
|
||||||
version: 1.0.0 # Semantic version
|
|
||||||
description: Enriches metadata from The Movie Database API
|
|
||||||
|
|
||||||
# Plugins that must execute BEFORE this one
|
|
||||||
dependencies:
|
|
||||||
- filename-parser # Needs parsed title/year
|
|
||||||
|
|
||||||
# Configuration schema (rendered in dashboard)
|
|
||||||
config:
|
|
||||||
apiKey:
|
|
||||||
type: string
|
|
||||||
label: API Key
|
|
||||||
required: true
|
|
||||||
secret: true # Masked in UI
|
|
||||||
language:
|
|
||||||
type: string
|
|
||||||
label: Language
|
|
||||||
default: en-US
|
|
||||||
includeAdult:
|
|
||||||
type: boolean
|
|
||||||
label: Include Adult Content
|
|
||||||
default: false
|
|
||||||
|
|
||||||
# Metadata schema for editor UI (optional)
|
|
||||||
# Describes keys this plugin writes
|
|
||||||
schema:
|
|
||||||
tmdb/id:
|
|
||||||
label: TMDB ID
|
|
||||||
type: string
|
|
||||||
readonly: true
|
|
||||||
tmdb/title:
|
|
||||||
label: Title
|
|
||||||
type: string
|
|
||||||
tmdb/overview:
|
|
||||||
label: Overview
|
|
||||||
type: text
|
|
||||||
tmdb/poster:
|
|
||||||
label: Poster URL
|
|
||||||
type: string
|
|
||||||
format: url
|
|
||||||
tmdb/genre/{n}:
|
|
||||||
label: Genre
|
|
||||||
type: string
|
|
||||||
indexed: true # Indicates indexed keys (0, 1, 2...)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Manifest Fields
|
|
||||||
|
|
||||||
| Field | Required | Description |
|
|
||||||
|-------|----------|-------------|
|
|
||||||
| `id` | Yes | Unique plugin identifier, must match folder name |
|
|
||||||
| `name` | Yes | Display name for dashboard |
|
|
||||||
| `version` | Yes | Semantic version string |
|
|
||||||
| `description` | No | Brief description |
|
|
||||||
| `dependencies` | No | Array of plugin IDs that must run before this one |
|
|
||||||
| `config` | No | Configuration schema for dashboard UI |
|
|
||||||
| `schema` | No | Metadata keys written by this plugin (for editor) |
|
|
||||||
|
|
||||||
### Config Field Types
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
config:
|
|
||||||
fieldName:
|
|
||||||
type: string | number | boolean | select
|
|
||||||
label: Display Label
|
|
||||||
required: true | false # Default: false
|
|
||||||
default: <value> # Default value
|
|
||||||
secret: true | false # Mask in UI (default: false)
|
|
||||||
options: # For type: select
|
|
||||||
- value: opt1
|
|
||||||
label: Option 1
|
|
||||||
- value: opt2
|
|
||||||
label: Option 2
|
|
||||||
```
|
|
||||||
|
|
||||||
## Plugin Interface
|
|
||||||
|
|
||||||
Plugins export a default object implementing the `Plugin` interface.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// types.ts
|
|
||||||
|
|
||||||
export interface Plugin {
|
|
||||||
/**
|
|
||||||
* Called once when the plugin is loaded at startup.
|
|
||||||
* Use for one-time initialization.
|
|
||||||
*/
|
|
||||||
onLoad?(ctx: PluginLifecycleContext): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Called when the plugin is activated.
|
|
||||||
* Triggered at startup (if previously active) or from dashboard.
|
|
||||||
*/
|
|
||||||
onActivate?(ctx: PluginLifecycleContext): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Called when the plugin is deactivated from the dashboard.
|
|
||||||
* Use for cleanup, releasing resources.
|
|
||||||
*/
|
|
||||||
onDeactivate?(ctx: PluginLifecycleContext): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Called when the plugin is unloaded at shutdown.
|
|
||||||
*/
|
|
||||||
onUnload?(ctx: PluginLifecycleContext): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Process a single file. Called for each file in the library.
|
|
||||||
* Must be implemented by all plugins.
|
|
||||||
*/
|
|
||||||
process(ctx: PluginContext): Promise<void>;
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface PluginLifecycleContext {
|
|
||||||
/** Plugin configuration from plugins.json */
|
|
||||||
config: Record<string, unknown>;
|
|
||||||
|
|
||||||
/** Absolute path to plugin's cache directory */
|
|
||||||
cacheDir: string;
|
|
||||||
|
|
||||||
/** Scoped logger */
|
|
||||||
log: PluginLogger;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Plugin Context
|
|
||||||
|
|
||||||
The `PluginContext` is injected into the `process()` function, providing access to file metadata and utilities.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export interface PluginContext {
|
|
||||||
/**
|
|
||||||
* Key-value storage for the current file's metadata.
|
|
||||||
* Keys use path notation: "video/codec", "audio/0/language"
|
|
||||||
*/
|
|
||||||
kv: PluginKVStore;
|
|
||||||
|
|
||||||
/** Absolute path to the file being processed */
|
|
||||||
filePath: string;
|
|
||||||
|
|
||||||
/** Plugin configuration values */
|
|
||||||
config: Record<string, unknown>;
|
|
||||||
|
|
||||||
/** Absolute path to plugin's cache directory */
|
|
||||||
cacheDir: string;
|
|
||||||
|
|
||||||
/** Cache utilities for persistent storage */
|
|
||||||
cache: PluginCache;
|
|
||||||
|
|
||||||
/** Scoped logger (prefixed with plugin name) */
|
|
||||||
log: PluginLogger;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### KV Store Interface
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export interface PluginKVStore {
|
|
||||||
/**
|
|
||||||
* Get a value by key.
|
|
||||||
* @param key Path-style key, e.g., "video/codec" or "audio/0/language"
|
|
||||||
*/
|
|
||||||
get(key: string): string | undefined;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Set a value. Only string values are supported.
|
|
||||||
* @param key Path-style key
|
|
||||||
* @param value String value
|
|
||||||
*/
|
|
||||||
set(key: string, value: string): void;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Delete a key.
|
|
||||||
*/
|
|
||||||
delete(key: string): void;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get all keys, optionally filtered by prefix.
|
|
||||||
* @param prefix Optional prefix filter, e.g., "audio/" returns ["audio/0/codec", "audio/0/language", ...]
|
|
||||||
*/
|
|
||||||
keys(prefix?: string): string[];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get all key-value pairs, optionally filtered by prefix.
|
|
||||||
*/
|
|
||||||
entries(prefix?: string): Array<[string, string]>;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Cache Interface
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export interface PluginCache {
|
|
||||||
/**
|
|
||||||
* Get full path for a cache file.
|
|
||||||
* @param filename Relative filename within cache directory
|
|
||||||
*/
|
|
||||||
getPath(filename: string): string;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Read and parse a JSON cache file.
|
|
||||||
* @returns Parsed data or null if not found
|
|
||||||
*/
|
|
||||||
readJson<T>(filename: string): Promise<T | null>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Write data as JSON to cache file.
|
|
||||||
*/
|
|
||||||
writeJson(filename: string, data: unknown): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Check if a cache file exists.
|
|
||||||
*/
|
|
||||||
exists(filename: string): Promise<boolean>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Delete a cache file.
|
|
||||||
*/
|
|
||||||
delete(filename: string): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Clear all files in the plugin's cache directory.
|
|
||||||
*/
|
|
||||||
clear(): Promise<void>;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Logger Interface
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export interface PluginLogger {
|
|
||||||
debug(message: string, ...args: unknown[]): void;
|
|
||||||
info(message: string, ...args: unknown[]): void;
|
|
||||||
warn(message: string, ...args: unknown[]): void;
|
|
||||||
error(message: string, ...args: unknown[]): void;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Plugin Implementation Example
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// plugins/tmdb/index.ts
|
|
||||||
import type { Plugin, PluginContext, PluginLifecycleContext } from '../../src/plugin/types';
|
|
||||||
|
|
||||||
interface TmdbConfig {
|
|
||||||
apiKey: string;
|
|
||||||
language: string;
|
|
||||||
includeAdult: boolean;
|
|
||||||
}
|
|
||||||
|
|
||||||
const plugin: Plugin = {
|
|
||||||
async onLoad(ctx: PluginLifecycleContext) {
|
|
||||||
ctx.log.info('TMDB plugin loaded');
|
|
||||||
},
|
|
||||||
|
|
||||||
async onActivate(ctx: PluginLifecycleContext) {
|
|
||||||
const config = ctx.config as TmdbConfig;
|
|
||||||
if (!config.apiKey) {
|
|
||||||
ctx.log.warn('No API key configured, plugin will skip processing');
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
async onDeactivate(ctx: PluginLifecycleContext) {
|
|
||||||
ctx.log.info('TMDB plugin deactivated');
|
|
||||||
},
|
|
||||||
|
|
||||||
async process(ctx: PluginContext) {
|
|
||||||
const config = ctx.config as TmdbConfig;
|
|
||||||
|
|
||||||
// Skip if not configured
|
|
||||||
if (!config.apiKey) return;
|
|
||||||
|
|
||||||
// Get data from previous plugins
|
|
||||||
const title = ctx.kv.get('filename/title');
|
|
||||||
const year = ctx.kv.get('filename/year');
|
|
||||||
const videoType = ctx.kv.get('video/type');
|
|
||||||
|
|
||||||
if (!title) return;
|
|
||||||
|
|
||||||
// Check cache first (keyed by title+year)
|
|
||||||
const cacheKey = `${sanitize(title)}-${year || 'unknown'}.json`;
|
|
||||||
const cached = await ctx.cache.readJson<TmdbResult>(cacheKey);
|
|
||||||
|
|
||||||
if (cached) {
|
|
||||||
applyResult(ctx.kv, cached);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Fetch from TMDB API
|
|
||||||
try {
|
|
||||||
const result = await fetchTmdb(title, year, videoType, config);
|
|
||||||
if (result) {
|
|
||||||
await ctx.cache.writeJson(cacheKey, result);
|
|
||||||
applyResult(ctx.kv, result);
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
ctx.log.error('TMDB fetch failed', error);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
function applyResult(kv: PluginContext['kv'], result: TmdbResult) {
|
|
||||||
kv.set('tmdb/id', String(result.id));
|
|
||||||
kv.set('tmdb/title', result.title);
|
|
||||||
kv.set('tmdb/overview', result.overview);
|
|
||||||
kv.set('tmdb/poster', result.poster_path);
|
|
||||||
kv.set('tmdb/backdrop', result.backdrop_path);
|
|
||||||
kv.set('tmdb/rating', String(result.vote_average));
|
|
||||||
kv.set('tmdb/year', result.release_date?.slice(0, 4) || '');
|
|
||||||
|
|
||||||
// Genres as indexed keys
|
|
||||||
result.genres?.forEach((genre, i) => {
|
|
||||||
kv.set(`tmdb/genre/${i}`, genre.name);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
export default plugin;
|
|
||||||
```
|
|
||||||
|
|
||||||
## Metadata Key Format
|
|
||||||
|
|
||||||
All metadata is stored as flat key-value pairs with string values only. Keys use path notation with `/` as separator.
|
|
||||||
|
|
||||||
### Key Naming Convention
|
|
||||||
|
|
||||||
```
|
|
||||||
{plugin-id}/{field} # Simple field
|
|
||||||
{plugin-id}/{group}/{field} # Grouped field
|
|
||||||
{plugin-id}/{group}/{index}/{field} # Indexed field (for lists)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```
|
|
||||||
# File info plugin
|
|
||||||
file/name = "Movie.2024.1080p.mkv"
|
|
||||||
file/size = "4294967296"
|
|
||||||
file/type = "video"
|
|
||||||
file/mime = "video/x-matroska"
|
|
||||||
|
|
||||||
# FFmpeg plugin
|
|
||||||
video/codec = "h264"
|
|
||||||
video/width = "1920"
|
|
||||||
video/height = "1080"
|
|
||||||
video/duration = "7200.5"
|
|
||||||
video/bitrate = "8000000"
|
|
||||||
audio/0/codec = "aac"
|
|
||||||
audio/0/language = "eng"
|
|
||||||
audio/0/channels = "6"
|
|
||||||
audio/1/codec = "dts"
|
|
||||||
audio/1/language = "fra"
|
|
||||||
audio/1/channels = "6"
|
|
||||||
subtitle/0/language = "eng"
|
|
||||||
subtitle/0/format = "subrip"
|
|
||||||
subtitle/1/language = "fra"
|
|
||||||
subtitle/1/format = "subrip"
|
|
||||||
|
|
||||||
# Filename parser plugin
|
|
||||||
filename/title = "Movie Title"
|
|
||||||
filename/year = "2024"
|
|
||||||
filename/quality = "1080p"
|
|
||||||
filename/source = "bluray"
|
|
||||||
|
|
||||||
# TMDB plugin
|
|
||||||
tmdb/id = "12345"
|
|
||||||
tmdb/title = "Movie Title"
|
|
||||||
tmdb/overview = "A movie about..."
|
|
||||||
tmdb/poster = "/path/to/poster.jpg"
|
|
||||||
tmdb/genre/0 = "Action"
|
|
||||||
tmdb/genre/1 = "Adventure"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Key Constraints
|
|
||||||
|
|
||||||
- Keys must be non-empty strings
|
|
||||||
- Keys use `/` as path separator
|
|
||||||
- Values must be strings (numbers converted via `String()`)
|
|
||||||
- No nested objects or arrays (use indexed keys instead)
|
|
||||||
- Keys are case-sensitive
|
|
||||||
- Recommended: use lowercase with hyphens for multi-word keys
|
|
||||||
|
|
||||||
## Plugin Manager
|
|
||||||
|
|
||||||
The `PluginManager` handles plugin lifecycle and orchestration.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export class PluginManager extends EventEmitter {
|
|
||||||
/**
|
|
||||||
* Scan plugins directory and load manifests.
|
|
||||||
* Called once at startup.
|
|
||||||
*/
|
|
||||||
async scanPlugins(): Promise<PluginInfo[]>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Load all plugins and activate those marked active in state.
|
|
||||||
* Called once at startup after scanning.
|
|
||||||
*/
|
|
||||||
async loadAll(): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Activate a plugin (hot, no restart required).
|
|
||||||
* Calls plugin.onActivate(), rebuilds execution order, saves state.
|
|
||||||
* Emits 'plugin:activated' event.
|
|
||||||
*/
|
|
||||||
async activate(pluginId: string): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Deactivate a plugin (hot, no restart required).
|
|
||||||
* Calls plugin.onDeactivate(), rebuilds execution order, saves state.
|
|
||||||
* Emits 'plugin:deactivated' event.
|
|
||||||
*/
|
|
||||||
async deactivate(pluginId: string): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Update plugin configuration.
|
|
||||||
* Saves to plugins.json.
|
|
||||||
* Emits 'plugin:config-changed' event.
|
|
||||||
*/
|
|
||||||
async updateConfig(pluginId: string, config: Record<string, unknown>): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Clear a plugin's cache directory.
|
|
||||||
*/
|
|
||||||
async clearCache(pluginId: string): Promise<void>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get all discovered plugins.
|
|
||||||
*/
|
|
||||||
getPlugins(): PluginInfo[];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get only active plugins.
|
|
||||||
*/
|
|
||||||
getActivePlugins(): PluginInfo[];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Get execution order (dependency-sorted active plugin IDs).
|
|
||||||
*/
|
|
||||||
getExecutionOrder(): string[];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Process a file through all active plugins.
|
|
||||||
*/
|
|
||||||
async processFile(filePath: string, kv: KVStore): Promise<ProcessingResult>;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Shutdown: deactivate and unload all plugins.
|
|
||||||
*/
|
|
||||||
async shutdown(): Promise<void>;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Events
|
|
||||||
|
|
||||||
| Event | Payload | Description |
|
|
||||||
|-------|---------|-------------|
|
|
||||||
| `plugin:activated` | `{ pluginId: string }` | Plugin was activated |
|
|
||||||
| `plugin:deactivated` | `{ pluginId: string }` | Plugin was deactivated |
|
|
||||||
| `plugin:config-changed` | `{ pluginId: string, config: object }` | Plugin config updated |
|
|
||||||
| `plugin:error` | `{ pluginId: string, error: Error }` | Plugin error occurred |
|
|
||||||
|
|
||||||
### Plugin Info
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
export interface PluginInfo {
|
|
||||||
id: string;
|
|
||||||
name: string;
|
|
||||||
version: string;
|
|
||||||
description?: string;
|
|
||||||
dependencies: string[];
|
|
||||||
active: boolean;
|
|
||||||
config: Record<string, unknown>;
|
|
||||||
configSchema?: ConfigSchema;
|
|
||||||
metadataSchema?: MetadataSchema;
|
|
||||||
status: 'loaded' | 'error' | 'unloaded';
|
|
||||||
error?: string;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## State Management
|
|
||||||
|
|
||||||
Plugin state is persisted in `plugins.json` located in the meta-sort-core package root.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"plugins": {
|
|
||||||
"file-info": {
|
|
||||||
"active": true,
|
|
||||||
"config": {}
|
|
||||||
},
|
|
||||||
"ffmpeg": {
|
|
||||||
"active": true,
|
|
||||||
"config": {}
|
|
||||||
},
|
|
||||||
"tmdb": {
|
|
||||||
"active": true,
|
|
||||||
"config": {
|
|
||||||
"apiKey": "your-api-key",
|
|
||||||
"language": "en-US",
|
|
||||||
"includeAdult": false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"custom-plugin": {
|
|
||||||
"active": false,
|
|
||||||
"config": {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### State Behavior
|
|
||||||
|
|
||||||
- **New plugins**: Discovered plugins not in state file are added with `active: false`
|
|
||||||
- **Removed plugins**: Plugins in state but not on disk are kept (preserves config for reinstall)
|
|
||||||
- **Config merge**: Default values from manifest are merged with saved config
|
|
||||||
- **Persistence**: State is saved immediately on any change
|
|
||||||
|
|
||||||
## Execution Pipeline
|
|
||||||
|
|
||||||
### Startup Sequence
|
|
||||||
|
|
||||||
```
|
|
||||||
1. PluginManager.scanPlugins()
|
|
||||||
└── Scan plugins/ directory
|
|
||||||
└── Parse each manifest.yml
|
|
||||||
└── Validate manifest structure
|
|
||||||
|
|
||||||
2. PluginManager.loadAll()
|
|
||||||
└── Load plugins.json state
|
|
||||||
└── For each discovered plugin:
|
|
||||||
├── Dynamic import(plugin/index.js)
|
|
||||||
├── Validate Plugin interface
|
|
||||||
├── Call plugin.onLoad()
|
|
||||||
└── If active in state:
|
|
||||||
└── Call plugin.onActivate()
|
|
||||||
└── Resolve execution order (topological sort)
|
|
||||||
```
|
|
||||||
|
|
||||||
### File Processing Sequence
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Create KVStore for file
|
|
||||||
|
|
||||||
2. For each active plugin (in dependency order):
|
|
||||||
├── Create PluginContext
|
|
||||||
├── Start timing
|
|
||||||
├── try:
|
|
||||||
│ └── await plugin.process(ctx)
|
|
||||||
└── catch:
|
|
||||||
└── Log error, continue to next plugin
|
|
||||||
|
|
||||||
3. Persist KV to Redis
|
|
||||||
```
|
|
||||||
|
|
||||||
### Activation Sequence (Hot)
|
|
||||||
|
|
||||||
```
|
|
||||||
1. PluginManager.activate(pluginId)
|
|
||||||
├── Validate plugin exists and is loaded
|
|
||||||
├── Check dependencies are active
|
|
||||||
├── Call plugin.onActivate()
|
|
||||||
├── Mark as active in memory
|
|
||||||
├── Rebuild execution order
|
|
||||||
├── Save plugins.json
|
|
||||||
└── Emit 'plugin:activated' event
|
|
||||||
|
|
||||||
2. Caller receives event
|
|
||||||
└── Trigger library rescan
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deactivation Sequence (Hot)
|
|
||||||
|
|
||||||
```
|
|
||||||
1. PluginManager.deactivate(pluginId)
|
|
||||||
├── Check no active plugins depend on this one
|
|
||||||
├── Call plugin.onDeactivate()
|
|
||||||
├── Mark as inactive in memory
|
|
||||||
├── Rebuild execution order
|
|
||||||
├── Save plugins.json
|
|
||||||
└── Emit 'plugin:deactivated' event
|
|
||||||
|
|
||||||
2. Caller receives event
|
|
||||||
└── Trigger library rescan
|
|
||||||
|
|
||||||
Note: Metadata written by the plugin remains in Redis.
|
|
||||||
A separate "clean metadata" feature can remove orphaned keys.
|
|
||||||
```
|
|
||||||
|
|
||||||
## Dependency Resolution
|
|
||||||
|
|
||||||
Plugins declare dependencies in their manifest. The system performs topological sort to determine execution order.
|
|
||||||
|
|
||||||
### Rules
|
|
||||||
|
|
||||||
1. A plugin cannot be activated if any of its dependencies are inactive
|
|
||||||
2. A plugin cannot be deactivated if any active plugin depends on it
|
|
||||||
3. Circular dependencies are detected and rejected at load time
|
|
||||||
4. Missing dependencies are logged as warnings; plugin can still load but may fail
|
|
||||||
|
|
||||||
### Example
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# filename-parser depends on file-info
|
|
||||||
# tmdb depends on filename-parser
|
|
||||||
# anime-detector depends on filename-parser
|
|
||||||
|
|
||||||
Resolved order: file-info → filename-parser → [tmdb, anime-detector]
|
|
||||||
```
|
|
||||||
|
|
||||||
Plugins at the same dependency level may execute in any order (or parallel in future).
|
|
||||||
|
|
||||||
## REST API
|
|
||||||
|
|
||||||
The plugin system exposes REST endpoints for dashboard integration.
|
|
||||||
|
|
||||||
### Endpoints
|
|
||||||
|
|
||||||
```
|
|
||||||
GET /api/plugins # List all plugins
|
|
||||||
GET /api/plugins/:id # Get plugin details
|
|
||||||
POST /api/plugins/:id/activate # Activate plugin
|
|
||||||
POST /api/plugins/:id/deactivate # Deactivate plugin
|
|
||||||
PUT /api/plugins/:id/config # Update plugin config
|
|
||||||
DELETE /api/plugins/:id/cache # Clear plugin cache
|
|
||||||
GET /api/plugins/:id/schema # Get metadata schema (for editor)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Response Format
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// GET /api/plugins
|
|
||||||
interface PluginsResponse {
|
|
||||||
plugins: PluginInfo[];
|
|
||||||
executionOrder: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
// POST /api/plugins/:id/activate
|
|
||||||
interface ActivateResponse {
|
|
||||||
success: boolean;
|
|
||||||
error?: string; # e.g., "Dependency 'x' is not active"
|
|
||||||
}
|
|
||||||
|
|
||||||
// PUT /api/plugins/:id/config
|
|
||||||
interface ConfigUpdateRequest {
|
|
||||||
config: Record<string, unknown>;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Error Handling
|
|
||||||
|
|
||||||
### Plugin Load Errors
|
|
||||||
|
|
||||||
- Invalid manifest: Plugin marked as `status: 'error'`, logged, skipped
|
|
||||||
- Import failure: Plugin marked as `status: 'error'`, logged, skipped
|
|
||||||
- Missing dependencies: Warning logged, plugin can load but activation may fail
|
|
||||||
|
|
||||||
### Plugin Execution Errors
|
|
||||||
|
|
||||||
- Errors in `process()` are caught, logged, and execution continues to next plugin
|
|
||||||
- The file's metadata from previous plugins is preserved
|
|
||||||
- Error count is tracked per plugin for monitoring
|
|
||||||
|
|
||||||
### Lifecycle Hook Errors
|
|
||||||
|
|
||||||
- Errors in `onActivate()` prevent activation, error returned to caller
|
|
||||||
- Errors in `onDeactivate()` are logged but deactivation proceeds
|
|
||||||
- Errors in `onLoad()`/`onUnload()` are logged, plugin marked as error state
|
|
||||||
|
|
||||||
## Future Considerations
|
|
||||||
|
|
||||||
### Metadata Cleanup
|
|
||||||
|
|
||||||
A future feature will allow cleaning metadata keys written by a specific plugin:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Future API
|
|
||||||
await pluginManager.cleanMetadata(pluginId);
|
|
||||||
// Removes all keys matching "{pluginId}/*" from Redis
|
|
||||||
```
|
|
||||||
|
|
||||||
### Parallel Execution
|
|
||||||
|
|
||||||
Plugins at the same dependency level could execute in parallel:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Future: parallel execution of independent plugins
|
|
||||||
const level1 = ['file-info'];
|
|
||||||
const level2 = ['ffmpeg', 'filename-parser']; // parallel
|
|
||||||
const level3 = ['tmdb', 'anime-detector']; // parallel
|
|
||||||
```
|
|
||||||
|
|
||||||
### Plugin Hot Reload
|
|
||||||
|
|
||||||
Reloading a plugin's code without full restart (for development):
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Future API
|
|
||||||
await pluginManager.reload(pluginId);
|
|
||||||
// Re-imports plugin code, preserves state
|
|
||||||
```
|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -35,7 +35,7 @@ lazyValue().then(value => console.log(value));
|
|||||||
### Listener Cleaner
|
### Listener Cleaner
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
import { ListenerCleaner } from "worph/async-utils";
|
import { ListenerCleaner } from "@worph/async-utils";
|
||||||
|
|
||||||
const cleaner = new ListenerCleaner();
|
const cleaner = new ListenerCleaner();
|
||||||
cleaner.add(() => console.log("Cleanup action"));
|
cleaner.add(() => console.log("Cleanup action"));
|
||||||
|
|||||||
Reference in New Issue
Block a user