This commit is contained in:
Worph
@@ -0,0 +1,6 @@
|
||||
node_modules
|
||||
dist
|
||||
.git
|
||||
.pnpm-store
|
||||
test/fixtures/*.mkv
|
||||
test/fixtures/*.mp4
|
||||
@@ -3,3 +3,8 @@ dist/
|
||||
*.log
|
||||
.env
|
||||
.pnpm-store
|
||||
|
||||
# Test fixtures are generated by test/fixtures/generate-test-fixtures.sh
|
||||
# (run automatically in Dockerfile.test) — no need to carry binaries in git.
|
||||
test/fixtures/*.mkv
|
||||
test/fixtures/*.mp4
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# MetaMesh Plugin: subtitle-extractor - Test Image
|
||||
# Builds the plugin and runs tests with ffmpeg available
|
||||
|
||||
FROM node:20-slim
|
||||
|
||||
# ffmpeg for fixture generation and the extraction tests
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files and install ALL dependencies (including devDependencies)
|
||||
COPY package.json tsconfig.json vitest.config.ts ./
|
||||
RUN npm install
|
||||
|
||||
# Copy source and test files
|
||||
COPY src/ ./src/
|
||||
COPY test/ ./test/
|
||||
|
||||
# Build the plugin
|
||||
RUN npm run build
|
||||
|
||||
# Generate test fixtures
|
||||
RUN chmod +x ./test/fixtures/generate-test-fixtures.sh && \
|
||||
./test/fixtures/generate-test-fixtures.sh
|
||||
|
||||
# Temp dir ffmpeg writes tracks into before upload
|
||||
RUN mkdir -p /cache/temp && chmod 777 /cache
|
||||
|
||||
ENV NODE_ENV=test
|
||||
ENV CACHE_PATH=/cache
|
||||
|
||||
CMD ["npm", "test"]
|
||||
@@ -1,157 +1,58 @@
|
||||
# MetaMesh Plugin: Subtitle Extractor
|
||||
|
||||
Extracts embedded subtitles from video files and saves them as standalone text files. The extracted subtitles are republished in the MetaMesh pipeline and linked to the original video via CID.
|
||||
Extracts a video's embedded **text** subtitle streams into standalone files and links them to the video so clients (meta-watch) can offer them.
|
||||
|
||||
## Features
|
||||
## What it writes (METADATA_KEYS.md §8 / §9)
|
||||
|
||||
- **Automatic extraction**: Extracts all text-based subtitle tracks from video files
|
||||
- **Multiple formats**: Supports SRT, ASS, SSA, WebVTT, and MOV_TEXT codecs
|
||||
- **Smart filtering**: Automatically skips image-based subtitles (PGS, DVD, DVB) that cannot be converted to text
|
||||
- **Language preservation**: Maintains language metadata from video streams
|
||||
- **Pipeline integration**: Extracted files are saved to `/output` and automatically picked up by meta-sort
|
||||
- **CID linking**: Stores subtitle CIDs as metadata on the source video
|
||||
On the **video record**, per extracted file `<subCid>` (midhash256 of its bytes) in `<lang3>` (ISO 639-2/B — `fre`, `ger`, `chi` — normalised the same way meta-watch does; `und` when the stream declares no language):
|
||||
|
||||
## Dependencies
|
||||
| Key | Value |
|
||||
|-----|-------|
|
||||
| `subtitles/<lang3>/<subCid>` | `"true"` — the leaf meta-watch lists |
|
||||
| `extractedSubtitles/<lang3>/<subCid>` | `"true"` — provenance: came out of the container |
|
||||
| `subtitleLanguages/<lang3>` | `"true"` (not for `und`) |
|
||||
| `languages/<lang3>` | `"true"` (not for `und`) |
|
||||
|
||||
This plugin requires the following plugins to run first:
|
||||
- `file-info` - Determines file type
|
||||
- `ffmpeg` - Provides subtitle stream information
|
||||
On the **subtitle file's own record**: `videos/<videoCid> = "true"` and `subtitleLanguage = <lang3>` (not for `und`).
|
||||
|
||||
## Installation
|
||||
Writes are a single `PATCH /meta/{cid}` carrying only keys the record does not already hold; a failed write fails the task. Legacy comma-joined scalars written by the old version (`extractedSubtitles`, `subtitleLanguages`, `subtitles`) are deleted first — a scalar next to `field/...` keys breaks meta-sort's nested document.
|
||||
|
||||
### Build from source
|
||||
## Behaviour
|
||||
|
||||
```bash
|
||||
cd packages/plugins/metamesh-plugin-subtitle-extractor
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Build Docker image
|
||||
|
||||
```bash
|
||||
docker build -t metamesh-plugin-subtitle-extractor:main .
|
||||
```
|
||||
|
||||
### Configure in plugins.yml
|
||||
|
||||
Add to `dev/plugins.yml`:
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
subtitle-extractor:
|
||||
enabled: true
|
||||
image: metamesh-plugin-subtitle-extractor:main
|
||||
instances: 1
|
||||
resources:
|
||||
memory: 256m
|
||||
cpus: 0.5
|
||||
```
|
||||
- Reads the ffmpeg plugin's stream table in any shape: meta-sort's nested `stream` array (JSON strings or objects), an index-keyed object, a JSON string, or meta-core's flat `stream/{n}`.
|
||||
- **One ffmpeg pass** extracts every text track (one `-map`/output per track). Subtitle packets are interleaved through the whole container, so that pass is still one full read of the video over WebDAV — but one, not one per track. If the shared pass fails, tracks are retried one by one.
|
||||
- Image codecs (PGS, VobSub, DVB, teletext, XSUB) are skipped explicitly; other unknown codecs are skipped and logged.
|
||||
- Skips a video that already carries the `extractedSubtitles` key-set (unless `forceRecompute`), and reuses an output file that already exists instead of re-reading the video.
|
||||
- Output: `/files/plugin/subtitle-extractor/<Title> (<Year>)[<videoCid>]_subtitle.s<streamIndex>.<lang3>[.forced].<ext>` — the stream index keeps two tracks in one language apart.
|
||||
- WebDAV endpoint resolved per request from the driving meta-core (`/urls` → `webdavUrlInternal`); `WEBDAV_URL` overrides.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `forceRecompute` | boolean | `false` | Re-extract subtitles even if already processed |
|
||||
| `outputFormat` | select | `srt` | Output format: `srt`, `vtt`, or `ass` |
|
||||
| `forceRecompute` | boolean | `false` | Re-extract even if already processed |
|
||||
| `outputFormat` | select | `native` | `native` keeps ASS as ASS (stream copy) and SubRip/WebVTT as-is, converts mov_text to SRT; `srt`, `vtt`, `ass` convert everything |
|
||||
| `extractionTimeoutMs` | number | `270000` | Timeout of the single ffmpeg pass |
|
||||
|
||||
## Output
|
||||
## Supported codecs
|
||||
|
||||
### Extracted Files
|
||||
Text (extracted): `subrip`/`srt`, `ass`, `ssa`, `webvtt`, `mov_text`, `text`.
|
||||
Image (skipped): `hdmv_pgs_subtitle`, `dvd_subtitle`, `dvb_subtitle`, `dvb_teletext`, `xsub`.
|
||||
|
||||
Subtitle files are saved to `/output` with the naming pattern:
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pnpm install && pnpm test # locally (ffmpeg tests need ffmpeg + ./test/fixtures/generate-test-fixtures.sh)
|
||||
./test.sh # in Docker, fixtures generated in the image
|
||||
```
|
||||
{Title} ({Year})[{VideoCID}]_subtitle.{lang}.{ext}
|
||||
```
|
||||
|
||||
Example: `Sintel (2010)[bafk...abc]_subtitle.eng.srt`
|
||||
|
||||
### Metadata
|
||||
|
||||
The plugin stores the following metadata on the source video:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `extractedSubtitles` | array | CIDs of extracted subtitle files |
|
||||
| `subtitleLanguages` | array | Language codes of extracted subtitles |
|
||||
|
||||
## Supported Codecs
|
||||
|
||||
### Text-based (supported)
|
||||
- `subrip` / `srt` - SubRip
|
||||
- `ass` - Advanced SubStation Alpha
|
||||
- `ssa` - SubStation Alpha
|
||||
- `webvtt` - WebVTT
|
||||
- `mov_text` - QuickTime text
|
||||
|
||||
### Image-based (skipped)
|
||||
- `hdmv_pgs_subtitle` - Blu-ray PGS
|
||||
- `dvd_subtitle` - DVD VOB
|
||||
- `dvb_subtitle` - DVB
|
||||
- `xsub` - DivX XSUB
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Video File (MKV/MP4/AVI)
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ ffmpeg plugin │ ← Provides stream info
|
||||
└───────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ subtitle-extractor│
|
||||
│ │
|
||||
│ 1. Parse streams │
|
||||
│ 2. Filter codecs │
|
||||
│ 3. Extract via │
|
||||
│ ffmpeg │
|
||||
│ 4. Save to │
|
||||
│ /output │
|
||||
│ 5. Compute CID │
|
||||
│ 6. Store in │
|
||||
│ Redis │
|
||||
└───────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ /output folder │ ← Watched by meta-sort
|
||||
│ *.srt files │
|
||||
└───────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────┐
|
||||
│ meta-sort │ ← Processes extracted files
|
||||
│ file watcher │ like any other media
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
## Container Mounts
|
||||
|
||||
| Mount | Access | Purpose |
|
||||
|-------|--------|---------|
|
||||
| `/files` | READ-ONLY | Source video files |
|
||||
| `/cache` | READ-WRITE | Plugin cache |
|
||||
| `/output` | READ-WRITE | Extracted subtitle output |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `PORT` | HTTP server port (default: 8080) |
|
||||
| `WEBDAV_URL` | WebDAV base URL for file access |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/health` | GET | Health check |
|
||||
| `/manifest` | GET | Plugin manifest |
|
||||
| `/configure` | POST | Update configuration |
|
||||
| `/process` | POST | Process a video file |
|
||||
| `WEBDAV_URL` | Optional WebDAV override (otherwise resolved from the meta-core `/urls`) |
|
||||
| `CACHE_PATH` | Temp root (default `/cache`; files go to `$CACHE_PATH/temp`) |
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+5
-2
@@ -7,13 +7,16 @@
|
||||
"main": "dist/index.js",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"start": "node dist/index.js"
|
||||
"start": "node dist/index.js",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest"
|
||||
},
|
||||
"dependencies": {
|
||||
"fastify": "^5.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^20.0.0",
|
||||
"typescript": "^5.3.0"
|
||||
"typescript": "^5.3.0",
|
||||
"vitest": "^1.6.1"
|
||||
}
|
||||
}
|
||||
|
||||
Generated
+1088
File diff suppressed because it is too large
Load Diff
+170
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* Language code normalisation — the `lang3` vocabulary meta-watch reads.
|
||||
*
|
||||
* Port of meta-watch's `MW_LANG.toLang3` (packages/meta-watch/ui/js/lang-codes.js),
|
||||
* the same normaliser the `language` plugin copied. Every spelling folds to
|
||||
* **ISO 639-2/B** — `fre` not `fra`, `ger` not `deu`, `chi` not `zho` — because
|
||||
* that is what meta-watch writes (`subtitles/fre/<cid>`, routes/subtitles.rs)
|
||||
* and what its readers compare against:
|
||||
*
|
||||
* ISO 639-1 "fr" -> fre
|
||||
* ISO 639-2/B "fre" -> fre
|
||||
* ISO 639-2/T "fra" -> fre (MP4 `mdhd` speaks T, Matroska speaks B)
|
||||
* BCP-47 "pt-BR" -> por (primary subtag only)
|
||||
*
|
||||
* One deliberate difference: "no language" is `und`, not "" — the code becomes a
|
||||
* key segment (`subtitles/und/<cid>`, METADATA_KEYS.md §8) and `und` is the
|
||||
* registry's spelling for undetermined.
|
||||
*
|
||||
* Copied, not imported: plugins are independent submodules. This file is
|
||||
* byte-identical in metamesh-plugin-subtitle and metamesh-plugin-subtitle-extractor.
|
||||
*/
|
||||
|
||||
// Canonical table keyed by 639-2/B: display name (lowercased for filename
|
||||
// matching) and the 639-1 code that folds into it.
|
||||
const LANGS: Record<string, { name: string; a1?: string }> = {
|
||||
eng: { name: 'english', a1: 'en' },
|
||||
fre: { name: 'french', a1: 'fr' },
|
||||
ger: { name: 'german', a1: 'de' },
|
||||
spa: { name: 'spanish', a1: 'es' },
|
||||
ita: { name: 'italian', a1: 'it' },
|
||||
por: { name: 'portuguese', a1: 'pt' },
|
||||
dut: { name: 'dutch', a1: 'nl' },
|
||||
swe: { name: 'swedish', a1: 'sv' },
|
||||
nor: { name: 'norwegian', a1: 'no' },
|
||||
dan: { name: 'danish', a1: 'da' },
|
||||
fin: { name: 'finnish', a1: 'fi' },
|
||||
ice: { name: 'icelandic', a1: 'is' },
|
||||
pol: { name: 'polish', a1: 'pl' },
|
||||
cze: { name: 'czech', a1: 'cs' },
|
||||
slo: { name: 'slovak', a1: 'sk' },
|
||||
slv: { name: 'slovenian', a1: 'sl' },
|
||||
hun: { name: 'hungarian', a1: 'hu' },
|
||||
rum: { name: 'romanian', a1: 'ro' },
|
||||
bul: { name: 'bulgarian', a1: 'bg' },
|
||||
rus: { name: 'russian', a1: 'ru' },
|
||||
ukr: { name: 'ukrainian', a1: 'uk' },
|
||||
bel: { name: 'belarusian', a1: 'be' },
|
||||
srp: { name: 'serbian', a1: 'sr' },
|
||||
hrv: { name: 'croatian', a1: 'hr' },
|
||||
bos: { name: 'bosnian', a1: 'bs' },
|
||||
mac: { name: 'macedonian', a1: 'mk' },
|
||||
alb: { name: 'albanian', a1: 'sq' },
|
||||
gre: { name: 'greek', a1: 'el' },
|
||||
tur: { name: 'turkish', a1: 'tr' },
|
||||
heb: { name: 'hebrew', a1: 'he' },
|
||||
ara: { name: 'arabic', a1: 'ar' },
|
||||
per: { name: 'persian', a1: 'fa' },
|
||||
urd: { name: 'urdu', a1: 'ur' },
|
||||
hin: { name: 'hindi', a1: 'hi' },
|
||||
ben: { name: 'bengali', a1: 'bn' },
|
||||
tam: { name: 'tamil', a1: 'ta' },
|
||||
tel: { name: 'telugu', a1: 'te' },
|
||||
mal: { name: 'malayalam', a1: 'ml' },
|
||||
kan: { name: 'kannada', a1: 'kn' },
|
||||
mar: { name: 'marathi', a1: 'mr' },
|
||||
guj: { name: 'gujarati', a1: 'gu' },
|
||||
pan: { name: 'punjabi', a1: 'pa' },
|
||||
nep: { name: 'nepali', a1: 'ne' },
|
||||
sin: { name: 'sinhala', a1: 'si' },
|
||||
tha: { name: 'thai', a1: 'th' },
|
||||
lao: { name: 'lao', a1: 'lo' },
|
||||
khm: { name: 'khmer', a1: 'km' },
|
||||
bur: { name: 'burmese', a1: 'my' },
|
||||
vie: { name: 'vietnamese', a1: 'vi' },
|
||||
ind: { name: 'indonesian', a1: 'id' },
|
||||
may: { name: 'malay', a1: 'ms' },
|
||||
tgl: { name: 'tagalog', a1: 'tl' },
|
||||
fil: { name: 'filipino' },
|
||||
jpn: { name: 'japanese', a1: 'ja' },
|
||||
kor: { name: 'korean', a1: 'ko' },
|
||||
chi: { name: 'chinese', a1: 'zh' },
|
||||
mon: { name: 'mongolian', a1: 'mn' },
|
||||
kaz: { name: 'kazakh', a1: 'kk' },
|
||||
uzb: { name: 'uzbek', a1: 'uz' },
|
||||
aze: { name: 'azerbaijani', a1: 'az' },
|
||||
geo: { name: 'georgian', a1: 'ka' },
|
||||
arm: { name: 'armenian', a1: 'hy' },
|
||||
est: { name: 'estonian', a1: 'et' },
|
||||
lav: { name: 'latvian', a1: 'lv' },
|
||||
lit: { name: 'lithuanian', a1: 'lt' },
|
||||
cat: { name: 'catalan', a1: 'ca' },
|
||||
glg: { name: 'galician', a1: 'gl' },
|
||||
baq: { name: 'basque', a1: 'eu' },
|
||||
wel: { name: 'welsh', a1: 'cy' },
|
||||
gle: { name: 'irish', a1: 'ga' },
|
||||
afr: { name: 'afrikaans', a1: 'af' },
|
||||
swa: { name: 'swahili', a1: 'sw' },
|
||||
amh: { name: 'amharic', a1: 'am' },
|
||||
zul: { name: 'zulu', a1: 'zu' },
|
||||
lat: { name: 'latin', a1: 'la' },
|
||||
epo: { name: 'esperanto', a1: 'eo' },
|
||||
yid: { name: 'yiddish', a1: 'yi' },
|
||||
jav: { name: 'javanese', a1: 'jv' },
|
||||
mul: { name: 'multiple' },
|
||||
};
|
||||
|
||||
// ISO 639-2/T -> 639-2/B: the only codes where the two standards disagree.
|
||||
const T2B: Record<string, string> = {
|
||||
sqi: 'alb', hye: 'arm', eus: 'baq', mya: 'bur', ces: 'cze',
|
||||
zho: 'chi', cym: 'wel', deu: 'ger', nld: 'dut', ell: 'gre', fas: 'per',
|
||||
fra: 'fre', kat: 'geo', isl: 'ice', mkd: 'mac', msa: 'may',
|
||||
ron: 'rum', slk: 'slo',
|
||||
};
|
||||
|
||||
// "This track has no language" spellings.
|
||||
const NONE = new Set(['und', 'zxx', 'mis', 'unknown', 'none']);
|
||||
|
||||
// 639-1 -> 639-2/B, from the table plus legacy two-letter codes.
|
||||
const A1: Record<string, string> = { nb: 'nor', nn: 'nor', iw: 'heb', in: 'ind', ji: 'yid', jw: 'jav' };
|
||||
const NAMES: Record<string, string> = {};
|
||||
for (const [b, entry] of Object.entries(LANGS)) {
|
||||
if (entry.a1) A1[entry.a1] = b;
|
||||
NAMES[entry.name] = b;
|
||||
}
|
||||
|
||||
/**
|
||||
* franc-min answers in ISO 639-3 individual-language codes; a few of them are
|
||||
* not 639-2 codes at all (Mandarin `cmn` is 639-2 `chi`). Folded before toLang3.
|
||||
*/
|
||||
const FRANC_TO_639_2: Record<string, string> = {
|
||||
cmn: 'chi', arb: 'ara', pes: 'per', swh: 'swa', uzn: 'uzb', zlm: 'may',
|
||||
npi: 'nep', azj: 'aze', plt: 'mlg', pbu: 'pus', ckb: 'kur', fuv: 'ful', qug: 'que',
|
||||
};
|
||||
|
||||
/** Canonical lang3 (ISO 639-2/B) for any code; `und` when there is none. */
|
||||
export function toLang3(code: unknown): string {
|
||||
const s = String(code ?? '').trim().toLowerCase();
|
||||
if (!s) return 'und';
|
||||
if (NAMES[s]) return NAMES[s];
|
||||
const p = s.split(/[-_]/)[0];
|
||||
if (NONE.has(p)) return 'und';
|
||||
if (p.length === 2) return A1[p] ?? 'und';
|
||||
if (p.length === 3 && /^[a-z]{3}$/.test(p)) return T2B[p] ?? p;
|
||||
return 'und';
|
||||
}
|
||||
|
||||
/** lang3 for a franc-min result (`und` stays `und`). */
|
||||
export function francToLang3(code: string): string {
|
||||
return toLang3(FRANC_TO_639_2[code] ?? code);
|
||||
}
|
||||
|
||||
/**
|
||||
* Strict variant for filename tokens (`video.fr.srt`, `video.English.srt`):
|
||||
* only a token that is recognisably a language resolves, so `S01`, `x264` or
|
||||
* `Extended` never read as one. Returns null for anything else.
|
||||
*/
|
||||
export function languageToken(token: string): string | null {
|
||||
const t = token.trim().toLowerCase();
|
||||
if (!t) return null;
|
||||
if (NAMES[t]) return NAMES[t];
|
||||
if (t === 'und') return 'und';
|
||||
const p = t.split(/[-_]/)[0];
|
||||
if (p !== t && !/^[a-z]{2,3}[-_][a-z0-9]{2,8}$/.test(t)) return null;
|
||||
if (p.length === 2) return A1[p] ?? null;
|
||||
if (p.length === 3) {
|
||||
if (LANGS[p]) return p;
|
||||
if (T2B[p]) return T2B[p];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
+39
-64
@@ -1,6 +1,10 @@
|
||||
/**
|
||||
* meta-core API Client for writing metadata
|
||||
* Fails gracefully if meta-core is unavailable (logs warning, returns without throwing)
|
||||
* meta-core API client for writing metadata.
|
||||
*
|
||||
* Writes report success so a failed write fails the task instead of being
|
||||
* swallowed. Key-sets are written as individual `prefix/member = "true"`
|
||||
* fields through PATCH — never via `_add`, which stores one comma-joined
|
||||
* string and can never produce the key-set shape.
|
||||
*/
|
||||
|
||||
export class MetaCoreClient {
|
||||
@@ -8,86 +12,57 @@ export class MetaCoreClient {
|
||||
|
||||
private async safeFetch(url: string, options: RequestInit): Promise<Response | null> {
|
||||
try {
|
||||
return await fetch(url, { ...options, signal: AbortSignal.timeout(5000) });
|
||||
return await fetch(url, { ...options, signal: AbortSignal.timeout(10000) });
|
||||
} catch (error) {
|
||||
console.warn(`[MetaCoreClient] Warning: meta-core unavailable at ${this.baseUrl}`);
|
||||
console.warn(`[MetaCoreClient] Warning: meta-core unavailable at ${this.baseUrl}: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async setProperty(hashId: string, key: string, value: string): Promise<void> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}/${key}`, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ value }),
|
||||
});
|
||||
if (response && !response.ok) {
|
||||
console.warn(`[MetaCoreClient] Failed to set property: ${response.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async getProperty(hashId: string, key: string): Promise<string | null> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}/${key}`, { method: 'GET' });
|
||||
if (!response || response.status === 404) return null;
|
||||
if (!response.ok) return null;
|
||||
const data = await response.json() as { value?: string };
|
||||
return data.value ?? null;
|
||||
if (!response || !response.ok) return null;
|
||||
const data = await response.json() as { value?: unknown };
|
||||
return data.value === undefined || data.value === null ? null : String(data.value);
|
||||
}
|
||||
|
||||
async mergeMetadata(hashId: string, metadata: Record<string, string>): Promise<void> {
|
||||
/**
|
||||
* Whole record, in whatever form the endpoint serves (meta-sort's /meta is
|
||||
* nested, meta-core's is flat) — callers flatten. `null` when the call
|
||||
* itself failed, `{}` when the record does not exist.
|
||||
*/
|
||||
async getMetadata(hashId: string): Promise<Record<string, unknown> | null> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}`, { method: 'GET' });
|
||||
if (!response) return null;
|
||||
if (response.status === 404) return {};
|
||||
if (!response.ok) return null;
|
||||
const data = await response.json() as { metadata?: Record<string, unknown> };
|
||||
return data.metadata ?? {};
|
||||
}
|
||||
|
||||
/** PATCH — merge `metadata` into the record. True on success. */
|
||||
async mergeMetadata(hashId: string, metadata: Record<string, string>): Promise<boolean> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}`, {
|
||||
method: 'PATCH',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(metadata),
|
||||
});
|
||||
if (response && !response.ok) {
|
||||
console.warn(`[MetaCoreClient] Failed to merge metadata: ${response.status}`);
|
||||
if (!response) return false;
|
||||
if (!response.ok) {
|
||||
console.warn(`[MetaCoreClient] Failed to merge metadata into ${hashId}: ${response.status}`);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
async deleteProperty(hashId: string, key: string): Promise<void> {
|
||||
/** DELETE one field. A missing field counts as deleted. */
|
||||
async deleteProperty(hashId: string, key: string): Promise<boolean> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}/${key}`, { method: 'DELETE' });
|
||||
if (response && !response.ok && response.status !== 404) {
|
||||
console.warn(`[MetaCoreClient] Failed to delete property: ${response.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async addToSet(hashId: string, key: string, value: string): Promise<void> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}/_add/${key}`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ value }),
|
||||
});
|
||||
if (response && !response.ok) {
|
||||
console.warn(`[MetaCoreClient] Failed to add to set: ${response.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async getMetadata(hashId: string): Promise<Record<string, string>> {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/meta/${hashId}`, { method: 'GET' });
|
||||
if (!response || response.status === 404) return {};
|
||||
if (!response.ok) return {};
|
||||
const data = await response.json() as { metadata?: Record<string, string> };
|
||||
return data.metadata ?? {};
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the CID (Content Identifier) for a file
|
||||
* Uses the meta-core /file/cid API endpoint
|
||||
*/
|
||||
async computeFileCID(filePath: string): Promise<string | null> {
|
||||
try {
|
||||
const response = await this.safeFetch(`${this.baseUrl}/file/cid`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ path: filePath }),
|
||||
});
|
||||
if (!response || !response.ok) return null;
|
||||
const data = await response.json() as { cid?: string };
|
||||
return data.cid ?? null;
|
||||
} catch (error) {
|
||||
console.debug(`[MetaCoreClient] Failed to compute CID for ${filePath}:`, error);
|
||||
return null;
|
||||
if (!response) return false;
|
||||
if (!response.ok && response.status !== 404) {
|
||||
console.warn(`[MetaCoreClient] Failed to delete ${hashId}/${key}: ${response.status}`);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* Reading a record regardless of the shape it arrives in, and writing only what
|
||||
* is new.
|
||||
*
|
||||
* meta-sort's scheduler fills `existingMeta` from meta-sort's own GET /meta/{hash},
|
||||
* which returns the NESTED document (`fileinfo: {duration: 1433.089}`,
|
||||
* `stream: [...]`, a key-set as `subtitles: {fre: {<cid>: true}}`). meta-core's
|
||||
* backend serves the FLAT form (`fileinfo/duration`, `stream/0`,
|
||||
* `subtitles/fre/<cid>`). A plugin that reads `existingMeta['x/y']` silently sees
|
||||
* nothing in production — so everything here goes through `flattenMeta`.
|
||||
*
|
||||
* Byte-identical in metamesh-plugin-subtitle and metamesh-plugin-subtitle-extractor.
|
||||
*/
|
||||
|
||||
/** Flatten either form into `a/b/c -> string` (meta-sort's flattenMetadata, plus array indices). */
|
||||
export function flattenMeta(meta: Record<string, unknown> | undefined | null): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
const walk = (value: unknown, key: string) => {
|
||||
if (value === null || value === undefined) return;
|
||||
if (Array.isArray(value)) {
|
||||
value.forEach((v, i) => walk(v, key ? `${key}/${i}` : String(i)));
|
||||
} else if (typeof value === 'object') {
|
||||
for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
|
||||
walk(v, key ? `${key}/${k}` : k);
|
||||
}
|
||||
} else if (key) {
|
||||
out[key] = String(value);
|
||||
}
|
||||
};
|
||||
walk(meta ?? {}, '');
|
||||
return out;
|
||||
}
|
||||
|
||||
/** The subset of `wanted` the record does not already hold with that exact value. */
|
||||
export function onlyNewKeys(existingFlat: Record<string, string>, wanted: Record<string, string>): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(wanted)) {
|
||||
if (existingFlat[k] !== v) out[k] = v;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Members of a LEGACY scalar csv-set (`subtitles = "cidA,cidB"`, written by the
|
||||
* old `_add` path), or null when the field is absent or already a key-set.
|
||||
*
|
||||
* Such a scalar must be deleted before any `field/...` key is written: meta-sort
|
||||
* rebuilds its nested document by walking the `/`-segments, and a string sitting
|
||||
* where an object is needed either throws or silently hides the key-set.
|
||||
*/
|
||||
export function legacyCsvMembers(meta: Record<string, unknown> | undefined | null, field: string): string[] | null {
|
||||
const v = meta?.[field];
|
||||
if (typeof v !== 'string' && typeof v !== 'number') return null;
|
||||
return String(v).split(/[,|]/).map((s) => s.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
/** Every `<prefix>/<lang>/<cid>` leaf for `cid` whose language is not `lang3`. */
|
||||
export function staleLanguageLeaves(existingFlat: Record<string, string>, prefix: string, cid: string, lang3: string): string[] {
|
||||
const stale: string[] = [];
|
||||
for (const key of Object.keys(existingFlat)) {
|
||||
const parts = key.split('/');
|
||||
if (parts.length === 3 && parts[0] === prefix && parts[2] === cid && parts[1] !== lang3) stale.push(key);
|
||||
}
|
||||
return stale;
|
||||
}
|
||||
+508
-343
@@ -1,125 +1,155 @@
|
||||
/**
|
||||
* Subtitle Extractor Plugin
|
||||
*
|
||||
* Extracts embedded subtitles from video files and saves them as text files.
|
||||
* Republishes extracted subtitles in the pipeline (similar to TMDB plugin with posters).
|
||||
* Extracts a video's embedded TEXT subtitle streams into standalone files under
|
||||
* /files/plugin/subtitle-extractor/ and links them to the video so meta-watch can
|
||||
* offer them.
|
||||
*
|
||||
* ============================================================================
|
||||
* PLUGIN FILE ACCESS ARCHITECTURE - WebDAV
|
||||
* WHAT IT WRITES (METADATA_KEYS.md §8 / §9)
|
||||
* ============================================================================
|
||||
* On the video record, per extracted file `<subCid>` (midhash256 of its bytes)
|
||||
* in language `<lang3>` (ISO 639-2/B, `und` when the stream declares none):
|
||||
* subtitles/<lang3>/<subCid> = "true" what meta-watch lists
|
||||
* extractedSubtitles/<lang3>/<subCid> = "true" provenance: came out of the container
|
||||
* subtitleLanguages/<lang3> = "true" (not for `und`)
|
||||
* languages/<lang3> = "true" (not for `und`)
|
||||
* On the subtitle file's own record:
|
||||
* videos/<videoCid> = "true"
|
||||
* subtitleLanguage = <lang3> (not for `und`; the `subtitle`
|
||||
* plugin's text sniff fills it)
|
||||
* Writes are one PATCH carrying only keys the record does not already hold.
|
||||
* Legacy scalar csv-sets from the old `_add` path (`extractedSubtitles`,
|
||||
* `subtitleLanguages`, `subtitles`) are deleted first; see meta-shape.ts.
|
||||
*
|
||||
* File access via WebDAV (WEBDAV_URL environment variable):
|
||||
* - Read media files: GET /webdav/watch/... or /webdav/test/...
|
||||
* - Write output: PUT /webdav/plugin/subtitle-extractor/...
|
||||
* - Cache: Local /cache mount (for temporary files during extraction)
|
||||
*
|
||||
* Benefits:
|
||||
* - No need for /files or /output volume mounts
|
||||
* - Works across multiple hosts
|
||||
* - Plugins are fully isolated
|
||||
* Image codecs (PGS, VobSub, DVB) are skipped: turning them into text is OCR.
|
||||
*
|
||||
* ============================================================================
|
||||
* READ COST
|
||||
* ============================================================================
|
||||
* Subtitle packets are interleaved through the whole container, so extracting
|
||||
* even one track means demuxing the file end to end — over WebDAV, one full
|
||||
* sequential read of the video. All text tracks therefore come out of ONE ffmpeg
|
||||
* pass with one output per track (one read, not N). A per-track retry happens
|
||||
* only when that pass fails outright (one broken track aborts every output).
|
||||
*
|
||||
* ============================================================================
|
||||
* FILE ACCESS - WebDAV
|
||||
* ============================================================================
|
||||
* The WebDAV endpoint is resolved per request from the meta-core that drove the
|
||||
* /process call (its /urls -> webdavUrlInternal; WEBDAV_URL overrides), so the
|
||||
* bytes land in the same core the CIDs are written to.
|
||||
* - Read media: GET /webdav/watch/...
|
||||
* - Write output: PUT /webdav/plugin/subtitle-extractor/...
|
||||
* - Temp: $CACHE_PATH/temp (default /cache/temp)
|
||||
*/
|
||||
|
||||
import { spawn } from 'child_process';
|
||||
import { existsSync, mkdirSync, statSync, unlinkSync, readFileSync } from 'fs';
|
||||
import { createHash } from 'crypto';
|
||||
import { existsSync, mkdirSync, readFileSync, unlinkSync } from 'fs';
|
||||
import * as path from 'path';
|
||||
import type { PluginManifest, ProcessRequest, CallbackPayload } from './types.js';
|
||||
import { MetaCoreClient } from './meta-core-client.js';
|
||||
import { createWebDAVClient } from './webdav-client.js';
|
||||
import { getWebDAVClient } from './webdav-client.js';
|
||||
import { toLang3 } from './langs.js';
|
||||
import { flattenMeta, onlyNewKeys, legacyCsvMembers, staleLanguageLeaves } from './meta-shape.js';
|
||||
|
||||
// Initialize WebDAV client if WEBDAV_URL is set
|
||||
const webdavClient = createWebDAVClient();
|
||||
if (webdavClient) {
|
||||
console.log('[subtitle-extractor] WebDAV client initialized for file access');
|
||||
} else {
|
||||
console.warn('[subtitle-extractor] WARNING: WEBDAV_URL not set - plugin will not be able to write output files');
|
||||
const PLUGIN_OUTPUT_WEBDAV_PATH = '/files/plugin/subtitle-extractor';
|
||||
|
||||
function tempDir(): string {
|
||||
return path.join(globalThis.process.env.CACHE_PATH || '/cache', 'temp');
|
||||
}
|
||||
|
||||
/**
|
||||
* ============================================================================
|
||||
* PLUGIN OUTPUT PATH - Written via WebDAV
|
||||
* ============================================================================
|
||||
* Output files (subtitles) are written via WebDAV PUT requests.
|
||||
* Path: /files/plugin/subtitle-extractor/<filename> (accessible at WEBDAV_URL/plugin/subtitle-extractor/)
|
||||
* TEMP_PATH (/cache/temp) - Local temp folder for ffmpeg extraction before WebDAV upload
|
||||
* ============================================================================
|
||||
*/
|
||||
const PLUGIN_OUTPUT_WEBDAV_PATH = '/files/plugin/subtitle-extractor';
|
||||
const TEMP_PATH = '/cache/temp';
|
||||
|
||||
// Image-based subtitle codecs that cannot be converted to text
|
||||
const UNSUPPORTED_SUBTITLE_CODECS = new Set([
|
||||
/** Bitmap subtitle codecs: no text to extract without OCR. */
|
||||
export const IMAGE_SUBTITLE_CODECS = new Set([
|
||||
'hdmv_pgs_subtitle',
|
||||
'pgssub',
|
||||
'dvd_subtitle',
|
||||
'dvdsub',
|
||||
'dvb_subtitle',
|
||||
'dvbsub',
|
||||
'dvb_teletext',
|
||||
'xsub',
|
||||
]);
|
||||
|
||||
// Supported text-based subtitle codecs
|
||||
const CODEC_EXTENSION_MAP: Record<string, string> = {
|
||||
'subrip': 'srt',
|
||||
'srt': 'srt',
|
||||
'ass': 'ass',
|
||||
'ssa': 'ssa',
|
||||
'webvtt': 'vtt',
|
||||
'mov_text': 'srt',
|
||||
'text': 'srt',
|
||||
type SubtitleExt = 'srt' | 'ass' | 'ssa' | 'vtt';
|
||||
|
||||
/** Text codecs we extract, and the file type each one naturally becomes. */
|
||||
const TEXT_SUBTITLE_CODECS: Record<string, SubtitleExt> = {
|
||||
subrip: 'srt',
|
||||
srt: 'srt',
|
||||
ass: 'ass',
|
||||
ssa: 'ssa',
|
||||
webvtt: 'vtt',
|
||||
mov_text: 'srt',
|
||||
text: 'srt',
|
||||
};
|
||||
|
||||
/** Codecs whose packets can be stream-copied into their natural file type. */
|
||||
const COPYABLE_CODECS = new Set(['subrip', 'srt', 'ass', 'ssa', 'webvtt']);
|
||||
|
||||
const ENCODER_FOR_EXT: Record<SubtitleExt, string> = { srt: 'srt', vtt: 'webvtt', ass: 'ass', ssa: 'ass' };
|
||||
|
||||
export type OutputFormat = 'native' | 'srt' | 'vtt' | 'ass';
|
||||
const OUTPUT_FORMATS = new Set<OutputFormat>(['native', 'srt', 'vtt', 'ass']);
|
||||
|
||||
export const manifest: PluginManifest = {
|
||||
id: 'subtitle-extractor',
|
||||
name: 'Subtitle Extractor',
|
||||
version: '1.0.0',
|
||||
description: 'Extracts embedded subtitles from video files and saves them as text files',
|
||||
version: '1.1.0',
|
||||
description: 'Extracts embedded text subtitles from video files and links them to the video',
|
||||
author: 'MetaMesh',
|
||||
// file-info gives `fileType`; ffmpeg gives the `stream` table.
|
||||
dependencies: ['file-info', 'ffmpeg'],
|
||||
priority: 50,
|
||||
color: '#FF5722',
|
||||
// One full read of the video per run. Never the fast queue.
|
||||
defaultQueue: 'background',
|
||||
timeout: 300000, // 5 minutes for large files
|
||||
timeout: 300000,
|
||||
schema: {
|
||||
extractedSubtitles: { label: 'Extracted Subtitles', type: 'array', readonly: true },
|
||||
subtitleLanguages: { label: 'Subtitle Languages', type: 'array', readonly: true },
|
||||
subtitles: { label: 'Subtitles', type: 'json', readonly: true, hint: 'subtitles/<lang3>/<cid>' },
|
||||
extractedSubtitles: { label: 'Extracted Subtitles', type: 'json', readonly: true, hint: 'extractedSubtitles/<lang3>/<cid>' },
|
||||
subtitleLanguages: { label: 'Subtitle Languages', type: 'json', readonly: true, hint: 'subtitleLanguages/<lang3>' },
|
||||
},
|
||||
config: {
|
||||
forceRecompute: {
|
||||
type: 'boolean',
|
||||
label: 'Force Recompute',
|
||||
default: false,
|
||||
},
|
||||
forceRecompute: { type: 'boolean', label: 'Force Recompute', default: false },
|
||||
outputFormat: {
|
||||
type: 'select',
|
||||
label: 'Output Format',
|
||||
default: 'srt',
|
||||
label: 'Output Format (native keeps ASS styling; srt/vtt/ass convert)',
|
||||
default: 'native',
|
||||
},
|
||||
extractionTimeoutMs: {
|
||||
type: 'number',
|
||||
label: 'Extraction timeout (ms) for the single ffmpeg pass',
|
||||
default: 270000,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
// Configuration
|
||||
let forceRecompute = false;
|
||||
let outputFormat = 'srt';
|
||||
let outputFormat: OutputFormat = 'native';
|
||||
let extractionTimeoutMs = 270000;
|
||||
|
||||
export function configure(config: Record<string, unknown>): void {
|
||||
forceRecompute = config.forceRecompute === true;
|
||||
outputFormat = (config.outputFormat as string) || 'srt';
|
||||
console.log(`[subtitle-extractor] Config: forceRecompute=${forceRecompute}, outputFormat=${outputFormat}`);
|
||||
const format = String(config.outputFormat ?? 'native') as OutputFormat;
|
||||
outputFormat = OUTPUT_FORMATS.has(format) ? format : 'native';
|
||||
const timeout = config.extractionTimeoutMs === null || config.extractionTimeoutMs === undefined
|
||||
? NaN
|
||||
: Number(config.extractionTimeoutMs);
|
||||
extractionTimeoutMs = Number.isFinite(timeout) && timeout > 0 ? timeout : 270000;
|
||||
console.log(`[subtitle-extractor] Config: forceRecompute=${forceRecompute}, outputFormat=${outputFormat}, extractionTimeoutMs=${extractionTimeoutMs}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute midhash256 CID from a Buffer (matches meta-hash algorithm)
|
||||
* Used for computing CID before/after WebDAV upload
|
||||
* midhash256 CID of a buffer (matches meta-hash's computeMidHash256Sync):
|
||||
* sha256(size_u64_be ‖ middle 1 MiB), wrapped as CIDv1 multibase base32.
|
||||
*/
|
||||
function computeMidHash256FromBuffer(data: Buffer): string {
|
||||
const SAMPLE_SIZE = 1024 * 1024; // 1MB
|
||||
export function computeMidHash256FromBuffer(data: Buffer): string {
|
||||
const SAMPLE_SIZE = 1024 * 1024;
|
||||
const MIDHASH_VARINT = Buffer.from([0x80, 0x20]);
|
||||
|
||||
const fileSize = data.length;
|
||||
|
||||
const sizeBuffer = Buffer.allocUnsafe(8);
|
||||
sizeBuffer.writeBigUInt64BE(BigInt(fileSize), 0);
|
||||
|
||||
@@ -131,16 +161,8 @@ function computeMidHash256FromBuffer(data: Buffer): string {
|
||||
sampleData = data.subarray(middleOffset, middleOffset + SAMPLE_SIZE);
|
||||
}
|
||||
|
||||
const hashInput = Buffer.concat([sizeBuffer, sampleData]);
|
||||
const hashBuffer = createHash('sha256').update(hashInput).digest();
|
||||
|
||||
const cidBytes = Buffer.concat([
|
||||
Buffer.from([0x01]),
|
||||
MIDHASH_VARINT,
|
||||
MIDHASH_VARINT,
|
||||
Buffer.from([0x20]),
|
||||
hashBuffer
|
||||
]);
|
||||
const hashBuffer = createHash('sha256').update(Buffer.concat([sizeBuffer, sampleData])).digest();
|
||||
const cidBytes = Buffer.concat([Buffer.from([0x01]), MIDHASH_VARINT, MIDHASH_VARINT, Buffer.from([0x20]), hashBuffer]);
|
||||
|
||||
const base32Chars = 'abcdefghijklmnopqrstuvwxyz234567';
|
||||
let cid = 'b';
|
||||
@@ -157,337 +179,480 @@ function computeMidHash256FromBuffer(data: Buffer): string {
|
||||
if (bits > 0) {
|
||||
cid += base32Chars[(value << (5 - bits)) & 0x1f];
|
||||
}
|
||||
|
||||
return cid;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sanitize filename by removing invalid characters
|
||||
*/
|
||||
function sanitizeFilename(name: string): string {
|
||||
/** Strip characters that are hostile to paths or to the WebDAV URL. */
|
||||
export function sanitizeFilename(name: string): string {
|
||||
return name
|
||||
.replace(/[<>:"/\\|?*]/g, '')
|
||||
.replace(/[<>:"/\\|?*#%\x00-\x1f]/g, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse subtitle streams from ffmpeg plugin metadata
|
||||
*/
|
||||
interface SubtitleStream {
|
||||
index: number;
|
||||
codec: string;
|
||||
language?: string;
|
||||
title?: string;
|
||||
// ----------------------------------------------------------------------------
|
||||
// Stream table
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
function numericKeyOrder(a: string, b: string): number {
|
||||
return Number(a) - Number(b);
|
||||
}
|
||||
|
||||
function parseSubtitleStreams(existingMeta: Record<string, string>): SubtitleStream[] {
|
||||
const streams: SubtitleStream[] = [];
|
||||
/**
|
||||
* The ffmpeg plugin's per-stream objects, in stream order, from any of the shapes
|
||||
* a record can arrive in:
|
||||
* - nested array of JSON strings stream: ["{...}", "{...}"] (meta-sort /meta today)
|
||||
* - nested array of objects stream: [{...}, {...}]
|
||||
* - nested object keyed by index stream: {"0": "{...}", "1": ...}
|
||||
* - JSON string of an array stream: "[...]"
|
||||
* - flat meta-core keys "stream/0": "{...}", "stream/10": ... (ordered numerically)
|
||||
*/
|
||||
export function readStreamEntries(existingMeta: Record<string, unknown> | undefined): Record<string, unknown>[] {
|
||||
if (!existingMeta) return [];
|
||||
const raw: unknown[] = [];
|
||||
|
||||
// ffmpeg plugin stores streams as 'stream' field - an array of JSON strings
|
||||
// Each element is a stringified JSON object like:
|
||||
// '{"type":"subtitle","codec":"ass","index":2,"language":"eng",...}'
|
||||
const streamData = existingMeta['stream'];
|
||||
if (streamData) {
|
||||
const nested = existingMeta['stream'];
|
||||
if (nested !== undefined && nested !== null && nested !== '') {
|
||||
try {
|
||||
// Parse the outer array (it's already an array from meta-core)
|
||||
const allStreams = Array.isArray(streamData)
|
||||
? streamData
|
||||
: JSON.parse(streamData) as string[];
|
||||
|
||||
let subtitleIndex = 0;
|
||||
for (const streamStr of allStreams) {
|
||||
// Each stream is a JSON string
|
||||
const stream = typeof streamStr === 'string'
|
||||
? JSON.parse(streamStr) as {
|
||||
type?: string;
|
||||
codec?: string;
|
||||
index?: number;
|
||||
language?: string;
|
||||
title?: string;
|
||||
}
|
||||
: streamStr;
|
||||
|
||||
if (stream.type === 'subtitle') {
|
||||
streams.push({
|
||||
index: subtitleIndex,
|
||||
codec: stream.codec || 'unknown',
|
||||
language: stream.language,
|
||||
title: stream.title,
|
||||
});
|
||||
subtitleIndex++;
|
||||
}
|
||||
const parsed = typeof nested === 'string' ? JSON.parse(nested) : nested;
|
||||
if (Array.isArray(parsed)) {
|
||||
raw.push(...parsed);
|
||||
} else if (parsed && typeof parsed === 'object') {
|
||||
const obj = parsed as Record<string, unknown>;
|
||||
raw.push(...Object.keys(obj).sort(numericKeyOrder).map((k) => obj[k]));
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('[subtitle-extractor] Failed to parse stream metadata:', e);
|
||||
} catch {
|
||||
// fall through to the flat form
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: parse individual fields (legacy format)
|
||||
if (streams.length === 0) {
|
||||
if (raw.length === 0) {
|
||||
const flat = Object.keys(existingMeta)
|
||||
.filter((k) => /^stream\/\d+$/.test(k))
|
||||
.sort((a, b) => numericKeyOrder(a.slice(7), b.slice(7)));
|
||||
raw.push(...flat.map((k) => existingMeta[k]));
|
||||
}
|
||||
|
||||
const entries: Record<string, unknown>[] = [];
|
||||
for (const item of raw) {
|
||||
try {
|
||||
const obj = typeof item === 'string' ? JSON.parse(item) : item;
|
||||
if (obj && typeof obj === 'object' && !Array.isArray(obj)) entries.push(obj as Record<string, unknown>);
|
||||
} catch {
|
||||
// skip an unparseable entry, keep the rest
|
||||
}
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
export interface SubtitleStream {
|
||||
/** Position among subtitle streams (ffmpeg `0:s:<ordinal>`). */
|
||||
ordinal: number;
|
||||
/** Global ffmpeg stream index (`0:<streamIndex>`), when the record has it. */
|
||||
streamIndex?: number;
|
||||
codec: string;
|
||||
/** ISO 639-2/B, `und` when undeclared. */
|
||||
lang3: string;
|
||||
title?: string;
|
||||
forced: boolean;
|
||||
}
|
||||
|
||||
const isTrue = (v: unknown) => v === true || v === 'true';
|
||||
|
||||
/** Subtitle streams of a record, from the stream table or the legacy `subtitle_{i}_*` fields. */
|
||||
export function parseSubtitleStreams(existingMeta: Record<string, unknown> | undefined): SubtitleStream[] {
|
||||
const streams: SubtitleStream[] = [];
|
||||
let ordinal = 0;
|
||||
for (const entry of readStreamEntries(existingMeta)) {
|
||||
if (entry.type !== 'subtitle') continue;
|
||||
const index = entry.index === undefined || entry.index === null ? NaN : Number(entry.index);
|
||||
streams.push({
|
||||
ordinal: ordinal++,
|
||||
streamIndex: Number.isInteger(index) && index >= 0 ? index : undefined,
|
||||
codec: String(entry.codec ?? 'unknown').toLowerCase(),
|
||||
lang3: toLang3(entry.language),
|
||||
title: typeof entry.title === 'string' ? entry.title : undefined,
|
||||
forced: isTrue(entry.forced),
|
||||
});
|
||||
}
|
||||
|
||||
if (streams.length === 0 && existingMeta) {
|
||||
for (let i = 0; i < 20; i++) {
|
||||
const codec = existingMeta[`subtitle_${i}_codec`];
|
||||
if (codec) {
|
||||
streams.push({
|
||||
index: i,
|
||||
codec,
|
||||
language: existingMeta[`subtitle_${i}_language`],
|
||||
title: existingMeta[`subtitle_${i}_title`],
|
||||
});
|
||||
}
|
||||
if (!codec) continue;
|
||||
const title = existingMeta[`subtitle_${i}_title`];
|
||||
streams.push({
|
||||
ordinal: i,
|
||||
codec: String(codec).toLowerCase(),
|
||||
lang3: toLang3(existingMeta[`subtitle_${i}_language`]),
|
||||
title: typeof title === 'string' ? title : undefined,
|
||||
forced: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return streams;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract a subtitle track using ffmpeg
|
||||
*/
|
||||
async function extractSubtitle(
|
||||
inputPath: string,
|
||||
outputPath: string,
|
||||
subtitleIndex: number,
|
||||
codec: string
|
||||
): Promise<boolean> {
|
||||
return new Promise((resolve) => {
|
||||
// Determine output codec based on format
|
||||
let outputCodec = 'srt';
|
||||
if (outputFormat === 'vtt') {
|
||||
outputCodec = 'webvtt';
|
||||
} else if (outputFormat === 'ass') {
|
||||
outputCodec = 'ass';
|
||||
/** File type and ffmpeg encoder for a codec under an output format; null when not a text codec. */
|
||||
export function outputFor(codec: string, format: OutputFormat = outputFormat): { ext: SubtitleExt; encoder: string } | null {
|
||||
const nativeExt = TEXT_SUBTITLE_CODECS[codec];
|
||||
if (!nativeExt) return null;
|
||||
const ext: SubtitleExt = format === 'native' ? nativeExt : format;
|
||||
const encoder = ext === nativeExt && COPYABLE_CODECS.has(codec) ? 'copy' : ENCODER_FOR_EXT[ext];
|
||||
return { ext, encoder };
|
||||
}
|
||||
|
||||
export type TextSubtitleStream = SubtitleStream & { ext: SubtitleExt; encoder: string };
|
||||
|
||||
export function classifySubtitleStreams(streams: SubtitleStream[], format: OutputFormat = outputFormat): {
|
||||
text: TextSubtitleStream[];
|
||||
image: SubtitleStream[];
|
||||
unsupported: SubtitleStream[];
|
||||
} {
|
||||
const text: TextSubtitleStream[] = [];
|
||||
const image: SubtitleStream[] = [];
|
||||
const unsupported: SubtitleStream[] = [];
|
||||
for (const s of streams) {
|
||||
if (IMAGE_SUBTITLE_CODECS.has(s.codec)) {
|
||||
image.push(s);
|
||||
continue;
|
||||
}
|
||||
const out = outputFor(s.codec, format);
|
||||
if (out) text.push({ ...s, ...out });
|
||||
else unsupported.push(s);
|
||||
}
|
||||
return { text, image, unsupported };
|
||||
}
|
||||
|
||||
const args = [
|
||||
'-y',
|
||||
'-hide_banner',
|
||||
'-loglevel', 'error',
|
||||
'-probesize', '1M',
|
||||
'-analyzeduration', '1M',
|
||||
'-i', inputPath,
|
||||
'-map', `0:s:${subtitleIndex}`,
|
||||
'-c:s', outputCodec,
|
||||
outputPath
|
||||
];
|
||||
const scalar = (v: unknown): string => (typeof v === 'string' || typeof v === 'number' ? String(v) : '');
|
||||
|
||||
console.log(`[subtitle-extractor] Running ffmpeg for subtitle ${subtitleIndex}`);
|
||||
/**
|
||||
* `Title (Year)[videoCid]_subtitle.s<streamIndex>.<lang3>[.forced].<ext>`.
|
||||
* The stream index keeps two tracks in one language (full + signs, es-ES +
|
||||
* es-419) from overwriting each other; the trailing `<lang3>` is what the
|
||||
* `subtitle` plugin reads back as the file's language.
|
||||
*/
|
||||
export function subtitleFilename(existingMeta: Record<string, unknown>, videoCid: string, stream: SubtitleStream, ext: string): string {
|
||||
const fileName = scalar(existingMeta.fileName);
|
||||
const rawTitle = scalar(existingMeta.originalTitle) || scalar(existingMeta.title) || (fileName ? path.parse(fileName).name : '');
|
||||
const safeTitle = sanitizeFilename(rawTitle).slice(0, 100) || 'video';
|
||||
const year = scalar(existingMeta.movieYear);
|
||||
const track = stream.streamIndex !== undefined ? `s${stream.streamIndex}` : `t${stream.ordinal}`;
|
||||
return `${safeTitle}${year ? ` (${year})` : ''}[${videoCid}]_subtitle.${track}.${stream.lang3}${stream.forced ? '.forced' : ''}.${ext}`;
|
||||
}
|
||||
|
||||
const ffmpeg = spawn('ffmpeg', args, { stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
// ----------------------------------------------------------------------------
|
||||
// Extraction
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
export interface ExtractJob {
|
||||
stream: TextSubtitleStream;
|
||||
tempFile: string;
|
||||
}
|
||||
|
||||
/** ONE ffmpeg invocation writing every job's track to its own output (one read of the input). */
|
||||
export function buildExtractionArgs(input: string, jobs: ExtractJob[]): string[] {
|
||||
const args = ['-y', '-hide_banner', '-nostdin', '-loglevel', 'error', '-i', input];
|
||||
for (const job of jobs) {
|
||||
const map = job.stream.streamIndex !== undefined ? `0:${job.stream.streamIndex}` : `0:s:${job.stream.ordinal}`;
|
||||
args.push('-map', map, '-c:s', job.stream.encoder, job.tempFile);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
/** Number of ffmpeg processes started — each one is a full read of the input. */
|
||||
export const extractionStats = { ffmpegPasses: 0 };
|
||||
|
||||
function runFfmpeg(args: string[], timeoutMs: number): Promise<{ code: number | null; stderr: string; timedOut: boolean }> {
|
||||
extractionStats.ffmpegPasses++;
|
||||
return new Promise((resolve) => {
|
||||
const ffmpeg = spawn('ffmpeg', args, { stdio: ['ignore', 'ignore', 'pipe'] });
|
||||
let stderr = '';
|
||||
ffmpeg.stderr?.on('data', (data) => {
|
||||
stderr += data.toString();
|
||||
});
|
||||
|
||||
ffmpeg.on('close', (code) => {
|
||||
if (code === 0 && existsSync(outputPath)) {
|
||||
const size = statSync(outputPath).size;
|
||||
if (size > 10) {
|
||||
console.log(`[subtitle-extractor] Extracted subtitle ${subtitleIndex} (${size} bytes)`);
|
||||
resolve(true);
|
||||
} else {
|
||||
console.log(`[subtitle-extractor] Subtitle ${subtitleIndex} extraction produced empty file`);
|
||||
try { unlinkSync(outputPath); } catch {}
|
||||
resolve(false);
|
||||
}
|
||||
} else {
|
||||
console.log(`[subtitle-extractor] Failed to extract subtitle ${subtitleIndex}: ${stderr.slice(0, 200)}`);
|
||||
resolve(false);
|
||||
}
|
||||
});
|
||||
|
||||
ffmpeg.on('error', (err) => {
|
||||
console.log(`[subtitle-extractor] ffmpeg error: ${err.message}`);
|
||||
resolve(false);
|
||||
});
|
||||
|
||||
// Timeout after 2 minutes per subtitle
|
||||
setTimeout(() => {
|
||||
let timedOut = false;
|
||||
let settled = false;
|
||||
const finish = (code: number | null) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve({ code, stderr, timedOut });
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
timedOut = true;
|
||||
ffmpeg.kill('SIGKILL');
|
||||
resolve(false);
|
||||
}, 120000);
|
||||
}, timeoutMs);
|
||||
ffmpeg.stderr?.on('data', (d) => { if (stderr.length < 4000) stderr += d.toString(); });
|
||||
ffmpeg.on('close', (code) => finish(code));
|
||||
ffmpeg.on('error', (err) => { stderr += err.message; finish(-1); });
|
||||
});
|
||||
}
|
||||
|
||||
/** Does an extracted file carry at least one cue (an ASS header alone does not count)? */
|
||||
export function hasCues(data: Buffer, ext: string): boolean {
|
||||
const text = data.toString('utf8');
|
||||
if (ext === 'ass' || ext === 'ssa') return /^Dialogue:/m.test(text);
|
||||
return text.includes('-->');
|
||||
}
|
||||
|
||||
function takeOutput(job: ExtractJob): Buffer | null {
|
||||
if (!existsSync(job.tempFile)) return null;
|
||||
const data = readFileSync(job.tempFile);
|
||||
try { unlinkSync(job.tempFile); } catch { /* ignore */ }
|
||||
return hasCues(data, job.stream.ext) ? data : null;
|
||||
}
|
||||
|
||||
function discard(job: ExtractJob): void {
|
||||
try { if (existsSync(job.tempFile)) unlinkSync(job.tempFile); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
export interface ExtractResult {
|
||||
outputs: Array<{ job: ExtractJob; data: Buffer | null }>;
|
||||
passes: number;
|
||||
timedOut: boolean;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export async function extractTracks(input: string, jobs: ExtractJob[], timeoutMs: number = extractionTimeoutMs): Promise<ExtractResult> {
|
||||
const first = await runFfmpeg(buildExtractionArgs(input, jobs), timeoutMs);
|
||||
if (first.code === 0) {
|
||||
return { outputs: jobs.map((job) => ({ job, data: takeOutput(job) })), passes: 1, timedOut: false };
|
||||
}
|
||||
jobs.forEach(discard);
|
||||
const error = first.timedOut ? `ffmpeg timed out after ${timeoutMs} ms` : first.stderr.slice(0, 300) || `ffmpeg exited ${first.code}`;
|
||||
if (first.timedOut || jobs.length === 1) {
|
||||
return { outputs: jobs.map((job) => ({ job, data: null })), passes: 1, timedOut: first.timedOut, error };
|
||||
}
|
||||
|
||||
// One broken track aborts every output of the shared pass. Salvage the rest
|
||||
// one track at a time — at the price of one more full read per track.
|
||||
console.warn(`[subtitle-extractor] single-pass extraction failed (${error}); retrying ${jobs.length} track(s) one by one`);
|
||||
const outputs: ExtractResult['outputs'] = [];
|
||||
let passes = 1;
|
||||
let timedOut = false;
|
||||
for (const job of jobs) {
|
||||
if (timedOut) {
|
||||
outputs.push({ job, data: null });
|
||||
continue;
|
||||
}
|
||||
const r = await runFfmpeg(buildExtractionArgs(input, [job]), timeoutMs);
|
||||
passes++;
|
||||
timedOut = r.timedOut;
|
||||
if (r.code === 0) {
|
||||
outputs.push({ job, data: takeOutput(job) });
|
||||
} else {
|
||||
discard(job);
|
||||
outputs.push({ job, data: null });
|
||||
}
|
||||
}
|
||||
return { outputs, passes, timedOut, error };
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Record keys
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
export interface ProducedSubtitle {
|
||||
subCid: string;
|
||||
lang3: string;
|
||||
}
|
||||
|
||||
/** True when this record already carries the `extractedSubtitles` key-set (nested or flat). */
|
||||
export function hasExtractedKeySet(existingMeta: Record<string, unknown> | undefined): boolean {
|
||||
if (!existingMeta) return false;
|
||||
const v = existingMeta['extractedSubtitles'];
|
||||
if (v && typeof v === 'object') return true;
|
||||
return Object.keys(existingMeta).some((k) => k.startsWith('extractedSubtitles/'));
|
||||
}
|
||||
|
||||
/** Keys on the video record for a set of extracted files (METADATA_KEYS.md §8, §9). */
|
||||
export function videoSubtitleKeys(produced: ProducedSubtitle[]): Record<string, string> {
|
||||
const keys: Record<string, string> = {};
|
||||
for (const { subCid, lang3 } of produced) {
|
||||
keys[`subtitles/${lang3}/${subCid}`] = 'true';
|
||||
keys[`extractedSubtitles/${lang3}/${subCid}`] = 'true';
|
||||
if (lang3 !== 'und') {
|
||||
keys[`subtitleLanguages/${lang3}`] = 'true';
|
||||
keys[`languages/${lang3}`] = 'true';
|
||||
}
|
||||
}
|
||||
return keys;
|
||||
}
|
||||
|
||||
/** Keys on the subtitle file's own record. */
|
||||
export function subtitleRecordKeys(videoCid: string, lang3: string): Record<string, string> {
|
||||
const keys: Record<string, string> = { [`videos/${videoCid}`]: 'true' };
|
||||
if (lang3 !== 'und') keys.subtitleLanguage = lang3;
|
||||
return keys;
|
||||
}
|
||||
|
||||
/**
|
||||
* Legacy scalar csv-sets the old version of this plugin wrote (and the old
|
||||
* `subtitle` plugin's `subtitles`), turned into deletes plus the key-set leaves
|
||||
* that preserve what they said. `extractedSubtitles` members are dropped: this
|
||||
* run re-derives them with their language.
|
||||
*/
|
||||
export function legacySubtitleMigration(existingMeta: Record<string, unknown> | undefined): { deletes: string[]; sets: Record<string, string> } {
|
||||
const deletes: string[] = [];
|
||||
const sets: Record<string, string> = {};
|
||||
|
||||
if (legacyCsvMembers(existingMeta, 'extractedSubtitles')) deletes.push('extractedSubtitles');
|
||||
|
||||
const langs = legacyCsvMembers(existingMeta, 'subtitleLanguages');
|
||||
if (langs) {
|
||||
deletes.push('subtitleLanguages');
|
||||
for (const l of langs.map(toLang3)) if (l !== 'und') sets[`subtitleLanguages/${l}`] = 'true';
|
||||
}
|
||||
|
||||
const subs = legacyCsvMembers(existingMeta, 'subtitles');
|
||||
if (subs) {
|
||||
deletes.push('subtitles');
|
||||
for (const cid of subs) sets[`subtitles/und/${cid}`] = 'true';
|
||||
}
|
||||
return { deletes, sets };
|
||||
}
|
||||
|
||||
/** Write the links for `produced` onto the video and each subtitle record. False on any failed write. */
|
||||
export async function linkSubtitles(
|
||||
metaCore: MetaCoreClient,
|
||||
videoCid: string,
|
||||
existingMeta: Record<string, unknown>,
|
||||
produced: ProducedSubtitle[],
|
||||
): Promise<boolean> {
|
||||
const { deletes, sets } = legacySubtitleMigration(existingMeta);
|
||||
const flat = flattenMeta(existingMeta);
|
||||
for (const key of deletes) delete flat[key];
|
||||
|
||||
for (const { subCid, lang3 } of produced) {
|
||||
deletes.push(
|
||||
...staleLanguageLeaves(flat, 'subtitles', subCid, lang3),
|
||||
...staleLanguageLeaves(flat, 'extractedSubtitles', subCid, lang3),
|
||||
);
|
||||
}
|
||||
for (const key of deletes) {
|
||||
if (!(await metaCore.deleteProperty(videoCid, key))) return false;
|
||||
delete flat[key];
|
||||
}
|
||||
|
||||
const fresh = onlyNewKeys(flat, { ...sets, ...videoSubtitleKeys(produced) });
|
||||
if (Object.keys(fresh).length > 0 && !(await metaCore.mergeMetadata(videoCid, fresh))) return false;
|
||||
|
||||
for (const { subCid, lang3 } of produced) {
|
||||
const current = await metaCore.getMetadata(subCid);
|
||||
const wanted = onlyNewKeys(flattenMeta(current), subtitleRecordKeys(videoCid, lang3));
|
||||
if (Object.keys(wanted).length > 0 && !(await metaCore.mergeMetadata(subCid, wanted))) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Task
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
export async function process(
|
||||
request: ProcessRequest,
|
||||
sendCallback: (payload: CallbackPayload) => Promise<void>
|
||||
): Promise<void> {
|
||||
const startTime = Date.now();
|
||||
const { taskId, cid, filePath } = request;
|
||||
const existingMeta = (request.existingMeta ?? {}) as Record<string, unknown>;
|
||||
const metaCore = new MetaCoreClient(request.metaCoreUrl);
|
||||
|
||||
const finish = (status: CallbackPayload['status'], extra: { reason?: string; error?: string } = {}) =>
|
||||
sendCallback({ taskId, status, duration: Date.now() - startTime, ...extra });
|
||||
const skip = (reason: string, quiet = false) => {
|
||||
if (!quiet) console.log(`[subtitle-extractor] ${filePath}: skipped — ${reason}`);
|
||||
return finish('skipped', { reason });
|
||||
};
|
||||
const fail = (error: string) => {
|
||||
console.error(`[subtitle-extractor] ${filePath}: failed — ${error}`);
|
||||
return finish('failed', { error });
|
||||
};
|
||||
|
||||
try {
|
||||
const { cid, filePath, existingMeta } = request;
|
||||
|
||||
// Only process video files
|
||||
if (existingMeta?.fileType !== 'video') {
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'skipped',
|
||||
duration: Date.now() - startTime,
|
||||
reason: 'Not a video file',
|
||||
});
|
||||
if (existingMeta.fileType !== 'video') {
|
||||
await skip('Not a video file', true);
|
||||
return;
|
||||
}
|
||||
if (!forceRecompute && hasExtractedKeySet(existingMeta)) {
|
||||
await skip('Subtitles already extracted', true);
|
||||
return;
|
||||
}
|
||||
|
||||
// Skip if already extracted (unless forceRecompute)
|
||||
if (existingMeta?.extractedSubtitles && !forceRecompute) {
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'skipped',
|
||||
duration: Date.now() - startTime,
|
||||
reason: 'Subtitles already extracted',
|
||||
});
|
||||
const streams = parseSubtitleStreams(existingMeta);
|
||||
if (streams.length === 0) {
|
||||
await skip('No subtitle streams found', true);
|
||||
return;
|
||||
}
|
||||
|
||||
// Parse subtitle streams from ffmpeg metadata
|
||||
const subtitleStreams = parseSubtitleStreams(existingMeta || {});
|
||||
|
||||
if (subtitleStreams.length === 0) {
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'skipped',
|
||||
duration: Date.now() - startTime,
|
||||
reason: 'No subtitle streams found',
|
||||
});
|
||||
const { text, image, unsupported } = classifySubtitleStreams(streams);
|
||||
if (image.length > 0) {
|
||||
console.log(`[subtitle-extractor] ${filePath}: skipping ${image.length} image-based subtitle stream(s) (${[...new Set(image.map((s) => s.codec))].join(', ')})`);
|
||||
}
|
||||
if (unsupported.length > 0) {
|
||||
console.log(`[subtitle-extractor] ${filePath}: skipping ${unsupported.length} unsupported subtitle stream(s) (${[...new Set(unsupported.map((s) => s.codec))].join(', ')})`);
|
||||
}
|
||||
if (text.length === 0) {
|
||||
await skip(image.length > 0 ? 'Only image-based subtitles (cannot convert to text)' : 'No supported text subtitle streams');
|
||||
return;
|
||||
}
|
||||
|
||||
// Filter out unsupported (image-based) codecs
|
||||
const textSubtitles = subtitleStreams.filter(s => !UNSUPPORTED_SUBTITLE_CODECS.has(s.codec));
|
||||
|
||||
if (textSubtitles.length === 0) {
|
||||
console.log(`[subtitle-extractor] All ${subtitleStreams.length} subtitles are image-based, skipping`);
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'skipped',
|
||||
duration: Date.now() - startTime,
|
||||
reason: 'Only image-based subtitles (cannot convert to text)',
|
||||
});
|
||||
const webdav = await getWebDAVClient(request.metaCoreUrl);
|
||||
if (!webdav) {
|
||||
await fail('No WebDAV endpoint available');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`[subtitle-extractor] Found ${textSubtitles.length} text-based subtitle(s) in ${filePath}`);
|
||||
const produced: ProducedSubtitle[] = [];
|
||||
const pending: Array<ExtractJob & { webdavPath: string }> = [];
|
||||
const tmp = tempDir();
|
||||
mkdirSync(tmp, { recursive: true });
|
||||
const safeTask = taskId.replace(/[^A-Za-z0-9_-]/g, '_');
|
||||
|
||||
// Require WebDAV client for writing output
|
||||
if (!webdavClient) {
|
||||
console.error('[subtitle-extractor] WebDAV client not available, cannot write output files');
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'failed',
|
||||
duration: Date.now() - startTime,
|
||||
error: 'WebDAV client not configured (WEBDAV_URL not set)',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Get input path (WebDAV URL for reading)
|
||||
let inputPath = webdavClient.toWebDAVUrl(filePath);
|
||||
console.log(`[subtitle-extractor] Using WebDAV input: ${inputPath}`);
|
||||
|
||||
// Ensure temp directory exists for ffmpeg extraction
|
||||
if (!existsSync(TEMP_PATH)) {
|
||||
mkdirSync(TEMP_PATH, { recursive: true });
|
||||
}
|
||||
|
||||
// Get video title for output filename
|
||||
const title = existingMeta?.originalTitle || existingMeta?.title || existingMeta?.fileName || 'video';
|
||||
const safeTitle = sanitizeFilename(title);
|
||||
const year = existingMeta?.movieYear;
|
||||
const yearStr = year ? ` (${year})` : '';
|
||||
|
||||
const extractedCids: string[] = [];
|
||||
const extractedLanguages: string[] = [];
|
||||
|
||||
// Extract each subtitle
|
||||
for (const sub of textSubtitles) {
|
||||
const langSuffix = sub.language ? `.${sub.language}` : `.${sub.index}`;
|
||||
const ext = outputFormat;
|
||||
|
||||
// Build output filename: Title (Year)[videoCID]_subtitle.lang.srt
|
||||
const outputFilename = `${safeTitle}${yearStr}[${cid}]_subtitle${langSuffix}.${ext}`;
|
||||
const webdavOutputPath = `${PLUGIN_OUTPUT_WEBDAV_PATH}/${outputFilename}`;
|
||||
const tempOutputPath = path.join(TEMP_PATH, outputFilename);
|
||||
|
||||
// Check if already extracted via WebDAV
|
||||
if (!forceRecompute) {
|
||||
const exists = await webdavClient.exists(webdavOutputPath);
|
||||
if (exists) {
|
||||
console.log(`[subtitle-extractor] Subtitle already exists: ${outputFilename}`);
|
||||
// Read existing file to get CID
|
||||
try {
|
||||
const existingData = await webdavClient.readFile(webdavOutputPath);
|
||||
const subtitleCid = computeMidHash256FromBuffer(existingData);
|
||||
extractedCids.push(subtitleCid);
|
||||
if (sub.language && !extractedLanguages.includes(sub.language)) {
|
||||
extractedLanguages.push(sub.language);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error(`[subtitle-extractor] Failed to read existing subtitle: ${e}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Extract subtitle to temp file
|
||||
const extracted = await extractSubtitle(inputPath, tempOutputPath, sub.index, sub.codec);
|
||||
|
||||
if (extracted) {
|
||||
for (const stream of text) {
|
||||
const webdavPath = `${PLUGIN_OUTPUT_WEBDAV_PATH}/${subtitleFilename(existingMeta, cid, stream, stream.ext)}`;
|
||||
if (!forceRecompute && await webdav.exists(webdavPath)) {
|
||||
try {
|
||||
// Read extracted subtitle file
|
||||
const subtitleData = readFileSync(tempOutputPath);
|
||||
|
||||
// Compute CID from buffer
|
||||
const subtitleCid = computeMidHash256FromBuffer(subtitleData);
|
||||
console.log(`[subtitle-extractor] Subtitle CID: ${subtitleCid}`);
|
||||
|
||||
// Upload to WebDAV
|
||||
await webdavClient.writeFile(webdavOutputPath, subtitleData);
|
||||
console.log(`[subtitle-extractor] Uploaded subtitle to WebDAV: ${webdavOutputPath}`);
|
||||
|
||||
extractedCids.push(subtitleCid);
|
||||
if (sub.language && !extractedLanguages.includes(sub.language)) {
|
||||
extractedLanguages.push(sub.language);
|
||||
}
|
||||
|
||||
// Store subtitle CID as metadata on the video
|
||||
await metaCore.addToSet(cid, 'extractedSubtitles', subtitleCid);
|
||||
|
||||
// Store language if available
|
||||
if (sub.language) {
|
||||
await metaCore.addToSet(cid, 'subtitleLanguages', sub.language);
|
||||
}
|
||||
|
||||
// Clean up temp file
|
||||
try { unlinkSync(tempOutputPath); } catch {}
|
||||
// A previous run already wrote this track — reuse it, no read of the video.
|
||||
produced.push({ subCid: computeMidHash256FromBuffer(await webdav.readFile(webdavPath)), lang3: stream.lang3 });
|
||||
continue;
|
||||
} catch (e) {
|
||||
console.error(`[subtitle-extractor] Failed to process extracted subtitle: ${e}`);
|
||||
// Clean up temp file on error
|
||||
try { unlinkSync(tempOutputPath); } catch {}
|
||||
console.warn(`[subtitle-extractor] could not reuse ${webdavPath}: ${e instanceof Error ? e.message : String(e)}`);
|
||||
}
|
||||
}
|
||||
pending.push({ stream, webdavPath, tempFile: path.join(tmp, `${safeTask}-${stream.ordinal}.${stream.ext}`) });
|
||||
}
|
||||
|
||||
let extraction: ExtractResult | undefined;
|
||||
if (pending.length > 0) {
|
||||
console.log(`[subtitle-extractor] ${filePath}: extracting ${pending.length} text track(s) in one pass`);
|
||||
extraction = await extractTracks(webdav.toWebDAVUrl(filePath), pending);
|
||||
for (const { job, data } of extraction.outputs) {
|
||||
if (!data) continue;
|
||||
const target = (job as ExtractJob & { webdavPath: string }).webdavPath;
|
||||
await webdav.writeFile(target, data);
|
||||
produced.push({ subCid: computeMidHash256FromBuffer(data), lang3: job.stream.lang3 });
|
||||
}
|
||||
}
|
||||
|
||||
if (extractedCids.length > 0) {
|
||||
console.log(`[subtitle-extractor] Extracted ${extractedCids.length} subtitle(s) from ${filePath}`);
|
||||
} else {
|
||||
console.log(`[subtitle-extractor] No subtitles could be extracted from ${filePath}`);
|
||||
if (produced.length === 0) {
|
||||
if (extraction?.error) {
|
||||
await fail(extraction.error);
|
||||
} else {
|
||||
await skip('No text subtitle track produced any cues');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'completed',
|
||||
duration: Date.now() - startTime,
|
||||
});
|
||||
if (!(await linkSubtitles(metaCore, cid, existingMeta, produced))) {
|
||||
await fail('meta-core write failed');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(
|
||||
`[subtitle-extractor] ${filePath}: linked ${produced.length} subtitle(s) ` +
|
||||
`[${produced.map((p) => p.lang3).join(', ')}] in ${extraction?.passes ?? 0} ffmpeg pass(es)`
|
||||
);
|
||||
await finish('completed');
|
||||
} catch (error) {
|
||||
console.error(`[subtitle-extractor] Error:`, error);
|
||||
await sendCallback({
|
||||
taskId: request.taskId,
|
||||
status: 'failed',
|
||||
duration: Date.now() - startTime,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
await fail(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
}
|
||||
|
||||
+3
-1
@@ -44,7 +44,9 @@ export interface ProcessRequest {
|
||||
filePath: string;
|
||||
callbackUrl: string;
|
||||
metaCoreUrl: string;
|
||||
existingMeta?: Record<string, string>;
|
||||
// meta-sort sends its NESTED /meta document (`fileinfo: {...}`, `stream: [...]`);
|
||||
// meta-core serves the flat `a/b` form. Top-level scalars match in both.
|
||||
existingMeta?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface ProcessResponse {
|
||||
|
||||
+60
-35
@@ -64,7 +64,8 @@ export class WebDAVClient {
|
||||
}
|
||||
|
||||
/**
|
||||
* Read bytes from a file via HTTP Range request
|
||||
* Read first N bytes of a file (for magic byte detection)
|
||||
* Uses HTTP Range request
|
||||
*/
|
||||
async readBytes(filePath: string, start: number, end: number): Promise<Buffer> {
|
||||
const url = this.toWebDAVUrl(filePath);
|
||||
@@ -85,7 +86,7 @@ export class WebDAVClient {
|
||||
}
|
||||
|
||||
/**
|
||||
* Read entire file as buffer (for small files)
|
||||
* Read entire file as buffer (for small files only)
|
||||
*/
|
||||
async readFile(filePath: string): Promise<Buffer> {
|
||||
const url = this.toWebDAVUrl(filePath);
|
||||
@@ -106,12 +107,9 @@ export class WebDAVClient {
|
||||
async writeFile(filePath: string, data: Buffer | string): Promise<void> {
|
||||
const url = this.toWebDAVUrl(filePath);
|
||||
|
||||
// Convert Buffer to Uint8Array for fetch compatibility
|
||||
const body = typeof data === 'string' ? data : new Uint8Array(data);
|
||||
|
||||
const response = await fetch(url, {
|
||||
method: 'PUT',
|
||||
body,
|
||||
body: data,
|
||||
headers: {
|
||||
'Content-Type': 'application/octet-stream',
|
||||
},
|
||||
@@ -135,40 +133,67 @@ export class WebDAVClient {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* List directory contents via HTTP GET (returns JSON from nginx autoindex)
|
||||
*/
|
||||
async readdir(dirPath: string): Promise<string[]> {
|
||||
const url = this.toWebDAVUrl(dirPath);
|
||||
const finalUrl = url.endsWith('/') ? url : url + '/';
|
||||
|
||||
const response = await fetch(finalUrl, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'Accept': 'application/json',
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`WebDAV GET (dir) failed for ${dirPath}: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
const entries = await response.json() as Array<{ name: string; type: string }>;
|
||||
return entries.map(e => e.name);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a WebDAV client from environment variables
|
||||
* WebDAV client for the meta-core that drove a /process call.
|
||||
*
|
||||
* The endpoint is read from that core's own /urls (`webdavUrlInternal`),
|
||||
* so blobs always land in the /files of the core we write metadata to. A feeder
|
||||
* whose plugins run beside it but enrich a remote hub core needs no per-stack
|
||||
* wiring, and it is impossible to write posters into one core while pointing
|
||||
* CIDs at another.
|
||||
*
|
||||
* WEBDAV_URL still wins when set — an explicit override for tests and for
|
||||
* topologies where the core's advertised internal URL isn't reachable from here.
|
||||
*
|
||||
* Resolutions are cached per meta-core URL; failures are not, so a core that is
|
||||
* still starting up is retried on the next task.
|
||||
*/
|
||||
export function createWebDAVClient(): WebDAVClient | null {
|
||||
const webdavUrl = process.env.WEBDAV_URL;
|
||||
const clientCache = new Map<string, Promise<WebDAVClient | null>>();
|
||||
|
||||
if (!webdavUrl) {
|
||||
console.warn('[webdav-client] WEBDAV_URL not set, WebDAV client unavailable');
|
||||
return null;
|
||||
async function resolveWebDAVUrl(metaCoreUrl: string): Promise<string | null> {
|
||||
// /urls, not /api/urls — metaCoreUrl points at the meta-core backend, whose
|
||||
// discovery routes are unprefixed (same base the /meta/* writes use).
|
||||
const base = metaCoreUrl.replace(/\/$/, '');
|
||||
const response = await fetch(`${base}/urls`, { signal: AbortSignal.timeout(5000) });
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`GET ${base}/urls failed: ${response.status} ${response.statusText}`);
|
||||
}
|
||||
|
||||
return new WebDAVClient(webdavUrl);
|
||||
const urls = await response.json() as { webdavUrlInternal?: string; webdavUrl?: string };
|
||||
return urls.webdavUrlInternal ?? urls.webdavUrl ?? null;
|
||||
}
|
||||
|
||||
export async function getWebDAVClient(metaCoreUrl: string): Promise<WebDAVClient | null> {
|
||||
const override = process.env.WEBDAV_URL;
|
||||
if (override) {
|
||||
return new WebDAVClient(override);
|
||||
}
|
||||
|
||||
let pending = clientCache.get(metaCoreUrl);
|
||||
if (!pending) {
|
||||
pending = resolveWebDAVUrl(metaCoreUrl).then((webdavUrl) => {
|
||||
if (!webdavUrl) {
|
||||
console.warn(`[webdav-client] meta-core ${metaCoreUrl} advertises no WebDAV URL`);
|
||||
return null;
|
||||
}
|
||||
console.log(`[webdav-client] Using WebDAV endpoint ${webdavUrl} (from ${metaCoreUrl})`);
|
||||
return new WebDAVClient(webdavUrl);
|
||||
});
|
||||
clientCache.set(metaCoreUrl, pending);
|
||||
}
|
||||
|
||||
try {
|
||||
const client = await pending;
|
||||
if (!client) {
|
||||
clientCache.delete(metaCoreUrl);
|
||||
}
|
||||
return client;
|
||||
} catch (error) {
|
||||
clientCache.delete(metaCoreUrl);
|
||||
console.warn(`[webdav-client] Failed to resolve WebDAV URL from ${metaCoreUrl}: ${error instanceof Error ? error.message : String(error)}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# Subtitle Extractor Plugin Test Runner
|
||||
#
|
||||
# Runs integration tests inside a Docker container with ffmpeg available.
|
||||
# Usage:
|
||||
# ./test.sh # Build and run tests
|
||||
# ./test.sh --no-cache # Rebuild without cache
|
||||
# ./test.sh --shell # Start shell in test container for debugging
|
||||
# =============================================================================
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
IMAGE_NAME="metamesh-plugin-subtitle-extractor-test"
|
||||
DOCKERFILE="Dockerfile.test"
|
||||
|
||||
# Parse arguments
|
||||
NO_CACHE=""
|
||||
SHELL_MODE=""
|
||||
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--no-cache)
|
||||
NO_CACHE="--no-cache"
|
||||
;;
|
||||
--shell)
|
||||
SHELL_MODE=1
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --no-cache Rebuild Docker image without cache"
|
||||
echo " --shell Start interactive shell in test container"
|
||||
echo " --help Show this help message"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo "=============================================="
|
||||
echo " Subtitle Extractor Plugin Test Runner"
|
||||
echo "=============================================="
|
||||
echo ""
|
||||
|
||||
# Check if Docker is available
|
||||
if ! command -v docker &> /dev/null; then
|
||||
echo "ERROR: Docker is not installed or not in PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build the test image
|
||||
echo "[1/2] Building test image..."
|
||||
docker build $NO_CACHE -f "$DOCKERFILE" -t "$IMAGE_NAME" .
|
||||
|
||||
if [ -n "$SHELL_MODE" ]; then
|
||||
echo ""
|
||||
echo "[2/2] Starting interactive shell..."
|
||||
echo " Run 'npm test' to execute tests"
|
||||
echo ""
|
||||
docker run --rm -it \
|
||||
-v "$SCRIPT_DIR/test:/app/test:ro" \
|
||||
-v "$SCRIPT_DIR/src:/app/src:ro" \
|
||||
"$IMAGE_NAME" \
|
||||
/bin/bash
|
||||
else
|
||||
echo ""
|
||||
echo "[2/2] Running tests..."
|
||||
echo ""
|
||||
|
||||
# Run tests and capture exit code
|
||||
docker run --rm \
|
||||
-e CI=true \
|
||||
"$IMAGE_NAME"
|
||||
|
||||
EXIT_CODE=$?
|
||||
|
||||
echo ""
|
||||
if [ $EXIT_CODE -eq 0 ]; then
|
||||
echo "=============================================="
|
||||
echo " All tests passed!"
|
||||
echo "=============================================="
|
||||
else
|
||||
echo "=============================================="
|
||||
echo " Tests failed (exit code: $EXIT_CODE)"
|
||||
echo "=============================================="
|
||||
fi
|
||||
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
@@ -0,0 +1,541 @@
|
||||
/**
|
||||
* Subtitle Extractor Plugin Tests
|
||||
*
|
||||
* Pure helpers run anywhere; extraction needs ffmpeg and the generated fixtures
|
||||
* (./test/fixtures/generate-test-fixtures.sh, run by ./test.sh).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
|
||||
import { execSync } from 'child_process';
|
||||
import { existsSync, mkdirSync, readFileSync } from 'fs';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
|
||||
import {
|
||||
manifest,
|
||||
configure,
|
||||
process as processFile,
|
||||
computeMidHash256FromBuffer,
|
||||
sanitizeFilename,
|
||||
readStreamEntries,
|
||||
parseSubtitleStreams,
|
||||
classifySubtitleStreams,
|
||||
outputFor,
|
||||
subtitleFilename,
|
||||
buildExtractionArgs,
|
||||
extractTracks,
|
||||
extractionStats,
|
||||
hasCues,
|
||||
hasExtractedKeySet,
|
||||
videoSubtitleKeys,
|
||||
subtitleRecordKeys,
|
||||
legacySubtitleMigration,
|
||||
linkSubtitles,
|
||||
type ExtractJob,
|
||||
type TextSubtitleStream,
|
||||
} from '../src/plugin.js';
|
||||
import { toLang3, languageToken } from '../src/langs.js';
|
||||
import { flattenMeta, onlyNewKeys } from '../src/meta-shape.js';
|
||||
import { MetaCoreClient } from '../src/meta-core-client.js';
|
||||
import type { CallbackPayload } from '../src/types.js';
|
||||
import { startFakeCore, writesTo, type FakeCore } from './fake-core.js';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const FIXTURES = path.join(__dirname, 'fixtures');
|
||||
const TEMP = '/tmp/subtitle-extractor-test';
|
||||
|
||||
const fixture = (name: string) => path.join(FIXTURES, name);
|
||||
|
||||
function hasFfmpeg(): boolean {
|
||||
try {
|
||||
execSync('ffmpeg -version', { stdio: 'ignore' });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
const ffmpegReady = hasFfmpeg() && existsSync(fixture('with-subs.mkv')) && existsSync(fixture('mov-text.mp4'));
|
||||
|
||||
/** The nested `stream` array exactly as meta-sort's /meta serves it (JSON strings). */
|
||||
function nestedStreamsFor(file: string): string[] {
|
||||
const probe = JSON.parse(
|
||||
execSync(`ffprobe -v quiet -print_format json -show_streams "${file}"`, { encoding: 'utf-8' })
|
||||
) as { streams: Array<Record<string, any>> };
|
||||
return probe.streams.map((s) => {
|
||||
const entry: Record<string, unknown> = {
|
||||
type: s.codec_type,
|
||||
codec: s.codec_name,
|
||||
index: s.index,
|
||||
duration: 'N/A',
|
||||
forced: s.disposition?.forced === 1,
|
||||
default: s.disposition?.default === 1,
|
||||
};
|
||||
if (s.tags?.language) entry.language = s.tags.language;
|
||||
if (s.tags?.title) entry.title = s.tags.title;
|
||||
return JSON.stringify(entry);
|
||||
});
|
||||
}
|
||||
|
||||
const SUB_ENG = { type: 'subtitle', codec: 'ass', index: 3, language: 'eng', title: 'English (Full)', forced: false };
|
||||
const SUB_ENG_SIGNS = { type: 'subtitle', codec: 'ass', index: 4, language: 'eng', title: 'Signs', forced: true };
|
||||
const SUB_FRE = { type: 'subtitle', codec: 'subrip', index: 10, language: 'fre' };
|
||||
const VIDEO = { type: 'video', codec: 'av1', index: 0 };
|
||||
|
||||
beforeEach(() => configure({}));
|
||||
|
||||
describe('manifest', () => {
|
||||
it('declares dependencies, queue and the keys it writes', () => {
|
||||
expect(manifest.id).toBe('subtitle-extractor');
|
||||
expect(manifest.dependencies).toEqual(['file-info', 'ffmpeg']);
|
||||
expect(manifest.defaultQueue).toBe('background');
|
||||
expect(Object.keys(manifest.schema ?? {})).toEqual(['subtitles', 'extractedSubtitles', 'subtitleLanguages']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeMidHash256FromBuffer', () => {
|
||||
it('matches meta-hash for a known buffer', () => {
|
||||
expect(computeMidHash256FromBuffer(Buffer.from('metamesh-still-extractor-cid-vector')))
|
||||
.toBe('bagacbabaecybg7wcyxl7su3dvjleuwgwil5tgeoybrwc35jasqtywb6bnjzk4');
|
||||
});
|
||||
});
|
||||
|
||||
describe('language normalisation (ISO 639-2/B, `und` when unknown)', () => {
|
||||
it('folds 639-1, 639-2/T and BCP-47 onto 639-2/B', () => {
|
||||
expect(toLang3('fr')).toBe('fre');
|
||||
expect(toLang3('fra')).toBe('fre');
|
||||
expect(toLang3('fre')).toBe('fre');
|
||||
expect(toLang3('deu')).toBe('ger');
|
||||
expect(toLang3('zho')).toBe('chi');
|
||||
expect(toLang3('pt-BR')).toBe('por');
|
||||
expect(toLang3('en')).toBe('eng');
|
||||
expect(toLang3('ENG')).toBe('eng');
|
||||
expect(toLang3('English')).toBe('eng');
|
||||
});
|
||||
|
||||
it('maps every "no language" spelling, and junk, to und', () => {
|
||||
for (const v of [undefined, null, '', 'und', 'zxx', 'mis', 'xx', '12', 'toolong']) {
|
||||
expect(toLang3(v)).toBe('und');
|
||||
}
|
||||
});
|
||||
|
||||
it('passes an unlisted but well-formed 3-letter code through', () => {
|
||||
expect(toLang3('tlh')).toBe('tlh');
|
||||
});
|
||||
|
||||
it('is strict about filename tokens', () => {
|
||||
expect(languageToken('fr')).toBe('fre');
|
||||
expect(languageToken('S01')).toBeNull();
|
||||
expect(languageToken('x264')).toBeNull();
|
||||
expect(languageToken('Extended')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('readStreamEntries — every payload shape', () => {
|
||||
const expectOrder = (meta: Record<string, unknown>) =>
|
||||
expect(readStreamEntries(meta).map((s) => s.index)).toEqual([0, 3, 10]);
|
||||
|
||||
it('nested array of JSON strings (meta-sort /meta)', () => {
|
||||
expectOrder({ stream: [VIDEO, SUB_ENG, SUB_FRE].map((s) => JSON.stringify(s)) });
|
||||
});
|
||||
|
||||
it('nested array of objects', () => {
|
||||
expectOrder({ stream: [VIDEO, SUB_ENG, SUB_FRE] });
|
||||
});
|
||||
|
||||
it('nested object keyed by index, ordered numerically', () => {
|
||||
expectOrder({ stream: { '10': JSON.stringify(SUB_FRE), '0': JSON.stringify(VIDEO), '2': JSON.stringify(SUB_ENG) } });
|
||||
});
|
||||
|
||||
it('a JSON string of the array', () => {
|
||||
expectOrder({ stream: JSON.stringify([VIDEO, SUB_ENG, SUB_FRE].map((s) => JSON.stringify(s))) });
|
||||
});
|
||||
|
||||
it('flat meta-core `stream/{n}` keys, stream/10 after stream/2', () => {
|
||||
expectOrder({
|
||||
'stream/10': JSON.stringify(SUB_FRE),
|
||||
'stream/2': JSON.stringify(SUB_ENG),
|
||||
'stream/0': JSON.stringify(VIDEO),
|
||||
});
|
||||
});
|
||||
|
||||
it('skips an unparseable entry and keeps the rest', () => {
|
||||
expect(readStreamEntries({ stream: ['not json', JSON.stringify(SUB_FRE)] }).length).toBe(1);
|
||||
expect(readStreamEntries(undefined)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseSubtitleStreams', () => {
|
||||
it('keeps the global index, per-type ordinal, lang3 and forced flag', () => {
|
||||
const streams = parseSubtitleStreams({ stream: [VIDEO, SUB_ENG, SUB_ENG_SIGNS, { ...SUB_FRE, language: 'fra' }] });
|
||||
expect(streams.map((s) => [s.ordinal, s.streamIndex, s.lang3, s.forced])).toEqual([
|
||||
[0, 3, 'eng', false],
|
||||
[1, 4, 'eng', true],
|
||||
[2, 10, 'fre', false],
|
||||
]);
|
||||
});
|
||||
|
||||
it('uses und when a stream declares no language', () => {
|
||||
expect(parseSubtitleStreams({ 'stream/0': JSON.stringify({ type: 'subtitle', codec: 'subrip', index: 0 }) })[0].lang3).toBe('und');
|
||||
});
|
||||
|
||||
it('falls back to the legacy subtitle_{i}_* fields', () => {
|
||||
const streams = parseSubtitleStreams({ subtitle_0_codec: 'subrip', subtitle_0_language: 'ger' });
|
||||
expect(streams).toHaveLength(1);
|
||||
expect(streams[0].lang3).toBe('ger');
|
||||
expect(streams[0].streamIndex).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifySubtitleStreams / outputFor', () => {
|
||||
const s = (codec: string, i: number) => ({ ordinal: i, streamIndex: i, codec, lang3: 'eng', forced: false });
|
||||
|
||||
it('skips image codecs explicitly and keeps unknown ones apart', () => {
|
||||
const { text, image, unsupported } = classifySubtitleStreams(
|
||||
[s('hdmv_pgs_subtitle', 0), s('dvd_subtitle', 1), s('dvb_subtitle', 2), s('ass', 3), s('eia_608', 4)],
|
||||
'native',
|
||||
);
|
||||
expect(image.map((x) => x.codec)).toEqual(['hdmv_pgs_subtitle', 'dvd_subtitle', 'dvb_subtitle']);
|
||||
expect(text.map((x) => x.codec)).toEqual(['ass']);
|
||||
expect(unsupported.map((x) => x.codec)).toEqual(['eia_608']);
|
||||
});
|
||||
|
||||
it('native keeps ASS as ASS by stream copy, converts mov_text to SRT', () => {
|
||||
expect(outputFor('ass', 'native')).toEqual({ ext: 'ass', encoder: 'copy' });
|
||||
expect(outputFor('subrip', 'native')).toEqual({ ext: 'srt', encoder: 'copy' });
|
||||
expect(outputFor('webvtt', 'native')).toEqual({ ext: 'vtt', encoder: 'copy' });
|
||||
expect(outputFor('mov_text', 'native')).toEqual({ ext: 'srt', encoder: 'srt' });
|
||||
});
|
||||
|
||||
it('an explicit format converts', () => {
|
||||
expect(outputFor('ass', 'srt')).toEqual({ ext: 'srt', encoder: 'srt' });
|
||||
expect(outputFor('subrip', 'vtt')).toEqual({ ext: 'vtt', encoder: 'webvtt' });
|
||||
expect(outputFor('hdmv_pgs_subtitle', 'srt')).toBeNull();
|
||||
});
|
||||
|
||||
it('configure rejects an unknown outputFormat', () => {
|
||||
configure({ outputFormat: 'pgs' });
|
||||
expect(classifySubtitleStreams([s('ass', 0)]).text[0].ext).toBe('ass');
|
||||
});
|
||||
});
|
||||
|
||||
describe('subtitleFilename', () => {
|
||||
const meta = { title: 'Dead Dead Demons', movieYear: 2024 };
|
||||
const stream = (streamIndex: number, lang3: string, forced = false) => ({ ordinal: 0, streamIndex, codec: 'ass', lang3, forced });
|
||||
|
||||
it('two tracks in one language never share a file', () => {
|
||||
const a = subtitleFilename(meta, 'bagvideo', stream(3, 'eng'), 'ass');
|
||||
const b = subtitleFilename(meta, 'bagvideo', stream(4, 'eng', true), 'ass');
|
||||
expect(a).not.toBe(b);
|
||||
expect(a).toBe('Dead Dead Demons (2024)[bagvideo]_subtitle.s3.eng.ass');
|
||||
expect(b).toBe('Dead Dead Demons (2024)[bagvideo]_subtitle.s4.eng.forced.ass');
|
||||
});
|
||||
|
||||
it('falls back to the file name and strips URL-hostile characters', () => {
|
||||
expect(subtitleFilename({ fileName: 'Show #1 100%?.mkv' }, 'c', stream(2, 'und'), 'srt'))
|
||||
.toBe('Show 1 100[c]_subtitle.s2.und.srt');
|
||||
expect(sanitizeFilename('a/b:c*d?e#f%g')).toBe('abcdefg');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildExtractionArgs', () => {
|
||||
it('is ONE ffmpeg invocation with one -map/output per track', () => {
|
||||
const jobs = [3, 4, 10].map((i) => ({
|
||||
stream: { ordinal: i, streamIndex: i, codec: 'ass', lang3: 'eng', forced: false, ext: 'ass', encoder: 'copy' } as TextSubtitleStream,
|
||||
tempFile: `/tmp/t-${i}.ass`,
|
||||
}));
|
||||
const args = buildExtractionArgs('http://core/webdav/watch/a.mkv', jobs);
|
||||
expect(args.filter((a) => a === '-i')).toHaveLength(1);
|
||||
expect(args.filter((a) => a === '-map')).toHaveLength(3);
|
||||
expect(args.join(' ')).toContain('-map 0:3 -c:s copy /tmp/t-3.ass -map 0:4 -c:s copy /tmp/t-4.ass -map 0:10 -c:s copy /tmp/t-10.ass');
|
||||
});
|
||||
|
||||
it('maps by subtitle ordinal when the global index is unknown', () => {
|
||||
const args = buildExtractionArgs('in.mkv', [{
|
||||
stream: { ordinal: 1, codec: 'subrip', lang3: 'ger', forced: false, ext: 'srt', encoder: 'copy' } as TextSubtitleStream,
|
||||
tempFile: 'out.srt',
|
||||
}]);
|
||||
expect(args).toContain('0:s:1');
|
||||
});
|
||||
});
|
||||
|
||||
describe('hasCues', () => {
|
||||
it('does not count a header-only ASS file', () => {
|
||||
expect(hasCues(Buffer.from('[Script Info]\nTitle: x\n[Events]\n'), 'ass')).toBe(false);
|
||||
expect(hasCues(Buffer.from('[Events]\nDialogue: 0,0:00:00.00,0:00:01.00,Default,,0,0,0,,Hi'), 'ass')).toBe(true);
|
||||
expect(hasCues(Buffer.from('1\n00:00:00,000 --> 00:00:01,000\nHi\n'), 'srt')).toBe(true);
|
||||
expect(hasCues(Buffer.from(''), 'srt')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('record keys (METADATA_KEYS.md §8)', () => {
|
||||
it('writes subtitles/<lang3>/<cid> and extractedSubtitles/<lang3>/<cid> leaves', () => {
|
||||
expect(videoSubtitleKeys([{ subCid: 'bagsubfre', lang3: 'fre' }, { subCid: 'bagsubund', lang3: 'und' }])).toEqual({
|
||||
'subtitles/fre/bagsubfre': 'true',
|
||||
'extractedSubtitles/fre/bagsubfre': 'true',
|
||||
'subtitleLanguages/fre': 'true',
|
||||
'languages/fre': 'true',
|
||||
'subtitles/und/bagsubund': 'true',
|
||||
'extractedSubtitles/und/bagsubund': 'true',
|
||||
});
|
||||
});
|
||||
|
||||
it('never writes a bare subtitles/<lang3> node or an und facet', () => {
|
||||
const keys = Object.keys(videoSubtitleKeys([{ subCid: 'c', lang3: 'und' }]));
|
||||
expect(keys.every((k) => k.split('/').length === 3)).toBe(true);
|
||||
});
|
||||
|
||||
it('subtitle record carries the back-pointer and a determined language only', () => {
|
||||
expect(subtitleRecordKeys('bagvideo', 'fre')).toEqual({ 'videos/bagvideo': 'true', subtitleLanguage: 'fre' });
|
||||
expect(subtitleRecordKeys('bagvideo', 'und')).toEqual({ 'videos/bagvideo': 'true' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('hasExtractedKeySet', () => {
|
||||
it('recognises the key-set in both payload forms', () => {
|
||||
expect(hasExtractedKeySet({ extractedSubtitles: { fre: { c: true } } })).toBe(true);
|
||||
expect(hasExtractedKeySet({ 'extractedSubtitles/fre/c': 'true' })).toBe(true);
|
||||
});
|
||||
|
||||
it('does not treat the legacy csv scalar as done — that record needs migrating', () => {
|
||||
expect(hasExtractedKeySet({ extractedSubtitles: 'cidA,cidB' })).toBe(false);
|
||||
expect(hasExtractedKeySet({})).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('legacySubtitleMigration', () => {
|
||||
it('deletes legacy scalars and keeps what they said as key-set leaves', () => {
|
||||
expect(legacySubtitleMigration({
|
||||
extractedSubtitles: 'cidA,cidB',
|
||||
subtitleLanguages: 'eng,fra',
|
||||
subtitles: 'cidS',
|
||||
})).toEqual({
|
||||
deletes: ['extractedSubtitles', 'subtitleLanguages', 'subtitles'],
|
||||
sets: { 'subtitleLanguages/eng': 'true', 'subtitleLanguages/fre': 'true', 'subtitles/und/cidS': 'true' },
|
||||
});
|
||||
});
|
||||
|
||||
it('leaves key-sets alone', () => {
|
||||
expect(legacySubtitleMigration({ subtitles: { fre: { c: true } }, subtitleLanguages: { fre: true } }))
|
||||
.toEqual({ deletes: [], sets: {} });
|
||||
});
|
||||
});
|
||||
|
||||
describe('meta-shape helpers', () => {
|
||||
it('flattens the nested document to the same keys meta-core stores', () => {
|
||||
const nested = { fileType: 'video', fileinfo: { duration: 1433.089 }, subtitles: { fre: { bagsub: true } }, stream: ['{"a":1}'] };
|
||||
expect(flattenMeta(nested)).toEqual({
|
||||
fileType: 'video',
|
||||
'fileinfo/duration': '1433.089',
|
||||
'subtitles/fre/bagsub': 'true',
|
||||
'stream/0': '{"a":1}',
|
||||
});
|
||||
});
|
||||
|
||||
it('onlyNewKeys drops what the record already holds', () => {
|
||||
expect(onlyNewKeys({ 'a/b': 'true', c: 'x' }, { 'a/b': 'true', c: 'y', d: 'true' })).toEqual({ c: 'y', d: 'true' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('process — skip logic (no I/O)', () => {
|
||||
const collect = () => {
|
||||
const seen: CallbackPayload[] = [];
|
||||
return { seen, cb: async (p: CallbackPayload) => { seen.push(p); } };
|
||||
};
|
||||
const req = (existingMeta: Record<string, unknown>) => ({
|
||||
taskId: 't1',
|
||||
cid: 'bagvideo',
|
||||
filePath: '/files/watch/test.mkv',
|
||||
callbackUrl: 'http://localhost/callback',
|
||||
metaCoreUrl: 'http://127.0.0.1:9', // unreachable: every case must bail first
|
||||
existingMeta,
|
||||
});
|
||||
|
||||
it('skips non-video files', async () => {
|
||||
const { seen, cb } = collect();
|
||||
await processFile(req({ fileType: 'subtitle' }), cb);
|
||||
expect(seen[0]).toMatchObject({ status: 'skipped', reason: 'Not a video file' });
|
||||
});
|
||||
|
||||
it('skips a record whose extractedSubtitles key-set is already there (nested form)', async () => {
|
||||
const { seen, cb } = collect();
|
||||
await processFile(req({ fileType: 'video', extractedSubtitles: { eng: { c: true } }, stream: [JSON.stringify(SUB_ENG)] }), cb);
|
||||
expect(seen[0]).toMatchObject({ status: 'skipped', reason: 'Subtitles already extracted' });
|
||||
});
|
||||
|
||||
it('skips a video with no subtitle streams', async () => {
|
||||
const { seen, cb } = collect();
|
||||
await processFile(req({ fileType: 'video', stream: [JSON.stringify(VIDEO)] }), cb);
|
||||
expect(seen[0]).toMatchObject({ status: 'skipped', reason: 'No subtitle streams found' });
|
||||
});
|
||||
|
||||
it('skips a video whose only subtitles are image-based', async () => {
|
||||
const { seen, cb } = collect();
|
||||
await processFile(req({
|
||||
fileType: 'video',
|
||||
stream: [VIDEO, { type: 'subtitle', codec: 'hdmv_pgs_subtitle', index: 1, language: 'eng' }].map((s) => JSON.stringify(s)),
|
||||
}), cb);
|
||||
expect(seen[0]).toMatchObject({ status: 'skipped', reason: 'Only image-based subtitles (cannot convert to text)' });
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!ffmpegReady)('extractTracks (ffmpeg)', () => {
|
||||
beforeAll(() => mkdirSync(TEMP, { recursive: true }));
|
||||
|
||||
const jobsFor = (file: string, tag: string): ExtractJob[] =>
|
||||
classifySubtitleStreams(parseSubtitleStreams({ stream: nestedStreamsFor(file) }), 'native').text
|
||||
.map((stream) => ({ stream, tempFile: path.join(TEMP, `${tag}-${stream.ordinal}.${stream.ext}`) }));
|
||||
|
||||
it('extracts every text track of the MKV in a single pass', async () => {
|
||||
const jobs = jobsFor(fixture('with-subs.mkv'), 'single');
|
||||
expect(jobs.map((j) => [j.stream.lang3, j.stream.ext, j.stream.forced])).toEqual([
|
||||
['eng', 'srt', false],
|
||||
['fre', 'ass', true],
|
||||
['und', 'srt', false],
|
||||
]);
|
||||
const before = extractionStats.ffmpegPasses;
|
||||
const result = await extractTracks(fixture('with-subs.mkv'), jobs, 60000);
|
||||
expect(extractionStats.ffmpegPasses - before).toBe(1);
|
||||
expect(result.passes).toBe(1);
|
||||
const [eng, fre, und] = result.outputs.map((o) => o.data?.toString('utf8') ?? '');
|
||||
expect(eng).toContain('quick brown fox');
|
||||
expect(fre).toContain('[Script Info]');
|
||||
expect(fre).toMatch(/^Dialogue:.*renard brun/m);
|
||||
expect(und).toContain('-->');
|
||||
});
|
||||
|
||||
it('converts mov_text to SRT', async () => {
|
||||
const jobs = jobsFor(fixture('mov-text.mp4'), 'movtext');
|
||||
expect(jobs.map((j) => [j.stream.lang3, j.stream.ext, j.stream.encoder])).toEqual([['ger', 'srt', 'srt']]);
|
||||
const result = await extractTracks(fixture('mov-text.mp4'), jobs, 60000);
|
||||
expect(result.outputs[0].data?.toString('utf8')).toContain('schnelle braune Fuchs');
|
||||
});
|
||||
|
||||
it('salvages good tracks one by one when the shared pass fails', async () => {
|
||||
const jobs = jobsFor(fixture('with-subs.mkv'), 'salvage');
|
||||
const broken: ExtractJob = { ...jobs[0], stream: { ...jobs[0].stream, streamIndex: 42 }, tempFile: path.join(TEMP, 'salvage-broken.srt') };
|
||||
const result = await extractTracks(fixture('with-subs.mkv'), [jobs[0], broken, jobs[1]], 60000);
|
||||
expect(result.passes).toBe(4);
|
||||
expect(result.outputs.map((o) => o.data !== null)).toEqual([true, false, true]);
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!ffmpegReady)('process — end to end against a fake core', () => {
|
||||
let core: FakeCore;
|
||||
const VIDEO_CID = 'bagacbabaevideo';
|
||||
const videoPath = '/files/watch/show/with-subs.mkv';
|
||||
let nestedMeta: Record<string, unknown>;
|
||||
|
||||
beforeAll(async () => {
|
||||
core = await startFakeCore();
|
||||
core.files.set('/watch/show/with-subs.mkv', readFileSync(fixture('with-subs.mkv')));
|
||||
process.env.WEBDAV_URL = core.webdavUrl;
|
||||
process.env.CACHE_PATH = TEMP;
|
||||
nestedMeta = {
|
||||
fileType: 'video',
|
||||
title: 'Test Show',
|
||||
fileName: 'with-subs.mkv',
|
||||
fileinfo: { duration: 3, formatName: 'matroska,webm' },
|
||||
stream: nestedStreamsFor(fixture('with-subs.mkv')),
|
||||
// Legacy scalars from the `_add` era, which must not survive next to key-sets.
|
||||
extractedSubtitles: 'bagold1,bagold2',
|
||||
subtitleLanguages: 'eng',
|
||||
};
|
||||
core.records.set(VIDEO_CID, flattenMeta(nestedMeta));
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
delete process.env.WEBDAV_URL;
|
||||
await core.close();
|
||||
});
|
||||
|
||||
const run = async (existingMeta: Record<string, unknown>) => {
|
||||
const seen: CallbackPayload[] = [];
|
||||
await processFile({
|
||||
taskId: 'task-e2e',
|
||||
cid: VIDEO_CID,
|
||||
filePath: videoPath,
|
||||
callbackUrl: 'http://unused',
|
||||
metaCoreUrl: core.url,
|
||||
existingMeta,
|
||||
}, async (p) => { seen.push(p); });
|
||||
return seen[0];
|
||||
};
|
||||
|
||||
it('extracts in one pass, uploads, and links subtitles/<lang3>/<cid>', async () => {
|
||||
const passes = extractionStats.ffmpegPasses;
|
||||
const result = await run(nestedMeta);
|
||||
expect(result.status).toBe('completed');
|
||||
expect(extractionStats.ffmpegPasses - passes).toBe(1);
|
||||
|
||||
const uploads = [...core.files.keys()].filter((p) => p.startsWith('/plugin/subtitle-extractor/')).sort();
|
||||
expect(uploads).toEqual([
|
||||
'/plugin/subtitle-extractor/Test Show[bagacbabaevideo]_subtitle.s1.eng.srt',
|
||||
'/plugin/subtitle-extractor/Test Show[bagacbabaevideo]_subtitle.s2.fre.forced.ass',
|
||||
'/plugin/subtitle-extractor/Test Show[bagacbabaevideo]_subtitle.s3.und.srt',
|
||||
]);
|
||||
const cidOf = (suffix: string) => computeMidHash256FromBuffer(core.files.get(uploads.find((u) => u.endsWith(suffix))!)!);
|
||||
const eng = cidOf('eng.srt');
|
||||
const fre = cidOf('fre.forced.ass');
|
||||
const und = cidOf('und.srt');
|
||||
|
||||
const video = core.records.get(VIDEO_CID)!;
|
||||
expect(video).toMatchObject({
|
||||
[`subtitles/eng/${eng}`]: 'true',
|
||||
[`subtitles/fre/${fre}`]: 'true',
|
||||
[`subtitles/und/${und}`]: 'true',
|
||||
[`extractedSubtitles/eng/${eng}`]: 'true',
|
||||
[`extractedSubtitles/fre/${fre}`]: 'true',
|
||||
[`extractedSubtitles/und/${und}`]: 'true',
|
||||
'subtitleLanguages/eng': 'true',
|
||||
'subtitleLanguages/fre': 'true',
|
||||
'languages/eng': 'true',
|
||||
'languages/fre': 'true',
|
||||
});
|
||||
expect(video.extractedSubtitles).toBeUndefined();
|
||||
expect(video.subtitleLanguages).toBeUndefined();
|
||||
expect(video['subtitleLanguages/und']).toBeUndefined();
|
||||
|
||||
expect(core.records.get(fre)).toEqual({ [`videos/${VIDEO_CID}`]: 'true', subtitleLanguage: 'fre' });
|
||||
expect(core.records.get(und)).toEqual({ [`videos/${VIDEO_CID}`]: 'true' });
|
||||
|
||||
// The video PATCH carried only new keys — nothing the record already had.
|
||||
const patch = writesTo(core, VIDEO_CID).find((c) => c.method === 'PATCH')!;
|
||||
expect(Object.keys(patch.body as object).some((k) => ['fileType', 'title', 'stream/0'].includes(k))).toBe(false);
|
||||
});
|
||||
|
||||
it('skips once the key-set is on the record', async () => {
|
||||
const result = await run(core.records.get(VIDEO_CID)!);
|
||||
expect(result).toMatchObject({ status: 'skipped', reason: 'Subtitles already extracted' });
|
||||
});
|
||||
|
||||
it('re-links from existing outputs without reading the video again', async () => {
|
||||
// Metadata lost, output files still there.
|
||||
const bare = flattenMeta({ ...nestedMeta, extractedSubtitles: undefined, subtitleLanguages: undefined });
|
||||
const passes = extractionStats.ffmpegPasses;
|
||||
const gets = core.fileGets.get('/watch/show/with-subs.mkv');
|
||||
const result = await run(bare);
|
||||
expect(result.status).toBe('completed');
|
||||
expect(extractionStats.ffmpegPasses).toBe(passes);
|
||||
expect(core.fileGets.get('/watch/show/with-subs.mkv')).toBe(gets);
|
||||
});
|
||||
|
||||
it('a second link over an up-to-date record writes nothing', async () => {
|
||||
const before = core.calls.length;
|
||||
const video = core.records.get(VIDEO_CID)!;
|
||||
const produced = Object.keys(video)
|
||||
.filter((k) => k.startsWith('subtitles/'))
|
||||
.map((k) => ({ lang3: k.split('/')[1], subCid: k.split('/')[2] }));
|
||||
expect(await linkSubtitles(new MetaCoreClient(core.url), VIDEO_CID, video, produced)).toBe(true);
|
||||
expect(core.calls.length).toBe(before);
|
||||
});
|
||||
|
||||
it('fails the task when meta-core rejects the write', async () => {
|
||||
core.failPatch = true;
|
||||
try {
|
||||
const result = await run(flattenMeta({ fileType: 'video', title: 'Test Show', stream: nestedStreamsFor(fixture('with-subs.mkv')) }));
|
||||
expect(result).toMatchObject({ status: 'failed', error: 'meta-core write failed' });
|
||||
} finally {
|
||||
core.failPatch = false;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,147 @@
|
||||
/**
|
||||
* In-process stand-in for the two HTTP surfaces a plugin talks to:
|
||||
* - WebDAV at /webdav/... (JSON directory listing, HEAD, Range GET, PUT)
|
||||
* - meta API at /meta/... (GET record / property, PATCH merge, DELETE property)
|
||||
* Records are stored FLAT (`subtitles/fre/<cid>`), the way meta-core keeps them.
|
||||
*/
|
||||
|
||||
import http from 'http';
|
||||
import type { AddressInfo } from 'net';
|
||||
|
||||
export interface FakeCall {
|
||||
method: string;
|
||||
path: string;
|
||||
body?: unknown;
|
||||
}
|
||||
|
||||
export interface FakeCore {
|
||||
url: string;
|
||||
webdavUrl: string;
|
||||
/** Keyed by path under /files, e.g. `/watch/show/a.mkv`. */
|
||||
files: Map<string, Buffer>;
|
||||
records: Map<string, Record<string, string>>;
|
||||
calls: FakeCall[];
|
||||
/** GET requests per file path (each ffmpeg pass opens the file at least once). */
|
||||
fileGets: Map<string, number>;
|
||||
failPatch: boolean;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
function readBody(req: http.IncomingMessage): Promise<Buffer> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const chunks: Buffer[] = [];
|
||||
req.on('data', (c) => chunks.push(c));
|
||||
req.on('end', () => resolve(Buffer.concat(chunks)));
|
||||
req.on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
export async function startFakeCore(): Promise<FakeCore> {
|
||||
const core = {
|
||||
files: new Map<string, Buffer>(),
|
||||
records: new Map<string, Record<string, string>>(),
|
||||
calls: [] as FakeCall[],
|
||||
fileGets: new Map<string, number>(),
|
||||
failPatch: false,
|
||||
};
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
const url = new URL(req.url ?? '/', 'http://x');
|
||||
const pathname = decodeURIComponent(url.pathname);
|
||||
const send = (status: number, body?: unknown, headers: Record<string, string> = {}) => {
|
||||
if (body === undefined) {
|
||||
res.writeHead(status, headers);
|
||||
res.end();
|
||||
} else if (Buffer.isBuffer(body)) {
|
||||
res.writeHead(status, headers);
|
||||
res.end(body);
|
||||
} else {
|
||||
res.writeHead(status, { 'Content-Type': 'application/json', ...headers });
|
||||
res.end(JSON.stringify(body));
|
||||
}
|
||||
};
|
||||
|
||||
if (pathname.startsWith('/webdav')) {
|
||||
const p = pathname.slice('/webdav'.length) || '/';
|
||||
if (req.method === 'PUT') {
|
||||
core.files.set(p, await readBody(req));
|
||||
return send(201);
|
||||
}
|
||||
if (p.endsWith('/')) {
|
||||
const dir = p.replace(/\/$/, '');
|
||||
const entries = new Map<string, { name: string; type: string; size: number }>();
|
||||
for (const [fp, data] of core.files) {
|
||||
if (!fp.startsWith(`${dir}/`)) continue;
|
||||
const rest = fp.slice(dir.length + 1);
|
||||
const slash = rest.indexOf('/');
|
||||
if (slash === -1) entries.set(rest, { name: rest, type: 'file', size: data.length });
|
||||
else entries.set(rest.slice(0, slash), { name: rest.slice(0, slash), type: 'directory', size: 512 });
|
||||
}
|
||||
return send(200, [...entries.values()]);
|
||||
}
|
||||
const data = core.files.get(p);
|
||||
if (!data) return send(404);
|
||||
if (req.method === 'HEAD') {
|
||||
return send(200, undefined, { 'Content-Length': String(data.length), 'Accept-Ranges': 'bytes' });
|
||||
}
|
||||
core.fileGets.set(p, (core.fileGets.get(p) ?? 0) + 1);
|
||||
const range = /^bytes=(\d+)-(\d*)$/.exec(req.headers.range ?? '');
|
||||
if (range) {
|
||||
const start = Number(range[1]);
|
||||
const end = range[2] ? Math.min(Number(range[2]), data.length - 1) : data.length - 1;
|
||||
if (start >= data.length) return send(416, undefined, { 'Content-Range': `bytes */${data.length}` });
|
||||
return send(206, data.subarray(start, end + 1), {
|
||||
'Content-Range': `bytes ${start}-${end}/${data.length}`,
|
||||
'Content-Length': String(end - start + 1),
|
||||
'Accept-Ranges': 'bytes',
|
||||
});
|
||||
}
|
||||
return send(200, data, { 'Content-Length': String(data.length), 'Accept-Ranges': 'bytes' });
|
||||
}
|
||||
|
||||
const m = /^\/meta\/([^/]+)(?:\/(.+))?$/.exec(pathname);
|
||||
if (m) {
|
||||
const [, hash, key] = m;
|
||||
const raw = await readBody(req);
|
||||
const body = raw.length ? JSON.parse(raw.toString('utf8')) : undefined;
|
||||
if (req.method !== 'GET') core.calls.push({ method: req.method ?? '', path: pathname, body });
|
||||
const record = core.records.get(hash);
|
||||
|
||||
if (req.method === 'GET' && !key) return record ? send(200, { metadata: record }) : send(404, { error: 'not found' });
|
||||
if (req.method === 'GET' && key) {
|
||||
return record && key in record ? send(200, { value: record[key] }) : send(404, { error: 'Property not found' });
|
||||
}
|
||||
if (req.method === 'PATCH' && !key) {
|
||||
if (core.failPatch) return send(500, { error: 'boom' });
|
||||
core.records.set(hash, { ...(record ?? {}), ...(body as Record<string, string>) });
|
||||
return send(200, { status: 'ok' });
|
||||
}
|
||||
if (req.method === 'DELETE' && key) {
|
||||
if (record) delete record[key];
|
||||
return send(200, { status: 'ok' });
|
||||
}
|
||||
}
|
||||
send(404, { error: 'no route' });
|
||||
});
|
||||
|
||||
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
|
||||
const { port } = server.address() as AddressInfo;
|
||||
const url = `http://127.0.0.1:${port}`;
|
||||
|
||||
return {
|
||||
...core,
|
||||
get failPatch() { return core.failPatch; },
|
||||
set failPatch(v: boolean) { core.failPatch = v; },
|
||||
url,
|
||||
webdavUrl: `${url}/webdav`,
|
||||
close: () => new Promise<void>((resolve) => {
|
||||
server.closeAllConnections?.();
|
||||
server.close(() => resolve());
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
/** Mutating calls (PATCH / DELETE) against one record. */
|
||||
export function writesTo(core: FakeCore, hash: string): FakeCall[] {
|
||||
return core.calls.filter((c) => c.path === `/meta/${hash}` || c.path.startsWith(`/meta/${hash}/`));
|
||||
}
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
#!/bin/bash
|
||||
# Generate media for subtitle-extractor integration tests.
|
||||
# Run inside the test container (ffmpeg required); safe to run locally too.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
echo "Generating test fixtures..."
|
||||
|
||||
# 1. MKV with three text tracks — the shape real releases have:
|
||||
# s:0 SubRip, language eng
|
||||
# s:1 ASS, language fre, forced (must stay ASS under outputFormat=native)
|
||||
# s:2 SubRip, language und (nothing declared)
|
||||
if [ ! -f "with-subs.mkv" ]; then
|
||||
echo "Creating with-subs.mkv..."
|
||||
cat > eng.srt << 'EOF'
|
||||
1
|
||||
00:00:00,000 --> 00:00:01,000
|
||||
The quick brown fox jumps over the lazy dog.
|
||||
|
||||
2
|
||||
00:00:01,000 --> 00:00:02,000
|
||||
Every good story starts somewhere.
|
||||
EOF
|
||||
cat > fre.srt << 'EOF'
|
||||
1
|
||||
00:00:00,000 --> 00:00:01,000
|
||||
Le renard brun saute par-dessus le chien paresseux.
|
||||
|
||||
2
|
||||
00:00:01,000 --> 00:00:02,000
|
||||
Chaque bonne histoire commence quelque part.
|
||||
EOF
|
||||
cat > und.srt << 'EOF'
|
||||
1
|
||||
00:00:00,500 --> 00:00:01,500
|
||||
~ ~ ~
|
||||
EOF
|
||||
|
||||
ffmpeg -y -f lavfi -i testsrc=duration=3:size=160x120:rate=12 \
|
||||
-i eng.srt -i fre.srt -i und.srt \
|
||||
-map 0:v -map 1:s -map 2:s -map 3:s \
|
||||
-c:v libx264 -preset ultrafast -crf 35 -pix_fmt yuv420p \
|
||||
-c:s:0 srt -c:s:1 ass -c:s:2 srt \
|
||||
-metadata:s:s:0 language=eng \
|
||||
-metadata:s:s:1 language=fre -disposition:s:1 forced \
|
||||
-metadata:s:s:2 language=und \
|
||||
with-subs.mkv
|
||||
rm -f eng.srt fre.srt und.srt
|
||||
fi
|
||||
|
||||
# 2. MP4 with a mov_text track tagged in ISO 639-2/T (`deu`) — must be converted
|
||||
# to SRT and normalised to `ger`.
|
||||
if [ ! -f "mov-text.mp4" ]; then
|
||||
echo "Creating mov-text.mp4..."
|
||||
cat > ger.srt << 'EOF'
|
||||
1
|
||||
00:00:00,000 --> 00:00:01,000
|
||||
Der schnelle braune Fuchs springt über den faulen Hund.
|
||||
EOF
|
||||
ffmpeg -y -f lavfi -i testsrc=duration=2:size=160x120:rate=12 \
|
||||
-i ger.srt \
|
||||
-map 0:v -map 1:s \
|
||||
-c:v libx264 -preset ultrafast -crf 35 -pix_fmt yuv420p \
|
||||
-c:s mov_text -metadata:s:s:0 language=deu \
|
||||
mov-text.mp4
|
||||
rm -f ger.srt
|
||||
fi
|
||||
|
||||
echo "Test fixtures generated:"
|
||||
ls -la *.mkv *.mp4 2>/dev/null || true
|
||||
+5
-3
@@ -3,13 +3,15 @@
|
||||
"target": "ES2022",
|
||||
"module": "NodeNext",
|
||||
"moduleResolution": "NodeNext",
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"lib": ["ES2022"],
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"declaration": true,
|
||||
"resolveJsonModule": true
|
||||
"sourceMap": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globals: true,
|
||||
environment: 'node',
|
||||
include: ['test/**/*.test.ts'],
|
||||
testTimeout: 30000, // 30s timeout for FFprobe tests
|
||||
hookTimeout: 60000, // 60s for setup/teardown
|
||||
coverage: {
|
||||
provider: 'v8',
|
||||
reporter: ['text', 'json', 'html'],
|
||||
include: ['src/**/*.ts'],
|
||||
exclude: ['src/types.ts'],
|
||||
},
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user