meta-core/docs/metadata-storage-structure.md

Redis Metadata Storage Architecture

This document describes how MetaMesh stores file metadata in Redis, the key structure, plugin metadata contributions, and special cases like FFmpeg stream handling.

Overview

MetaMesh uses Redis as its primary metadata storage, storing file metadata as Redis Hashes with flattened key-value pairs. This design enables:

• Efficient nested object storage without JSON serialization overhead
• Individual field updates without full document rewrites
• Cross-service access via Redis pub/sub notifications
• Fast VFS lookups using content-based hash IDs

Key Structure

Hash-Based Storage

Each file's metadata is stored as a Redis Hash:

Hash Key:    file:{hashId}
Hash Fields: property/path (e.g., "fileinfo/duration", "titles/eng")
Hash Values: String values

Example Redis commands:

# Store file metadata
HMSET file:bafkr4ih5kapbjzq... \
  "cid_midhash256" "bafkr4ih5kapbjzq..." \
  "filePath" "/files/watch/movie.mkv" \
  "fileinfo/duration" "7200.5" \
  "fileinfo/formatName" "matroska,webm"

# Retrieve all metadata
HGETALL file:bafkr4ih5kapbjzq...

# Get single field
HGET file:bafkr4ih5kapbjzq... "fileinfo/duration"

File Index

A Redis Set tracks all known file hash IDs:

Key:   file:__index__
Type:  Set
Value: {hashId1, hashId2, hashId3, ...}

This enables fast bulk queries like "get all files" without scanning keys.

Primary Hash ID

The primary identifier for each file is cid_midhash256 - an IPFS-compatible CID generated from:

1. SHA-256 hash of the middle 1MB of the file
2. File size in bytes
3. IPFS CID v1 encoding (multicodec 0x1000)

Example: bafkr4ih5kapbjzqvylwj7kx7zmkpxn6qj5xw...

This provides:

• Content-based addressing (same file = same ID)
• Fast computation (only reads 1MB)
• IPFS compatibility for future distributed storage

Collision Handling

If two different files produce the same midhash (rare), the system falls back to full SHA-256:

// Normal case
hashId = midHash256;  // "bafkr4ih5kapbjzq..."

// Collision detected (different filePath with same midhash)
hashId = metadata['cid_sha2-256'];  // Full hash fallback

Metadata Flattening

Nested objects are flattened into path-based keys for efficient Redis storage.

Flattening Rules

// Input (nested object)
{
  title: "Inception",
  video: {
    codec: "h265",
    width: "1920"
  },
  streams: [
    { language: "eng" },
    { language: "jpn" }
  ]
}

// Output (flattened for Redis)
title                  = "Inception"
video/codec            = "h265"
video/width            = "1920"
streams/0/language     = "eng"
streams/1/language     = "jpn"

Key rules:

• Nested objects use / as path separator
• Arrays use numeric indices (0, 1, 2...)
• All values stored as strings
• null/undefined stored as empty strings
• Numbers and booleans converted via String()

Reconstruction

When reading, the flattened structure is reconstructed into nested objects:

const metadata = await kvClient.getMetadataFlat(hashId);
// Returns: { title: "Inception", video: { codec: "h265", width: "1920" }, ... }

Plugin Metadata Structure

Each plugin writes metadata using a prefixed key pattern:

{plugin-id}/{field}                   # Simple field
{plugin-id}/{group}/{field}           # Grouped field
{plugin-id}/{group}/{index}/{field}   # Indexed field (for arrays)

Plugin Contribution Flow

File Discovered
    ↓
[FileProcessorPiscina]
    ├─ Compute midhash256
    ├─ Store preliminary metadata
    └─ Dispatch plugin tasks
         ↓
[Plugin Processing]
    ├─ file-info    → fileType, mimeType, sizeByte
    ├─ ffmpeg       → fileinfo/duration, fileinfo/streamdetails/...
    ├─ filename-parser → titles/eng, season, episode, videoType
    ├─ anime-detector  → anime, titles/jpn, titles/jpl
    ├─ tmdb        → tmdbid, plot/eng, genres
    └─ full-hash   → cid_sha2-256, cid_sha1
         ↓
[Plugin Callbacks]
    └─ Merge metadata into Redis via HMSET

Core Metadata Types

HashMeta - Content Identification

cid_crc32         = "bafkr..."    # CRC32 (multicodec 0x0132)
cid_md5           = "bafkr..."    # MD5 (multicodec 0xd5)
cid_sha1          = "bafkr..."    # SHA-1 (multicodec 0x11)
cid_sha2-256      = "bafkr..."    # SHA-256 (multicodec 0x12)
cid_midhash256    = "bafkr..."    # Middle-hash (PRIMARY ID, multicodec 0x1000)

FileStatMeta - File Properties

fileType          = "video"              # video, audio, image, text
mimeType          = "video/x-matroska"   # MIME type
sizeByte          = "4294967296"         # File size in bytes

VideoMeta - Classification

videoType         = "movie"              # movie, episode, unknown
originalTitle     = "Inception"
titles/eng        = "Inception"          # Localized titles
titles/jpn        = "インセプション"
season            = "1"
episode           = "5"
movieYear         = "2010"

AnimeMeta - Anime Detection

anime             = "true"               # "true" or "false"
titles/jpn        = "進撃の巨人"         # Japanese title
titles/jpl        = "Shingeki no Kyojin" # Romaji

SubtitleMeta - Subtitle References

subtitles/eng/0   = "bafkr..."          # Reference to subtitle file CID
subtitles/eng/1   = "bafks..."
subtitles/jpn/0   = "bafkt..."

JellyfinMeta - Comprehensive Metadata

imdbid            = "tt1375666"
tmdbid            = "27205"
anidbid           = "12345"
plot/eng          = "A movie about..."
genre/0           = "Action"
genre/1           = "Sci-Fi"
rating            = "8.8"
mpaa              = "PG-13"
releasedate       = "2010-07-16"
art/poster        = "bafkr..."          # Reference to poster image CID
art/fanart        = "bafks..."          # Reference to backdrop image CID

FFmpeg Plugin - Special Case

The FFmpeg plugin extracts detailed stream information with a specific nested structure.

Stream Details Structure

fileinfo/duration                              = "7200.5"
fileinfo/formatName                            = "matroska,webm"
fileinfo/streamdetails/video/0/codec           = "h264"
fileinfo/streamdetails/video/0/width           = "1920"
fileinfo/streamdetails/video/0/height          = "1080"
fileinfo/streamdetails/video/0/aspect          = "16:9"
fileinfo/streamdetails/video/0/bitrate         = "8000000"
fileinfo/streamdetails/audio/0/codec           = "aac"
fileinfo/streamdetails/audio/0/language        = "eng"
fileinfo/streamdetails/audio/0/channels        = "6"
fileinfo/streamdetails/audio/1/codec           = "dts"
fileinfo/streamdetails/audio/1/language        = "fra"
fileinfo/streamdetails/audio/1/channels        = "6"
fileinfo/streamdetails/subtitle/0/language     = "eng"
fileinfo/streamdetails/subtitle/0/format       = "subrip"
fileinfo/streamdetails/subtitle/1/language     = "jpn"
fileinfo/streamdetails/subtitle/1/format       = "ass"
fileinfo/streamdetails/embeddedimage/0/type    = "cover"

Stream Type Keys

Stream Type	Key Pattern	Description
Video	fileinfo/streamdetails/video/{n}/...	Video streams (0, 1, 2...)
Audio	fileinfo/streamdetails/audio/{n}/...	Audio tracks
Subtitle	fileinfo/streamdetails/subtitle/{n}/...	Embedded subtitles
Embedded Image	fileinfo/streamdetails/embeddedimage/{n}/...	Cover art, thumbnails

Common Stream Fields

Video streams:

• codec - h264, hevc, av1, vp9, etc.
• width, height - Resolution
• aspect - Aspect ratio
• bitrate - Video bitrate
• profile - Codec profile (main, high, etc.)
• level - Codec level

Audio streams:

• codec - aac, dts, ac3, flac, etc.
• language - ISO 639-2 code (eng, jpn, fra)
• channels - Channel count (2, 6, 8)
• bitrate - Audio bitrate
• samplerate - Sample rate

Subtitle streams:

• language - ISO 639-2 code
• format - subrip, ass, pgs, vobsub

Redis Pub/Sub Notifications

MetaMesh uses Redis pub/sub to notify other services of metadata changes.

Channels

meta-sort:file:batch    # Batch file updates (every 5 seconds)
meta-sort:scan:reset    # Full rescan notification

Batch Update Message Format

{
  "timestamp": 1704067200000,
  "changes": [
    { "action": "add", "hashId": "bafkr4ih5..." },
    { "action": "update", "hashId": "bafkr4ih5..." },
    { "action": "remove", "hashId": "bafkr4ih5..." }
  ]
}

This enables meta-fuse to update its VFS in near real-time without polling.

Complete Example

Here's a complete metadata example for a movie file:

# Query all metadata
redis-cli HGETALL file:bafkr4ih5kapbjzq...

# Result (reconstructed as nested object)
{
  # Content identification
  "cid_midhash256": "bafkr4ih5kapbjzq...",
  "cid_sha2-256": "bafks7ij2kbpck...",
  "filePath": "/files/watch/movies/Inception.2010.1080p.BluRay.x264.mkv",

  # File properties
  "fileType": "video",
  "mimeType": "video/x-matroska",
  "sizeByte": "4294967296",

  # Classification
  "videoType": "movie",
  "originalTitle": "Inception",
  "movieYear": "2010",

  # Localized titles
  "titles": {
    "eng": "Inception",
    "fra": "Inception",
    "jpn": "インセプション"
  },

  # FFmpeg stream details
  "fileinfo": {
    "duration": "8878.5",
    "formatName": "matroska,webm",
    "streamdetails": {
      "video": {
        "0": {
          "codec": "h264",
          "width": "1920",
          "height": "1080",
          "aspect": "16:9",
          "bitrate": "8000000"
        }
      },
      "audio": {
        "0": { "codec": "dts", "language": "eng", "channels": "6" },
        "1": { "codec": "ac3", "language": "fra", "channels": "6" }
      },
      "subtitle": {
        "0": { "language": "eng", "format": "subrip" },
        "1": { "language": "fra", "format": "subrip" }
      }
    }
  },

  # External metadata (TMDB)
  "tmdbid": "27205",
  "imdbid": "tt1375666",
  "plot": {
    "eng": "A thief who steals corporate secrets through dream-sharing technology..."
  },
  "genre": ["Action", "Science Fiction", "Adventure"],
  "rating": "8.4",
  "releasedate": "2010-07-16",

  # Media assets
  "art": {
    "poster": "bafkr4poster...",
    "fanart": "bafkr4fanart..."
  },

  # Detection flags
  "anime": "false"
}

Key Implementation Files

Component	File Path
Redis Client	packages/meta-sort/packages/meta-sort-core/src/kv/RedisClient.ts
Metadata Utils	packages/meta-sort/packages/meta-sort-core/src/kv/MetadataUtils.ts
KV Interface	packages/meta-sort/packages/meta-sort-core/src/kv/IKVClient.ts
File Processor	packages/meta-sort/packages/meta-sort-core/src/logic/fileProcessor/FileProcessorPiscina.ts
Pub/Sub Pipeline	packages/meta-sort/packages/meta-sort-core/src/logic/pipeline/StreamingPipeline.ts
Type Definitions	packages/meta-interface/src/lib/metadata-type/*.ts

Debugging

# List all file keys
docker exec meta-sort-dev redis-cli KEYS 'file:*' | head -20

# Get all metadata for a file
docker exec meta-sort-dev redis-cli HGETALL 'file:bafkr4ih5...'

# Get specific field
docker exec meta-sort-dev redis-cli HGET 'file:bafkr4ih5...' 'fileinfo/duration'

# Count total files
docker exec meta-sort-dev redis-cli SCARD 'file:__index__'

# Monitor metadata changes in real-time
docker exec meta-sort-dev redis-cli SUBSCRIBE 'meta-sort:file:batch'