Compare commits

..
2 Commits
Author SHA1 Message Date
Worph f367ed037f Report inbox: /api/reports in its own bounded reports.db, dashboard Reports page (list, filter, triage, copy as text)
Build and Push Docker Image / build (push) Failing after 1m20s
2026-11-04 11:43:01 +00:00
Worph cef9dba59e migrate
Build and Push Docker Image / build (push) Failing after 2m28s
2026-11-04 11:55:00 +01:00
14 changed files with 2018 additions and 9 deletions
+14
View File
@@ -31,6 +31,7 @@ in the Go binary (no database port at all) — all metadata I/O is mediated by t
| **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. | | **UUID-rooted storage** | Roots are UUIDv7 (Crockford Base32, ULID layout); CIDs are reverse-index aliases. Schema-version sentinel refuses to boot against stale data. |
| **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. | | **Identity keystore** | Multi-account secp256k1 signing keys under `/meta-core/identity/accounts/`, created from the dashboard's Identity menu (never auto-generated). A private key is returned only by generate, or by reveal against a proof-of-possession signature (delete needs the same proof); `/api/identity/sign` refuses challenge-domain payloads. |
| **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). | | **User Data Layer** | Version-gated store for per-user signed (and, for private-tier keys, encrypted) records — My List, Continue Watching, ratings — indexed by user and CID. meta-core never inspects the crypto. See [User Data Layer](../../docs/project-architecture/user-data-layer.md). |
| **Report inbox** | `/api/reports` stores the bug reports / feedback the client apps' 🐛 buttons file (forwarded server-to-server by meta-watch and meta-read), in its own bounded `db/reports.db`; the dashboard's **Reports** page lists, reads and triages them. See [architecture: Report inbox](docs/meta-core-architecture.md#report-inbox). |
| **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. | | **Dashboard + editor** | nginx serves a React dashboard at `/` and the standalone metadata editor at `/editor/`. |
### Design characteristics ### Design characteristics
@@ -152,6 +153,8 @@ Environment variables (from `internal/config/config.go` unless noted):
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection (`internal/storage/client.go`). | | `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection (`internal/storage/client.go`). |
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. | | `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. | | `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
| `META_CORE_REPORTS_DB_PATH` | `<META_CORE_PATH>/db/reports.db` | Report inbox database. |
| `META_CORE_REPORTS_MAX` / `META_CORE_REPORTS_MAX_MB` | `5000` / `256` | Inbox bounds; closed reports are evicted first, then the oldest. |
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. | | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. | | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind. |
| `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). | | `ENABLE_FILE_WATCHER` | `true` | Disable to suppress the in-process watcher entirely (`POST /api/files/register` stays available). |
@@ -341,6 +344,16 @@ GET /api/udl/cid/{cid}/aggregate # meta-watch Aggregate wire shape
GET /api/udl/users/stats GET /api/udl/users/stats
``` ```
### Reports (bug-report / feedback inbox)
```
POST /api/reports # {app, note?, page?, route?, cid?, title?, uid?, build?, filedAt?, log?, …blocks} → 201 {id}
GET /api/reports # ?app=&status=open|triaged|closed&q=&before=<id>&limit= → {reports, apps, counts, more}
GET /api/reports/{id} # full report incl. body
PATCH /api/reports/{id} # {status?, triage_note?}
DELETE /api/reports/{id}
```
### Mounts ### Mounts
```bash ```bash
@@ -427,6 +440,7 @@ the repo-root [`METADATA_KEYS.md`](../../METADATA_KEYS.md).
| `internal/kv` | Storage engine: SQLite `kv` table, single writer + group commit, reader pool, in-transaction change events (`kv.go`), event streams as rows (`stream.go`). | | `internal/kv` | Storage engine: SQLite `kv` table, single writer + group commit, reader pool, in-transaction change events (`kv.go`), event streams as rows (`stream.go`). |
| `internal/storage` | Flat-key records (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, search over `kv_field_v` + FTS (`search.go`), derived counters (`onChange`), UDL store (`udl.go`, `udl_admin.go`), stats. | | `internal/storage` | Flat-key records (`client.go`), UUIDv7 minting, CID reverse index (`cid_resolution.go`), schema sentinel, search over `kv_field_v` + FTS (`search.go`), derived counters (`onChange`), UDL store (`udl.go`, `udl_admin.go`), stats. |
| `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). | | `internal/cid` | CID parsing + rank ladder for canonical-CID selection (`cid-rank-vectors.json`, guarded by the meta-root `scripts/check-cid-vectors.sh`). |
| `internal/reports` | Report inbox: its own SQLite file, normalisation/bounds, list/filter/paging, prune (closed first). |
| `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. | | `internal/identity` | Multi-account secp256k1 keystore, challenge/proof-of-possession, signature verification. |
| `internal/api` | HTTP server + all handlers + SSE event endpoints + search matchers + `/api/stats`. | | `internal/api` | HTTP server + all handlers + SSE event endpoints + search matchers + `/api/stats`. |
| `internal/schema` | Per-field schema, folded from counters the write path keeps exact (`idx:schema/…`). | | `internal/schema` | Per-field schema, folded from counters the write path keeps exact (`idx:schema/…`). |
+20
View File
@@ -14,6 +14,7 @@ import (
"github.com/metazla/meta-core/internal/config" "github.com/metazla/meta-core/internal/config"
"github.com/metazla/meta-core/internal/leader" "github.com/metazla/meta-core/internal/leader"
"github.com/metazla/meta-core/internal/meshdisco" "github.com/metazla/meta-core/internal/meshdisco"
"github.com/metazla/meta-core/internal/reports"
"github.com/metazla/meta-core/internal/storage" "github.com/metazla/meta-core/internal/storage"
) )
@@ -115,9 +116,22 @@ func main() {
log.Println("[meta-core] UDP discovery disabled by ENABLE_UDP_DISCOVERY") log.Println("[meta-core] UDP discovery disabled by ENABLE_UDP_DISCOVERY")
} }
// Report inbox: its own SQLite file beside truth. Not load-bearing — a
// failure to open it leaves /api/reports answering 503, never the core down.
reportStore, err := reports.Open(cfg.ReportsDBPath(), cfg.ReportsMax, int64(cfg.ReportsMaxMB)<<20)
if err != nil {
log.Printf("[meta-core] Warning: report inbox disabled: %v", err)
reportStore = nil
} else {
log.Printf("[meta-core] Report inbox %s (max %d reports, %d MiB)", reportStore.Path(), cfg.ReportsMax, cfg.ReportsMaxMB)
}
// Create and start API server // Create and start API server
apiServer := api.NewServer(cfg, leaderProvider, storageClient) apiServer := api.NewServer(cfg, leaderProvider, storageClient)
apiServer.SetMeshNode(mesh) apiServer.SetMeshNode(mesh)
if reportStore != nil {
apiServer.SetReports(reportStore)
}
if err := apiServer.Start(); err != nil { if err := apiServer.Start(); err != nil {
log.Fatalf("[meta-core] Failed to start API server: %v", err) log.Fatalf("[meta-core] Failed to start API server: %v", err)
} }
@@ -140,6 +154,12 @@ func main() {
} }
} }
if reportStore != nil {
if err := reportStore.Close(); err != nil {
log.Printf("[meta-core] Error closing report inbox: %v", err)
}
}
if err := storageClient.Close(); err != nil { if err := storageClient.Close(); err != nil {
log.Printf("[meta-core] Error closing storage client: %v", err) log.Printf("[meta-core] Error closing storage client: %v", err)
} }
+8
View File
@@ -4,6 +4,7 @@ import Mounts from './pages/Mounts';
import Watchers from './pages/Watchers'; import Watchers from './pages/Watchers';
import Files from './pages/Files'; import Files from './pages/Files';
import Identity from './pages/Identity'; import Identity from './pages/Identity';
import Reports from './pages/Reports';
import ServiceNav from './components/ServiceNav'; import ServiceNav from './components/ServiceNav';
const navStyle: React.CSSProperties = { const navStyle: React.CSSProperties = {
@@ -81,6 +82,12 @@ export default function App() {
> >
Identity Identity
</NavLink> </NavLink>
<NavLink
to="/reports"
style={({ isActive }) => (isActive ? activeLinkStyle : linkStyle)}
>
Reports
</NavLink>
<NavLink <NavLink
to="/editor" to="/editor"
style={({ isActive }) => (isActive ? activeLinkStyle : linkStyle)} style={({ isActive }) => (isActive ? activeLinkStyle : linkStyle)}
@@ -97,6 +104,7 @@ export default function App() {
<Route path="/watchers" element={<Watchers />} /> <Route path="/watchers" element={<Watchers />} />
<Route path="/files" element={<Files />} /> <Route path="/files" element={<Files />} />
<Route path="/identity" element={<Identity />} /> <Route path="/identity" element={<Identity />} />
<Route path="/reports" element={<Reports />} />
<Route path="/editor" element={<EditorPage />} /> <Route path="/editor" element={<EditorPage />} />
</Routes> </Routes>
</div> </div>
+271
View File
@@ -0,0 +1,271 @@
// How a report reads at a glance (the inbox row headline) and in full (the
// copy-as-text dump pasted into the Deck board).
//
// meta-core stores a report's body verbatim and knows nothing about it, so
// the meaning lives here, per app. The meta-watch headline is ported from the
// /info page that used to own reports (meta-watch ui/info.html,
// reportHeadline) — its comments explain why each chip earned its place; the
// short version is that every chip there is a stall or desync that once read
// as "healthy" until it was measured. Anything an app adds that this file does
// not know still renders: in the dump, and as a JSON section in the detail.
/* eslint-disable @typescript-eslint/no-explicit-any */
export type Body = Record<string, any>;
export interface ReportSummary {
id: number;
ts: number;
received: number;
app: string;
build?: string;
uid?: string;
page?: string;
route?: string;
cid?: string;
title?: string;
note?: string;
status: 'open' | 'triaged' | 'closed';
triage_note?: string;
updated?: number;
log_lines: number;
bytes: number;
}
export interface Report extends ReportSummary {
body: Body;
}
/** One headline segment; `strong` ones are what a human wrote or a failure. */
export interface Bit {
text: string;
strong?: boolean;
}
export function fmtDur(ms: number): string {
if (ms == null || !isFinite(ms)) return '?';
if (ms < 1000) return `${Math.round(ms)}ms`;
const s = ms / 1000;
if (s < 60) return `${s.toFixed(s < 10 ? 1 : 0)}s`;
const m = Math.floor(s / 60);
const r = Math.round(s % 60);
if (m < 60) return `${m}m${r ? ` ${r}s` : ''}`;
return `${Math.floor(m / 60)}h ${m % 60}m`;
}
export function shortCid(cid: string): string {
return cid && cid.length > 16 ? `${cid.slice(0, 8)}…${cid.slice(-6)}` : cid || '';
}
const clip = (s: string, n: number) => (s.length > n ? s.slice(0, n - 1) + '…' : s);
/** First line of the build blob, minus the app prefix: which build filed it. */
export function buildLabel(r: ReportSummary): string {
const first = (r.build || '').split('\n')[0];
return first.replace(/^meta-(watch|read|listen)( page| server)?\s+/, '').trim();
}
/** The device a report came from, in one line. */
export function deviceLine(d: any): string {
if (!d || typeof d !== 'object') return '';
const bits: string[] = [];
if (d.model) bits.push(d.model);
else if (d.platform) bits.push(d.platform);
if (d.isTv) bits.push(d.tvForced ? 'TV (forced)' : 'TV');
if (d.screen) bits.push(d.screen);
if (d.cores) bits.push(`${d.cores} cores`);
if (d.memoryGb) bits.push(`${d.memoryGb} GB`);
if (d.gpu) bits.push(d.gpu);
return bits.join(' · ');
}
/** What the row is about: a title, a page + route, or a cid. */
export function subject(r: ReportSummary, body?: Body): string {
const c = (body && body.context) || {};
if (r.page === 'search') return c.query ? `search “${c.query}”` : 'search';
if (r.title) return r.title;
if (r.page && r.page !== 'watch' && r.page !== 'read') return r.page;
if (r.cid) return shortCid(r.cid);
return r.page || '—';
}
// ── Headlines ─────────────────────────────────────────────────────────────
function failureBit(c: any): Bit | null {
const f = c && (c.failure || c.lastFailure);
if (!f) return null;
const who = c.lastFailure ? ` ${f.title || shortCid(f.cid || '')}` : '';
const tries = Array.isArray(f.attempts) && f.attempts.length > 1 ? ` · ${f.attempts.length} tries` : '';
return {
text: `FAILED${who}${f.elapsedMs != null ? ' after ' + fmtDur(f.elapsedMs) : ''}: ${clip(f.message || '', 90)}${tries}`,
strong: true,
};
}
function watchBits(r: ReportSummary, b: Body): string[] {
const bits: string[] = [];
const c = b.context || {};
if (r.page === 'search') {
if (c.state) bits.push(c.state === 'errored' ? 'STREAM ERRORED' : c.state);
if (c.anchored != null) {
bits.push(
`${c.anchored} anchored · ${c.raw || 0} raw` +
(c.shown != null && c.shown !== c.anchored ? ` · ${c.shown} after language filter` : '')
);
}
if (c.elapsedMs != null) bits.push(`stream ${fmtDur(c.elapsedMs)}`);
if (c.cacheSeeded) bits.push('painted from cache');
const ups = Array.isArray(c.upstreams) ? c.upstreams : [];
if (c.state !== 'idle' && c.state !== 'youtube-paste') {
if (ups.length) {
const recs = ups.reduce((n: number, u: any) => n + (u.records || 0), 0);
bits.push(`${ups.length} upstream${ups.length === 1 ? '' : 's'} (${recs} records)`);
} else bits.push('no upstream seen');
}
} else if (r.page === 'title') {
if (c.episodes != null) {
bits.push(`${c.episodes} release${c.episodes === 1 ? '' : 's'}` + (c.rawSources ? ` · ${c.rawSources} raw` : ''));
}
if (c.streamsOpen) bits.push(`${c.streamsOpen} stream${c.streamsOpen === 1 ? '' : 's'} open`);
else if (c.settled != null) bits.push(c.settled ? 'settled' : 'not settled');
if (c.placeholderCard) bits.push('placeholder hero');
}
if (c.lastPlayback) bits.push(`after playing ${c.lastPlayback.title || shortCid(c.lastPlayback.cid || '')}`);
const d = b.decode || {};
const p = b.pipeline || {};
if (d.underruns != null) bits.push(`${d.underruns} underrun${d.underruns === 1 ? '' : 's'}`);
if (d.minAhead != null) bits.push(`min ${(+d.minAhead).toFixed(1)}s ahead`);
if (d.decoded) {
const pct = (d.dropped / d.decoded) * 100;
bits.push(`${d.dropped} dropped (${pct.toFixed(pct >= 1 ? 1 : 2)}%)`);
}
if (d.capability) bits.push(d.capability);
if (p.pctDemuxCpu != null) bits.push(`demux CPU ${p.pctDemuxCpu}%`);
// Subtitle verdict: a stalled overlay is invisible in every other number.
const sub = b.subtitles;
if (sub && sub.up) {
const ago = sub.lastDrawAgoMs;
if (ago == null) bits.push('subs: NO DRAWS');
else if (ago > 4000) {
const why =
sub.paused === true ? ' while PAUSED'
: sub.advancing === true ? ' while playing'
: sub.paused === false ? ' (playing, not advancing)'
: '';
bits.push(`subs: STALLED ${fmtDur(ago)}${why}`);
} else bits.push(`subs ok (${sub.draws} draws)`);
if (sub.skipped > 0 && sub.draws > 0) {
const share = Math.round((100 * sub.skipped) / (sub.skipped + sub.draws));
if (share >= 15) bits.push(`subs: ${share}% of frames dropped`);
}
if (sub.commitMaxMs != null && sub.commitMaxMs > 250) {
bits.push(
`subs: commit ${sub.commitMs}/${sub.commitMaxMs}ms` +
(sub.pruned ? ` (${sub.lines}+${sub.pruned}p lines)` : ` (${sub.lines} lines)`)
);
}
if (sub.lastEventAgoMs != null && sub.lastEventAgoMs > 30000 && sub.advancing) {
bits.push(`subs: no packets for ${fmtDur(sub.lastEventAgoMs)}`);
}
if (sub.canvas && sub.canvas !== 'ok') bits.push(`subs canvas ${sub.canvas}`);
if (sub.videoW === 0 || sub.resizeDeferred) bits.push('subs: overlay resize owed (video has no size)');
else if (sub.resizesBlocked > 0) {
bits.push(`subs: ${sub.resizesBlocked} zero-size resize${sub.resizesBlocked > 1 ? 's' : ''} blocked`);
}
} else if (sub && sub.failed) {
const dd = sub.death;
if (!dd) bits.push('subs: fell back to WebVTT (build too old to say why)');
else {
bits.push(`subs: fell back to WebVTT at ${fmtDur(dd.mediaTime)} — ${dd.why}`);
if (dd.commitFailures) bits.push(`after ${dd.commitFailures} rejected commits`);
if (dd.strikes) bits.push(`after ${dd.strikes} stuck-loop strikes`);
bits.push(dd.draws ? `had drawn ${dd.draws} frames (${dd.lines}+${dd.pruned}p lines)` : 'never drew a frame');
if (dd.canvas && dd.canvas !== 'ok') bits.push(`canvas ${dd.canvas} at death`);
if (dd.resizesBlocked) bits.push(`${dd.resizesBlocked} zero-size resizes blocked`);
if (dd.videoW === 0) bits.push('video had no size at death');
}
}
// A/V sync: only the peak earns a chip.
const au = b.audio;
if (au && au.driftMaxMs != null && Math.abs(au.driftMaxMs) >= 40) {
const at = au.driftMaxAtSec != null ? ` @ ${fmtDur(au.driftMaxAtSec * 1000)}` : '';
bits.push(`A/V DESYNC ${au.driftMaxMs > 0 ? '+' : ''}${au.driftMaxMs}ms${at}`);
}
if (au && au.path === 'polyfill') bits.push(`audio ${au.codec || '?'} via polyfill`);
const rs = b.restart;
if (rs && rs.audioMinusVideoSec != null && rs.audioMinusVideoSec < -0.001) {
bits.push(`restart base from AUDIO (${rs.audioMinusVideoSec.toFixed(3)}s)`);
}
if (b.ttmvpMs != null) bits.push(`TTMVP ${fmtDur(b.ttmvpMs)}`);
if (b.capturedMsAgo != null && b.capturedMsAgo > 2000) bits.push(`measured ${fmtDur(b.capturedMsAgo)} before filing`);
return bits;
}
function readBits(r: ReportSummary, b: Body): string[] {
const bits: string[] = [];
const rd = b.reader;
if (rd && typeof rd === 'object') {
if (rd.page != null) bits.push(`page ${rd.page + 1}${rd.pages ? `/${rd.pages}` : ''}`);
if (rd.flow) bits.push(rd.flow);
if (rd.direction) bits.push(rd.direction);
if (rd.spread) bits.push('spread');
if (rd.strip && rd.strip.segments) bits.push(`strip ${rd.strip.active ?? '?'}/${rd.strip.segments}`);
if (rd.via) bits.push(`via ${rd.via}`);
if (rd.atEnd) bits.push('at end');
}
const c = b.context || {};
if (r.page === 'search') {
if (c.state) bits.push(c.state);
if (c.count != null) bits.push(`${c.count} shown`);
}
return bits;
}
/** The row headline: the note first (the only part a human wrote), then a
* failure if there was one, then the app's own measurements. */
export function headline(r: ReportSummary, body?: Body): Bit[] {
const out: Bit[] = [];
if (r.note) out.push({ text: `“${clip(r.note, 80)}”`, strong: true });
if (!body) return out;
const f = failureBit(body.context);
if (f) out.push(f);
const bits = r.app === 'meta-watch' ? watchBits(r, body) : r.app === 'meta-read' ? readBits(r, body) : [];
for (const t of bits) out.push({ text: t });
if (body.flags && typeof body.flags === 'object' && Object.keys(body.flags).length) {
out.push({ text: 'overrides: ' + Object.entries(body.flags).map(([k, v]) => `${k}=${v}`).join(' ') });
}
return out;
}
// ── Full text dump ────────────────────────────────────────────────────────
const LIFTED = new Set(['app', 'build', 'uid', 'page', 'route', 'cid', 'title', 'note', 'log']);
/** Every block of a report that is not one of the lifted fields or the log,
* in a stable order: context first, device second, then alphabetical. */
export function blocks(body: Body): [string, any][] {
const keys = Object.keys(body).filter((k) => !LIFTED.has(k));
const rank = (k: string) => (k === 'context' ? 0 : k === 'device' ? 1 : 2);
keys.sort((a, b) => rank(a) - rank(b) || a.localeCompare(b));
return keys.map((k) => [k, body[k]]);
}
/** The whole report as text, for pasting into a ticket. */
export function dump(r: Report): string {
const parts: string[] = [];
parts.push(`REPORT #${r.id} · ${r.app} · ${new Date(r.ts).toISOString()} · ${r.status}` +
(r.uid ? `\nUID ${r.uid}` : ''));
if (r.note) parts.push('NOTE\n' + r.note);
if (r.page || r.route) parts.push('PAGE ' + (r.page || '—') + (r.route ? '\nROUTE ' + r.route : ''));
if (r.cid || r.title) parts.push(`CONTENT ${r.title || ''}${r.cid ? ` (${r.cid})` : ''}`.trim());
if (r.build) parts.push(r.build);
for (const [k, v] of blocks(r.body || {})) {
parts.push(k.toUpperCase() + '\n' + (typeof v === 'string' ? v : JSON.stringify(v, null, 2)));
}
const log: string[] = Array.isArray(r.body?.log) ? r.body.log : [];
if (log.length) parts.push(`LOG (${log.length} lines)\n` + log.join('\n'));
if (r.triage_note) parts.push('TRIAGE\n' + r.triage_note);
return parts.join('\n\n');
}
+406
View File
@@ -0,0 +1,406 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { getJSON } from '../lib/api';
import {
blocks,
buildLabel,
deviceLine,
dump,
headline,
type Report,
type ReportSummary,
shortCid,
subject,
} from '../lib/reportText';
// The bug-report / feedback inbox. Client apps (meta-watch, meta-read, …)
// forward what their 🐛 buttons file to POST /api/reports; this page lists,
// reads and triages them. See internal/reports for storage and bounds.
type Status = 'open' | 'triaged' | 'closed';
const STATUSES: Status[] = ['open', 'triaged', 'closed'];
interface ListPage {
reports: ReportSummary[];
apps: Record<string, number>;
counts: Record<string, number>;
more: boolean;
}
const card: React.CSSProperties = {
background: '#16213e',
borderRadius: '8px',
padding: '1rem',
display: 'flex',
flexDirection: 'column',
overflow: 'hidden',
minHeight: 0,
};
const chip = (active: boolean): React.CSSProperties => ({
padding: '0.25rem 0.7rem',
borderRadius: '999px',
border: '1px solid #0f3460',
background: active ? '#0f3460' : 'transparent',
color: active ? '#fff' : '#aaa',
cursor: 'pointer',
fontSize: '0.85rem',
});
const button: React.CSSProperties = {
padding: '0.35rem 0.8rem',
borderRadius: '4px',
border: '1px solid #0f3460',
background: '#0f3460',
color: '#fff',
cursor: 'pointer',
fontSize: '0.85rem',
};
const mono: React.CSSProperties = { fontFamily: 'ui-monospace, Menlo, Consolas, monospace', fontSize: '0.8rem' };
const statusColor: Record<Status, string> = { open: '#b45309', triaged: '#1d4ed8', closed: '#374151' };
function StatusBadge({ s }: { s: Status }) {
return (
<span style={{ background: statusColor[s], color: '#fff', borderRadius: '3px', padding: '0.05rem 0.4rem', fontSize: '0.72rem' }}>
{s}
</span>
);
}
const fmtTime = (ms: number) => new Date(ms).toLocaleString();
export default function Reports() {
const [app, setApp] = useState('');
const [status, setStatus] = useState<Status | ''>('open');
const [q, setQ] = useState('');
const [page, setPage] = useState<ListPage | null>(null);
const [older, setOlder] = useState<ReportSummary[]>([]);
const [more, setMore] = useState(false);
const [error, setError] = useState<string | null>(null);
const [selected, setSelected] = useState<number | null>(null);
const query = useCallback(
(before?: number) => {
const p = new URLSearchParams();
if (app) p.set('app', app);
if (status) p.set('status', status);
if (q.trim()) p.set('q', q.trim());
if (before) p.set('before', String(before));
return `/api/reports?${p}`;
},
[app, status, q]
);
// The newest page is polled; pages loaded with "older" are kept beneath it
// until a filter changes, and then they own the "more" flag.
const olderCount = useRef(0);
olderCount.current = older.length;
const refresh = useCallback(async () => {
try {
const data = await getJSON<ListPage>(query());
setPage(data);
if (olderCount.current === 0) setMore(data.more);
setError(null);
} catch (e) {
setError(String((e as Error).message || e));
}
}, [query]);
useEffect(() => {
setOlder([]);
setMore(false);
}, [app, status, q]);
useEffect(() => {
refresh();
const t = setInterval(refresh, 10000);
return () => clearInterval(t);
}, [refresh]);
const loadOlder = async () => {
const all = [...(page?.reports || []), ...older];
const last = all[all.length - 1];
if (!last) return;
try {
const data = await getJSON<ListPage>(query(last.id));
setOlder((prev) => [...prev, ...data.reports]);
setMore(data.more);
} catch (e) {
setError(String((e as Error).message || e));
}
};
const rows = [...(page?.reports || []), ...older.filter((o) => !(page?.reports || []).some((r) => r.id === o.id))];
const apps = Object.entries(page?.apps || {}).sort((a, b) => a[0].localeCompare(b[0]));
const total = Object.values(page?.counts || {}).reduce((n, c) => n + c, 0);
return (
<div style={{ padding: '1rem', height: '100%', display: 'flex', flexDirection: 'column', overflow: 'hidden', boxSizing: 'border-box' }}>
<div style={{ display: 'flex', alignItems: 'baseline', gap: '1rem', flexShrink: 0, flexWrap: 'wrap' }}>
<h1 style={{ margin: '0 0 0.75rem 0' }}>Reports</h1>
<span style={{ color: '#888', fontSize: '0.9rem' }}>bug reports and feedback filed from the apps' 🐛 buttons</span>
</div>
<div style={{ display: 'flex', gap: '0.5rem', flexWrap: 'wrap', alignItems: 'center', marginBottom: '0.75rem', flexShrink: 0 }}>
<span style={chip(status === '')} onClick={() => setStatus('')}>all {total}</span>
{STATUSES.map((s) => (
<span key={s} style={chip(status === s)} onClick={() => setStatus(s)}>
{s} {page?.counts?.[s] ?? 0}
</span>
))}
<span style={{ width: '1rem' }} />
<span style={chip(app === '')} onClick={() => setApp('')}>every app</span>
{apps.map(([a, n]) => (
<span key={a} style={chip(app === a)} onClick={() => setApp(a)}>
{a} {n}
</span>
))}
<input
value={q}
onChange={(e) => setQ(e.target.value)}
placeholder="search note, title, cid, uid"
style={{ marginLeft: 'auto', padding: '0.35rem 0.6rem', borderRadius: '4px', border: '1px solid #0f3460', background: '#0b1630', color: '#eee', minWidth: '16rem' }}
/>
</div>
{error && <div style={{ color: '#f87171', marginBottom: '0.5rem', flexShrink: 0 }}>Could not load reports: {error}</div>}
<div style={{ display: 'grid', gridTemplateColumns: 'minmax(22rem, 2fr) 3fr', gap: '1rem', flex: 1, minHeight: 0 }}>
<div style={card}>
{rows.length === 0 ? (
<p style={{ color: '#888', margin: 0 }}>{page ? 'No reports match.' : 'Loading…'}</p>
) : (
<div style={{ flex: 1, overflow: 'auto', minHeight: 0 }}>
{rows.map((r) => (
<div
key={r.id}
onClick={() => setSelected(r.id)}
style={{
padding: '0.55rem 0.5rem',
borderBottom: '1px solid #0f3460',
cursor: 'pointer',
background: selected === r.id ? '#0f3460' : 'transparent',
}}
>
<div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', fontSize: '0.8rem', color: '#888' }}>
<span style={mono}>#{r.id}</span>
<StatusBadge s={r.status} />
<span>{r.app}</span>
<span style={{ marginLeft: 'auto' }}>{fmtTime(r.ts)}</span>
</div>
<div style={{ margin: '0.2rem 0', color: '#eee' }}>{subject(r)}</div>
<div style={{ fontSize: '0.85rem', color: r.note ? '#ddd' : '#666' }}>
{r.note ? `“${r.note.length > 120 ? r.note.slice(0, 119) + '…' : r.note}”` : 'no note'}
</div>
<div style={{ ...mono, color: '#666', marginTop: '0.15rem' }}>
{buildLabel(r) || 'build unknown'}
{r.uid ? ` · ${shortCid(r.uid)}` : ''}
{r.log_lines ? ` · ${r.log_lines} log lines` : ''}
</div>
</div>
))}
{more && (
<div style={{ padding: '0.75rem', textAlign: 'center' }}>
<button style={button} onClick={loadOlder}>Load older</button>
</div>
)}
</div>
)}
</div>
<div style={card}>
{selected == null ? (
<p style={{ color: '#888', margin: 0 }}>Pick a report to read it.</p>
) : (
<ReportDetail
key={selected}
id={selected}
onChanged={refresh}
onDeleted={() => {
setSelected(null);
setOlder((prev) => prev.filter((r) => r.id !== selected));
refresh();
}}
/>
)}
</div>
</div>
</div>
);
}
function ReportDetail({ id, onChanged, onDeleted }: { id: number; onChanged: () => void; onDeleted: () => void }) {
const [r, setR] = useState<Report | null>(null);
const [error, setError] = useState<string | null>(null);
const [note, setNote] = useState('');
const [busy, setBusy] = useState(false);
const [copied, setCopied] = useState(false);
const copyTimer = useRef<number>();
useEffect(() => {
getJSON<Report>(`/api/reports/${id}`)
.then((rep) => {
setR(rep);
setNote(rep.triage_note || '');
})
.catch((e) => setError(String(e.message || e)));
return () => window.clearTimeout(copyTimer.current);
}, [id]);
const patch = async (change: { status?: Status; triage_note?: string }) => {
setBusy(true);
try {
const rep = await getJSON<Report>(`/api/reports/${id}`, {
method: 'PATCH',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(change),
});
setR(rep);
setNote(rep.triage_note || '');
setError(null);
onChanged();
} catch (e) {
setError(String((e as Error).message || e));
} finally {
setBusy(false);
}
};
const remove = async () => {
if (!window.confirm(`Delete report #${id}? This cannot be undone.`)) return;
setBusy(true);
try {
const res = await fetch(`/api/reports/${id}`, { method: 'DELETE' });
if (!res.ok && res.status !== 404) throw new Error(`/api/reports/${id} → HTTP ${res.status}`);
onDeleted();
} catch (e) {
setError(String((e as Error).message || e));
setBusy(false);
}
};
const copy = async () => {
if (!r) return;
try {
await navigator.clipboard.writeText(dump(r));
setCopied(true);
copyTimer.current = window.setTimeout(() => setCopied(false), 1800);
} catch {
setError('Clipboard unavailable here — select the text below and copy it by hand.');
}
};
if (error && !r) return <p style={{ color: '#f87171', margin: 0 }}>Could not load report #{id}: {error}</p>;
if (!r) return <p style={{ color: '#888', margin: 0 }}>Loading #{id}…</p>;
const body = r.body || {};
const bits = headline(r, body);
const device = deviceLine(body.device);
const log: string[] = Array.isArray(body.log) ? body.log : [];
return (
<div style={{ flex: 1, overflow: 'auto', minHeight: 0 }}>
<div style={{ display: 'flex', gap: '0.5rem', alignItems: 'center', flexWrap: 'wrap' }}>
<h2 style={{ margin: 0, fontSize: '1.15rem' }}>#{r.id} · {subject(r, body)}</h2>
<StatusBadge s={r.status} />
<span style={{ marginLeft: 'auto', display: 'flex', gap: '0.4rem' }}>
{STATUSES.filter((s) => s !== r.status).map((s) => (
<button key={s} style={button} disabled={busy} onClick={() => patch({ status: s })}>
Mark {s}
</button>
))}
<button style={button} onClick={copy}>{copied ? 'Copied' : 'Copy as text'}</button>
<button style={{ ...button, background: '#7f1d1d', borderColor: '#7f1d1d' }} disabled={busy} onClick={remove}>
Delete
</button>
</span>
</div>
{error && <div style={{ color: '#f87171', marginTop: '0.5rem' }}>{error}</div>}
<div style={{ color: '#888', fontSize: '0.85rem', margin: '0.4rem 0 0.75rem' }}>
{r.app} · filed {fmtTime(r.ts)}
{r.received - r.ts > 60000 ? ` · delivered ${fmtTime(r.received)}` : ''}
{r.uid ? ` · uid ${r.uid}` : ''}
</div>
{bits.length > 0 && (
<div style={{ marginBottom: '0.75rem', lineHeight: 1.6 }}>
{bits.map((b, i) => (
<span key={i}>
{i > 0 && <span style={{ color: '#555' }}> · </span>}
{b.strong ? <b style={{ color: '#fff' }}>{b.text}</b> : <span style={{ color: '#ccc' }}>{b.text}</span>}
</span>
))}
</div>
)}
<Field label="Page">
{r.page || '—'}
{r.route && <span style={{ ...mono, color: '#888' }}> {r.route}</span>}
</Field>
{(r.title || r.cid) && (
<Field label="Content">
{r.title}
{r.cid && <span style={{ ...mono, color: '#888' }}> {r.cid}</span>}
</Field>
)}
{device && <Field label="Device">{device}</Field>}
{r.build && (
<Field label="Build">
<pre style={{ ...mono, margin: 0, whiteSpace: 'pre-wrap', color: '#ccc' }}>{r.build}</pre>
</Field>
)}
<Field label="Triage">
<div style={{ display: 'flex', gap: '0.5rem' }}>
<input
value={note}
onChange={(e) => setNote(e.target.value)}
placeholder="e.g. Deck #812, duplicate of #40, fixed in 1.0.95"
style={{ flex: 1, padding: '0.35rem 0.6rem', borderRadius: '4px', border: '1px solid #0f3460', background: '#0b1630', color: '#eee' }}
/>
<button style={button} disabled={busy || note === (r.triage_note || '')} onClick={() => patch({ triage_note: note })}>
Save
</button>
</div>
</Field>
{blocks(body)
.filter(([, v]) => v === null || typeof v !== 'object')
.map(([k, v]) => (
<Field key={k} label={k}>
<span style={mono}>{String(v)}</span>
</Field>
))}
{blocks(body).filter(([, v]) => v !== null && typeof v === 'object').map(([k, v]) => (
<details key={k} open={k === 'context' || k === 'reader'} style={{ marginTop: '0.6rem' }}>
<summary style={{ cursor: 'pointer', color: '#9ab' }}>{k}</summary>
<pre style={{ ...mono, background: '#0b1630', padding: '0.6rem', borderRadius: '4px', whiteSpace: 'pre-wrap', wordBreak: 'break-word', margin: '0.3rem 0 0' }}>
{typeof v === 'string' ? v : JSON.stringify(v, null, 2)}
</pre>
</details>
))}
{log.length > 0 && (
<details open style={{ marginTop: '0.6rem' }}>
<summary style={{ cursor: 'pointer', color: '#9ab' }}>log ({log.length} lines)</summary>
<pre style={{ ...mono, background: '#0b1630', padding: '0.6rem', borderRadius: '4px', whiteSpace: 'pre-wrap', wordBreak: 'break-word', margin: '0.3rem 0 0', maxHeight: '28rem', overflow: 'auto' }}>
{log.join('\n')}
</pre>
</details>
)}
</div>
);
}
function Field({ label, children }: { label: string; children: React.ReactNode }) {
return (
<div style={{ display: 'grid', gridTemplateColumns: '5.5rem 1fr', gap: '0.5rem', margin: '0.3rem 0', alignItems: 'baseline' }}>
<span style={{ color: '#888', fontSize: '0.85rem' }}>{label}</span>
<div>{children}</div>
</div>
);
}
+40
View File
@@ -485,10 +485,46 @@ Watchers
were removed with the 2026-10 SQLite switch; they had been run once on were removed with the 2026-10 SQLite switch; they had been run once on
watch.nsl.sh beforehand.) watch.nsl.sh beforehand.)
Report inbox (internal/reports, internal/api/reports.go) — bug reports and
feedback the client apps' 🐛 buttons file; read on the dashboard's Reports page
POST /api/reports {app, note?, page?, route?, cid?, title?, uid?,
build?, filedAt?, log?, <any block>} → 201 {id}
GET /api/reports ?app=&status=open|triaged|closed&q=&before=<id>&limit=
→ {reports: [summary…], apps, counts, more}
GET /api/reports/{id} full report incl. body
PATCH /api/reports/{id} {status?, triage_note?}
DELETE /api/reports/{id}
WebDAV WebDAV
/webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND) /webdav/... (GET, PUT, DELETE, MKCOL, COPY, MOVE, PROPFIND)
``` ```
### Report inbox
Client apps (meta-watch, meta-read) build a report in the browser and their
server forwards it **server-to-server to the box's own core** — the uid is
stamped by the app from its bearer token, never taken from the body. Apps that
cannot reach the core spool the report locally and retry
(`report_outbox.rs` in each app), so `filedAt` (when it was filed) and
`received` (when meta-core stored it) can differ; the row's `ts` is `filedAt`
when it is plausible (≤ 30 days old, ≤ 5 min in the future).
- **Own file, `db/reports.db`.** Reports are diagnostics, not metadata: neither
the truth DB (signed claims only) nor the cache (thrown away on recovery).
`POST /api/metadata/clear` and a cache rebuild never touch it. A failure to
open it disables `/api/reports*` (503), never the core.
- **meta-core does not interpret a report.** `app`, `build`, `uid`, `page`,
`route`, `cid`, `title`, `note` are lifted into columns for filtering; every
other top-level key is stored verbatim in `body` and rendered generically
(app-specific headlines live in `dashboard/src/lib/reportText.ts`). Bounds:
512 KiB per POST, note 1000 chars, log tail 200 × 600 chars, any other block
over 32 KiB replaced by `{truncated, bytes}`.
- **Bounded.** `META_CORE_REPORTS_MAX` reports and `META_CORE_REPORTS_MAX_MB`
of bodies; on overflow closed reports are evicted first, then the oldest —
an open report is the last to go. Enforced on every insert and hourly.
- **Auth:** perimeter-gated like the rest of `/api`; do not add it to the
hash-lock bypass. A report is viewing/reading history (cids, titles, a uid).
No `/metrics` endpoint is exposed. Observability is via container logs + No `/metrics` endpoint is exposed. Observability is via container logs +
the `/health` and `/status` JSON. the `/health` and `/status` JSON.
@@ -738,6 +774,9 @@ via `/health`; metadata I/O and SSE event streams stop.
| `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. | | `META_CORE_DB_CACHE_MB` | `64` | Page cache of the writer connection. |
| `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. | | `META_CORE_DB_READER_CACHE_MB` | `16` | Page cache per reader connection. |
| `META_CORE_DB_READERS` | `8` | Reader connection pool size. | | `META_CORE_DB_READERS` | `8` | Reader connection pool size. |
| `META_CORE_REPORTS_DB_PATH` | `<META_CORE_PATH>/db/reports.db` | Report inbox database (beside truth, so it is backed up with it). |
| `META_CORE_REPORTS_MAX` | `5000` | Reports kept; closed ones are evicted first, then the oldest. |
| `META_CORE_REPORTS_MAX_MB` | `256` | Total report body size kept, MiB. |
| `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. | | `META_CORE_HTTP_PORT` | `9000` | Go HTTP+SSE+WebDAV port. |
| `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. | | `META_CORE_HTTP_HOST` | `0.0.0.0` | HTTP bind address. |
| `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. | | `HEALTH_CHECK_INTERVAL_MS` | `5000` | Internal health-loop cadence. |
@@ -758,6 +797,7 @@ via `/health`; metadata I/O and SSE event streams stop.
| `locks/kv-leader.info` | Leader API URL (plaintext). | | `locks/kv-leader.info` | Leader API URL (plaintext). |
| `db/meta-core.db` (+ `-wal`, `-shm`) | Truth database (claims, principals, `cid_group`). Backed up. Replaced `db/redis/` in 2026-10; holds the claims schema since v2.0.0. | | `db/meta-core.db` (+ `-wal`, `-shm`) | Truth database (claims, principals, `cid_group`). Backed up. Replaced `db/redis/` in 2026-10; holds the claims schema since v2.0.0. |
| `db-cache/cache.db` (+ `-wal`, `-shm`) | Cache database (v1-layout `kv` table: merged view, lookup, counters, events, service scratch keys; field index, FTS, `claim_by_pid`, `local_file`). Never backed up. | | `db-cache/cache.db` (+ `-wal`, `-shm`) | Cache database (v1-layout `kv` table: merged view, lookup, counters, events, service scratch keys; field index, FTS, `claim_by_pid`, `local_file`). Never backed up. |
| `db/reports.db` (+ `-wal`, `-shm`) | Report inbox (bug reports / feedback from the client apps). Diagnostics, not metadata; bounded. |
| `services/*.json` | Service registry. | | `services/*.json` | Service registry. |
| `mounts/mounts.json` | rclone mount configuration. | | `mounts/mounts.json` | rclone mount configuration. |
| `mounts/errors/{id}.error` | Mount error logs. | | `mounts/errors/{id}.error` | Mount error logs. |
+204
View File
@@ -0,0 +1,204 @@
package api
// Report inbox (/api/reports): bug reports and feedback filed from the client
// apps. An app's browser builds the report, the app's server forwards it here
// server-to-server, and the dashboard's Reports page reads it back. Storage
// and bounds live in internal/reports.
//
// Auth: like the rest of /api, gated at the perimeter — do NOT add to the
// nginx-hash-lock unauth bypass. Apps reach meta-core on the internal network.
import (
"encoding/json"
"errors"
"log"
"net/http"
"strconv"
"strings"
"time"
"github.com/gorilla/mux"
"github.com/metazla/meta-core/internal/reports"
)
// maxReportBody bounds one POST. A report is at most a few blocks of 32 KiB
// plus a 200-line log tail; anything larger is not a report.
const maxReportBody = 512 << 10
// SetReports attaches the report inbox. Separate from NewServer so its
// signature (and its tests) stay untouched; nil leaves the routes answering
// 503.
func (s *Server) SetReports(r *reports.Store) {
s.reports = r
}
func (s *Server) setupReportRoutes() {
s.router.HandleFunc("/api/reports", s.handlePostReport).Methods("POST")
s.router.HandleFunc("/api/reports", s.handleListReports).Methods("GET")
s.router.HandleFunc("/api/reports/{id:[0-9]+}", s.handleGetReport).Methods("GET")
s.router.HandleFunc("/api/reports/{id:[0-9]+}", s.handlePatchReport).Methods("PATCH")
s.router.HandleFunc("/api/reports/{id:[0-9]+}", s.handleDeleteReport).Methods("DELETE")
}
func (s *Server) reportsReady(w http.ResponseWriter) bool {
if s.reports == nil {
writeError(w, http.StatusServiceUnavailable, "report inbox not available")
return false
}
return true
}
func reportID(r *http.Request) int64 {
id, _ := strconv.ParseInt(mux.Vars(r)["id"], 10, 64)
return id
}
// handlePostReport handles POST /api/reports → 201 {id}.
func (s *Server) handlePostReport(w http.ResponseWriter, r *http.Request) {
if !s.reportsReady(w) {
return
}
var in map[string]json.RawMessage
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxReportBody)).Decode(&in); err != nil {
var tooBig *http.MaxBytesError
if errors.As(err, &tooBig) {
writeError(w, http.StatusRequestEntityTooLarge, "report larger than 512 KiB")
return
}
writeError(w, http.StatusBadRequest, "body must be a JSON object")
return
}
rep, err := reports.Normalize(in, time.Now())
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
id, err := s.reports.Insert(rep)
if err != nil {
log.Printf("[reports] insert failed: %v", err)
writeError(w, http.StatusInternalServerError, "could not store the report")
return
}
log.Printf("[reports] #%d filed by %s (%s)", id, rep.App, firstLine(rep.Note))
writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
}
// handleListReports handles GET /api/reports?app=&status=&q=&before=&limit=.
func (s *Server) handleListReports(w http.ResponseWriter, r *http.Request) {
if !s.reportsReady(w) {
return
}
q := r.URL.Query()
f := reports.Filter{App: q.Get("app"), Status: q.Get("status"), Q: q.Get("q")}
f.Before, _ = strconv.ParseInt(q.Get("before"), 10, 64)
f.Limit, _ = strconv.Atoi(q.Get("limit"))
if f.Status != "" && !reports.ValidStatus(f.Status) {
writeError(w, http.StatusBadRequest, "status must be open, triaged or closed")
return
}
page, err := s.reports.List(f)
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, page)
}
// handleGetReport handles GET /api/reports/{id}.
func (s *Server) handleGetReport(w http.ResponseWriter, r *http.Request) {
if !s.reportsReady(w) {
return
}
rep, err := s.reports.Get(reportID(r))
if errors.Is(err, reports.ErrNotFound) {
writeError(w, http.StatusNotFound, "no such report")
return
}
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, rep)
}
// handlePatchReport handles PATCH /api/reports/{id} {status?, triage_note?}.
func (s *Server) handlePatchReport(w http.ResponseWriter, r *http.Request) {
if !s.reportsReady(w) {
return
}
var in struct {
Status *string `json:"status"`
TriageNote *string `json:"triage_note"`
}
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10)).Decode(&in); err != nil {
writeError(w, http.StatusBadRequest, "body must be a JSON object")
return
}
if in.Status == nil && in.TriageNote == nil {
writeError(w, http.StatusBadRequest, "nothing to change: send status and/or triage_note")
return
}
if in.Status != nil && !reports.ValidStatus(*in.Status) {
writeError(w, http.StatusBadRequest, "status must be open, triaged or closed")
return
}
rep, err := s.reports.Update(reportID(r), in.Status, in.TriageNote)
if errors.Is(err, reports.ErrNotFound) {
writeError(w, http.StatusNotFound, "no such report")
return
}
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
writeJSON(w, http.StatusOK, rep)
}
// handleDeleteReport handles DELETE /api/reports/{id}.
func (s *Server) handleDeleteReport(w http.ResponseWriter, r *http.Request) {
if !s.reportsReady(w) {
return
}
err := s.reports.Delete(reportID(r))
if errors.Is(err, reports.ErrNotFound) {
writeError(w, http.StatusNotFound, "no such report")
return
}
if err != nil {
writeError(w, http.StatusInternalServerError, err.Error())
return
}
w.WriteHeader(http.StatusNoContent)
}
// pruneReports re-applies the inbox bounds hourly. Insert already prunes, so
// this only matters when a bound was lowered between restarts.
func (s *Server) pruneReports() {
run := func() {
if n, err := s.reports.Prune(); err != nil {
log.Printf("[reports] prune failed: %v", err)
} else if n > 0 {
log.Printf("[reports] pruned %d report(s) over the inbox bounds", n)
}
}
run()
ticker := time.NewTicker(time.Hour)
defer ticker.Stop()
for range ticker.C {
run()
}
}
func firstLine(s string) string {
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
if r := []rune(s); len(r) > 80 {
s = string(r[:80]) + "…"
}
if s == "" {
return "no note"
}
return strconv.Quote(s)
}
+121
View File
@@ -0,0 +1,121 @@
package api
import (
"bytes"
"net/http"
"net/http/httptest"
"path/filepath"
"strconv"
"strings"
"testing"
"github.com/gorilla/mux"
"github.com/metazla/meta-core/internal/reports"
)
func newReportServer(t *testing.T) *Server {
t.Helper()
store, err := reports.Open(filepath.Join(t.TempDir(), "reports.db"), 0, 0)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = store.Close() })
s := &Server{router: mux.NewRouter(), reports: store}
s.setupReportRoutes()
return s
}
func TestReportsRoundTrip(t *testing.T) {
s := newReportServer(t)
rec, out := do(t, s, "POST", "/api/reports", map[string]any{
"app": "meta-read", "note": "blank page", "uid": "u1",
"reader": map[string]any{"page": 3}, "log": []string{"a", "b"},
})
if rec.Code != http.StatusCreated || out["id"] == nil {
t.Fatalf("post: %d %s", rec.Code, rec.Body)
}
id := int64(out["id"].(float64))
rec, out = do(t, s, "GET", "/api/reports", nil)
list, _ := out["reports"].([]any)
if rec.Code != 200 || len(list) != 1 {
t.Fatalf("list: %d %s", rec.Code, rec.Body)
}
first := list[0].(map[string]any)
if _, hasBody := first["body"]; hasBody || first["log_lines"].(float64) != 2 {
t.Fatalf("summary should carry log_lines, not the body: %v", first)
}
path := "/api/reports/" + itoa(id)
rec, out = do(t, s, "GET", path, nil)
body, _ := out["body"].(map[string]any)
if rec.Code != 200 || body["reader"] == nil || out["uid"] != "u1" {
t.Fatalf("get: %d %s", rec.Code, rec.Body)
}
rec, out = do(t, s, "PATCH", path, map[string]any{"status": "triaged", "triage_note": "Deck #1"})
if rec.Code != 200 || out["status"] != "triaged" || out["triage_note"] != "Deck #1" {
t.Fatalf("patch: %d %s", rec.Code, rec.Body)
}
rec, _ = do(t, s, "GET", "/api/reports?status=triaged&app=meta-read", nil)
if rec.Code != 200 || !strings.Contains(rec.Body.String(), `"triaged":1`) {
t.Fatalf("filtered list: %d %s", rec.Code, rec.Body)
}
if rec, _ = do(t, s, "DELETE", path, nil); rec.Code != http.StatusNoContent {
t.Fatalf("delete: %d", rec.Code)
}
if rec, _ = do(t, s, "GET", path, nil); rec.Code != http.StatusNotFound {
t.Fatalf("get after delete: %d", rec.Code)
}
}
func TestReportsRejects(t *testing.T) {
s := newReportServer(t)
cases := []struct {
name, method, path string
body any
want int
}{
{"no app", "POST", "/api/reports", map[string]any{"note": "x"}, 400},
{"bad status filter", "GET", "/api/reports?status=done", nil, 400},
{"patch nothing", "PATCH", "/api/reports/1", map[string]any{}, 400},
{"patch bad status", "PATCH", "/api/reports/1", map[string]any{"status": "done"}, 400},
{"patch missing", "PATCH", "/api/reports/1", map[string]any{"status": "closed"}, 404},
{"delete missing", "DELETE", "/api/reports/1", nil, 404},
}
for _, c := range cases {
if rec, _ := do(t, s, c.method, c.path, c.body); rec.Code != c.want {
t.Errorf("%s: got %d want %d (%s)", c.name, rec.Code, c.want, rec.Body)
}
}
// Not a JSON object.
req := httptest.NewRequest("POST", "/api/reports", strings.NewReader(`[1,2]`))
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != 400 {
t.Errorf("array body: %d", rec.Code)
}
// Over the body cap.
big := bytes.Repeat([]byte("x"), maxReportBody+1)
req = httptest.NewRequest("POST", "/api/reports", strings.NewReader(`{"app":"a","note":"`+string(big)+`"}`))
rec = httptest.NewRecorder()
s.router.ServeHTTP(rec, req)
if rec.Code != http.StatusRequestEntityTooLarge {
t.Errorf("oversized body: %d", rec.Code)
}
}
func TestReportsUnavailableWithoutStore(t *testing.T) {
s := &Server{router: mux.NewRouter()}
s.setupReportRoutes()
if rec, _ := do(t, s, "GET", "/api/reports", nil); rec.Code != http.StatusServiceUnavailable {
t.Fatalf("no store: %d", rec.Code)
}
}
func itoa(n int64) string { return strconv.FormatInt(n, 10) }
+11 -1
View File
@@ -13,6 +13,7 @@ import (
"github.com/metazla/meta-core/internal/leader" "github.com/metazla/meta-core/internal/leader"
"github.com/metazla/meta-core/internal/meshdisco" "github.com/metazla/meta-core/internal/meshdisco"
"github.com/metazla/meta-core/internal/mounts" "github.com/metazla/meta-core/internal/mounts"
"github.com/metazla/meta-core/internal/reports"
"github.com/metazla/meta-core/internal/schema" "github.com/metazla/meta-core/internal/schema"
"github.com/metazla/meta-core/internal/storage" "github.com/metazla/meta-core/internal/storage"
"github.com/metazla/meta-core/internal/watcher" "github.com/metazla/meta-core/internal/watcher"
@@ -47,6 +48,8 @@ type Server struct {
mesh *meshdisco.Node mesh *meshdisco.Node
router *mux.Router router *mux.Router
server *http.Server server *http.Server
// reports is the bug-report inbox (reports.go); nil = routes answer 503.
reports *reports.Store
// stats caches the keyspace-walk half of GET /api/stats so a polling // stats caches the keyspace-walk half of GET /api/stats so a polling
// dashboard never puts a multi-second store walk on the request path. // dashboard never puts a multi-second store walk on the request path.
stats *statsCache stats *statsCache
@@ -279,6 +282,10 @@ func (s *Server) setupRoutes() {
s.router.HandleFunc("/neighbors", s.handleListNeighbors).Methods("GET") s.router.HandleFunc("/neighbors", s.handleListNeighbors).Methods("GET")
s.router.HandleFunc("/api/neighbors", s.handleListNeighbors).Methods("GET") s.router.HandleFunc("/api/neighbors", s.handleListNeighbors).Methods("GET")
// Report inbox: bug reports / feedback forwarded by the client apps,
// read by the dashboard's Reports page. Perimeter-gated (reports.go).
s.setupReportRoutes()
// Mount management routes (if manager initialized) // Mount management routes (if manager initialized)
if s.mountsHandlers != nil { if s.mountsHandlers != nil {
s.mountsHandlers.RegisterRoutes(s.router) s.mountsHandlers.RegisterRoutes(s.router)
@@ -363,6 +370,9 @@ func (s *Server) Start() error {
}() }()
go s.logWriteStats() go s.logWriteStats()
} }
if s.reports != nil {
go s.pruneReports()
}
return nil return nil
} }
@@ -468,7 +478,7 @@ func loggingMiddleware(next http.Handler) http.Handler {
func corsMiddleware(next http.Handler) http.Handler { func corsMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*") w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS") w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, PATCH, DELETE, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type") w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
if r.Method == "OPTIONS" { if r.Method == "OPTIONS" {
+18
View File
@@ -23,6 +23,11 @@ type Config struct {
DBPathOverride string // META_CORE_DB_PATH; empty = <MetaCorePath>/db/meta-core.db (truth) DBPathOverride string // META_CORE_DB_PATH; empty = <MetaCorePath>/db/meta-core.db (truth)
CachePathOverride string // META_CORE_CACHE_PATH; empty = <MetaCorePath>/db-cache/cache.db CachePathOverride string // META_CORE_CACHE_PATH; empty = <MetaCorePath>/db-cache/cache.db
// Report inbox (internal/reports): its own SQLite file, bounded.
ReportsPathOverride string // META_CORE_REPORTS_DB_PATH; empty = <MetaCorePath>/db/reports.db
ReportsMax int // META_CORE_REPORTS_MAX: reports kept (default 5000; closed evicted first)
ReportsMaxMB int // META_CORE_REPORTS_MAX_MB: total body size kept, MiB (default 256)
// HTTP API configuration // HTTP API configuration
HTTPPort int // HTTP API port (default: 9000) HTTPPort int // HTTP API port (default: 9000)
HTTPHost string // HTTP API host (default: "0.0.0.0" — meta-core always runs in a container and must answer siblings) HTTPHost string // HTTP API host (default: "0.0.0.0" — meta-core always runs in a container and must answer siblings)
@@ -63,6 +68,9 @@ func Load() *Config {
PublicURL: getEnv("META_CORE_PUBLIC_URL", ""), PublicURL: getEnv("META_CORE_PUBLIC_URL", ""),
DBPathOverride: getEnv("META_CORE_DB_PATH", ""), DBPathOverride: getEnv("META_CORE_DB_PATH", ""),
CachePathOverride: getEnv("META_CORE_CACHE_PATH", ""), CachePathOverride: getEnv("META_CORE_CACHE_PATH", ""),
ReportsPathOverride: getEnv("META_CORE_REPORTS_DB_PATH", ""),
ReportsMax: getEnvInt("META_CORE_REPORTS_MAX", 5000),
ReportsMaxMB: getEnvInt("META_CORE_REPORTS_MAX_MB", 256),
HTTPPort: getEnvInt("META_CORE_HTTP_PORT", 9000), HTTPPort: getEnvInt("META_CORE_HTTP_PORT", 9000),
HTTPHost: getEnv("META_CORE_HTTP_HOST", "0.0.0.0"), HTTPHost: getEnv("META_CORE_HTTP_HOST", "0.0.0.0"),
HealthCheckIntervalMS: getEnvInt("HEALTH_CHECK_INTERVAL_MS", 5000), HealthCheckIntervalMS: getEnvInt("HEALTH_CHECK_INTERVAL_MS", 5000),
@@ -110,6 +118,16 @@ func (c *Config) CachePath() string {
return c.MetaCorePath + "/db-cache/cache.db" return c.MetaCorePath + "/db-cache/cache.db"
} }
// ReportsDBPath returns the path of the report inbox database. It sits beside
// the truth database (so it is backed up with it) but is a separate file:
// reports are diagnostics, never metadata.
func (c *Config) ReportsDBPath() string {
if c.ReportsPathOverride != "" {
return c.ReportsPathOverride
}
return c.MetaCorePath + "/db/reports.db"
}
// MountsFilePath returns the path to the mounts configuration file // MountsFilePath returns the path to the mounts configuration file
func (c *Config) MountsFilePath() string { func (c *Config) MountsFilePath() string {
return c.MountsDir + "/mounts.json" return c.MountsDir + "/mounts.json"
+561
View File
@@ -0,0 +1,561 @@
// Package reports is meta-core's bug-report / feedback inbox.
//
// Client apps (meta-watch, meta-read, …) build a report in the browser — the
// line the person typed, the page they were on, the build, the device, the
// tail of the client log, and whatever app-specific state explains the moment
// (a player pipeline, a reader position) — and their server forwards it here,
// server-to-server, to the box's own core. meta-core stores it, lists it and
// shows it on the dashboard's Reports page, so triage reads one inbox instead
// of one per app.
//
// # Why a third SQLite file
//
// Reports are diagnostics, not metadata. They do not belong in the truth
// database (signed claims only) nor in the cache (unsynchronised, never backed
// up, deleted as the recovery procedure). They live in their own file,
// reports.db, so neither a cache rebuild nor POST /api/metadata/clear touches
// them, and a full report table can never slow a record write.
//
// # What meta-core knows about a report
//
// Almost nothing, on purpose. A handful of top-level fields are lifted into
// columns so the inbox can filter and headline them (app, build, uid, page,
// route, cid, title, note); everything else is kept verbatim in body as JSON
// and rendered generically by the dashboard. An app can add a new block of
// state without a meta-core release.
//
// ⚠ A report is a piece of viewing/reading history (cids, titles, a uid, a
// timestamp). It stays on the box, behind the same perimeter as the rest of
// meta-core's API; nothing here is federated.
package reports
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"unicode/utf8"
"github.com/metazla/meta-core/internal/kv"
)
const schema = `
CREATE TABLE IF NOT EXISTS report(
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts INTEGER NOT NULL, -- when the report was filed (ms)
received INTEGER NOT NULL, -- when meta-core stored it (ms)
app TEXT NOT NULL,
build TEXT NOT NULL DEFAULT '',
uid TEXT NOT NULL DEFAULT '',
page TEXT NOT NULL DEFAULT '',
route TEXT NOT NULL DEFAULT '',
cid TEXT NOT NULL DEFAULT '',
title TEXT NOT NULL DEFAULT '',
note TEXT NOT NULL DEFAULT '',
status TEXT NOT NULL DEFAULT 'open',
triage_note TEXT NOT NULL DEFAULT '',
updated INTEGER NOT NULL DEFAULT 0,
log_lines INTEGER NOT NULL DEFAULT 0,
bytes INTEGER NOT NULL,
body TEXT NOT NULL
) STRICT;
CREATE INDEX IF NOT EXISTS report_app_status ON report(app, status, id DESC);
CREATE INDEX IF NOT EXISTS report_status ON report(status, id DESC);
`
// Statuses a report moves through. "open" is where every report starts.
const (
StatusOpen = "open"
StatusTriaged = "triaged"
StatusClosed = "closed"
)
// ValidStatus reports whether s is a status a report can be set to.
func ValidStatus(s string) bool {
return s == StatusOpen || s == StatusTriaged || s == StatusClosed
}
// Bounds on what one report may carry. The string caps match the ones
// meta-watch applied when it kept reports itself, so moving the inbox loses
// nothing a report used to hold.
const (
maxApp = 32
maxBuild = 4000
maxUID = 128
maxPage = 32
maxRoute = 512
maxCID = 128
maxTitle = 200
maxNote = 1000
maxTriageNote = 2000
// MaxLogLines / MaxLogLineChars bound the client log tail.
MaxLogLines = 200
MaxLogLineChars = 600
// MaxBlockBytes bounds every other top-level block (context, device,
// app-specific state). A larger block is replaced by a marker naming its
// size, never dropped silently.
MaxBlockBytes = 32 * 1024
// maxBlocks bounds the number of top-level keys.
maxBlocks = 64
// filed-at plausibility window: a report spooled by an app while the core
// was down arrives late but keeps the time it was filed.
maxFiledAge = 30 * 24 * time.Hour
maxFiledFuture = 5 * time.Minute
)
// ErrNotFound is returned for an id that does not exist.
var ErrNotFound = errors.New("report not found")
// Summary is a report without its body: what the inbox list shows.
type Summary struct {
ID int64 `json:"id"`
TS int64 `json:"ts"`
Received int64 `json:"received"`
App string `json:"app"`
Build string `json:"build,omitempty"`
UID string `json:"uid,omitempty"`
Page string `json:"page,omitempty"`
Route string `json:"route,omitempty"`
CID string `json:"cid,omitempty"`
Title string `json:"title,omitempty"`
Note string `json:"note,omitempty"`
Status string `json:"status"`
TriageNote string `json:"triage_note,omitempty"`
Updated int64 `json:"updated,omitempty"`
LogLines int `json:"log_lines"`
Bytes int64 `json:"bytes"`
}
// Report is a stored report with its full body.
type Report struct {
Summary
Body json.RawMessage `json:"body"`
}
// Filter selects reports for List.
type Filter struct {
App string
Status string
Q string // substring of note, title or cid
Before int64 // only ids below this (paging); 0 = from the newest
Limit int
}
// Page is one page of List.
type Page struct {
Reports []Summary `json:"reports"`
Apps map[string]int64 `json:"apps"` // every app with a report, and how many
Counts map[string]int64 `json:"counts"` // per status, within the App filter
More bool `json:"more"`
}
// Store is the open reports database.
type Store struct {
db *kv.Store
maxRows int
maxBytes int64
now func() time.Time
}
// Open opens (creating if needed) the reports database at path. maxRows and
// maxBytes bound what Prune keeps; <= 0 disables that bound.
func Open(path string, maxRows int, maxBytes int64) (*Store, error) {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return nil, fmt.Errorf("reports dir: %w", err)
}
db, err := kv.Open(path, kv.Options{
NoKVTable: true,
Schema: schema,
WriterCacheMB: 4,
ReaderCacheMB: 2,
Readers: 2,
})
if err != nil {
return nil, fmt.Errorf("open reports %s: %w", path, err)
}
return &Store{db: db, maxRows: maxRows, maxBytes: maxBytes, now: time.Now}, nil
}
// Close closes the database.
func (s *Store) Close() error { return s.db.Close() }
// Path is the database file.
func (s *Store) Path() string { return s.db.Path() }
func opCtx() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), 30*time.Second)
}
// ── Normalisation ─────────────────────────────────────────────────────────
// Normalize turns a posted envelope into a storable report: the indexed
// fields lifted and clamped, the log tail bounded, every other block size-
// checked. It never fails on content an app might plausibly send; the only
// error is a report that names no app.
func Normalize(in map[string]json.RawMessage, now time.Time) (Report, error) {
var r Report
str := func(key string, n int) string {
raw, ok := in[key]
if !ok {
return ""
}
var v string
if json.Unmarshal(raw, &v) != nil {
return ""
}
return clamp(strings.TrimSpace(v), n)
}
r.App = str("app", maxApp)
if r.App == "" {
return r, errors.New("app is required")
}
r.Build = str("build", maxBuild)
r.UID = str("uid", maxUID)
r.Page = str("page", maxPage)
r.Route = str("route", maxRoute)
r.CID = str("cid", maxCID)
r.Title = str("title", maxTitle)
r.Note = str("note", maxNote)
r.Status = StatusOpen
r.Received = now.UnixMilli()
r.TS = r.Received
if raw, ok := in["filedAt"]; ok {
var f float64
if json.Unmarshal(raw, &f) == nil {
t := time.UnixMilli(int64(f))
if t.After(now.Add(-maxFiledAge)) && t.Before(now.Add(maxFiledFuture)) {
r.TS = t.UnixMilli()
}
}
}
// The body repeats the lifted fields (clamped), so a report exported as
// JSON is complete on its own.
body := make(map[string]any, len(in))
for k, v := range map[string]string{
"app": r.App, "build": r.Build, "uid": r.UID, "page": r.Page,
"route": r.Route, "cid": r.CID, "title": r.Title, "note": r.Note,
} {
if v != "" {
body[k] = v
}
}
if raw, ok := in["log"]; ok {
var lines []string
if json.Unmarshal(raw, &lines) == nil {
if len(lines) > MaxLogLines {
lines = lines[len(lines)-MaxLogLines:]
}
for i, l := range lines {
lines[i] = clamp(l, MaxLogLineChars)
}
if len(lines) > 0 {
body["log"] = lines
r.LogLines = len(lines)
}
}
}
// Remaining blocks in key order, so which ones survive maxBlocks does not
// depend on map iteration.
keys := make([]string, 0, len(in))
for k := range in {
if k != "log" && k != "filedAt" && !isLifted(k) {
keys = append(keys, k)
}
}
sort.Strings(keys)
blocks := 0
for _, k := range keys {
raw := in[k]
if string(raw) == "null" {
continue
}
if blocks >= maxBlocks {
break
}
blocks++
key := clamp(k, 64)
if len(raw) > MaxBlockBytes {
body[key] = map[string]any{"truncated": true, "bytes": len(raw)}
continue
}
body[key] = raw
}
b, err := json.Marshal(body)
if err != nil {
return r, fmt.Errorf("encode body: %w", err)
}
r.Body = b
r.Bytes = int64(len(b))
return r, nil
}
func isLifted(k string) bool {
switch k {
case "app", "build", "uid", "page", "route", "cid", "title", "note":
return true
}
return false
}
// clamp keeps at most n characters (runes) of s, never splitting one.
func clamp(s string, n int) string {
if utf8.RuneCountInString(s) <= n {
return s
}
i := 0
for j := range s {
if i == n {
return s[:j]
}
i++
}
return s
}
// ── Writes ────────────────────────────────────────────────────────────────
// Insert stores a normalised report and returns its id. The store is pruned
// to its bounds in the same transaction, so it never sits over them.
func (s *Store) Insert(r Report) (int64, error) {
ctx, cancel := opCtx()
defer cancel()
var id int64
err := s.db.Update(ctx, func(tx *kv.Tx) error {
res, err := tx.Exec(`INSERT INTO report(ts, received, app, build, uid, page, route, cid, title, note,
status, log_lines, bytes, body) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
r.TS, r.Received, r.App, r.Build, r.UID, r.Page, r.Route, r.CID, r.Title, r.Note,
StatusOpen, r.LogLines, r.Bytes, string(r.Body))
if err != nil {
return err
}
if id, err = res.LastInsertId(); err != nil {
return err
}
_, err = s.prune(tx)
return err
})
return id, err
}
// Update sets a report's status and/or triage note. A nil argument leaves
// that field as it is.
func (s *Store) Update(id int64, status, triageNote *string) (Report, error) {
if status != nil && !ValidStatus(*status) {
return Report{}, fmt.Errorf("invalid status %q", *status)
}
ctx, cancel := opCtx()
defer cancel()
err := s.db.Update(ctx, func(tx *kv.Tx) error {
sets, args := []string{"updated = ?"}, []any{s.now().UnixMilli()}
if status != nil {
sets, args = append(sets, "status = ?"), append(args, *status)
}
if triageNote != nil {
sets, args = append(sets, "triage_note = ?"), append(args, clamp(strings.TrimSpace(*triageNote), maxTriageNote))
}
res, err := tx.Exec(`UPDATE report SET `+strings.Join(sets, ", ")+` WHERE id = ?`, append(args, id)...)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
return nil
})
if err != nil {
return Report{}, err
}
return s.Get(id)
}
// Delete removes a report.
func (s *Store) Delete(id int64) error {
ctx, cancel := opCtx()
defer cancel()
return s.db.Update(ctx, func(tx *kv.Tx) error {
res, err := tx.Exec(`DELETE FROM report WHERE id = ?`, id)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
return nil
})
}
// Prune brings the store back within its bounds and returns how many reports
// it removed. Insert already prunes; this is for a bound lowered by config.
func (s *Store) Prune() (int, error) {
ctx, cancel := opCtx()
defer cancel()
var n int
err := s.db.Update(ctx, func(tx *kv.Tx) error {
var err error
n, err = s.prune(tx)
return err
})
return n, err
}
// prune evicts closed reports first, oldest first, then any report, oldest
// first, until both the row and the byte bound hold. An open report is the
// last thing to go: it is the one nobody has read yet.
func (s *Store) prune(tx *kv.Tx) (int, error) {
if s.maxRows <= 0 && s.maxBytes <= 0 {
return 0, nil
}
var rows, bytes int64
if err := tx.QueryRow(`SELECT count(*), coalesce(sum(bytes), 0) FROM report`).Scan(&rows, &bytes); err != nil {
return 0, err
}
over := func() bool {
return (s.maxRows > 0 && rows > int64(s.maxRows)) || (s.maxBytes > 0 && bytes > s.maxBytes)
}
if !over() {
return 0, nil
}
q, err := tx.Query(`SELECT id, bytes FROM report ORDER BY (status = 'closed') DESC, id ASC`)
if err != nil {
return 0, err
}
var victims []int64
for q.Next() && over() {
var id, b int64
if err := q.Scan(&id, &b); err != nil {
q.Close()
return 0, err
}
victims = append(victims, id)
rows--
bytes -= b
}
q.Close()
for _, id := range victims {
if _, err := tx.Exec(`DELETE FROM report WHERE id = ?`, id); err != nil {
return 0, err
}
}
return len(victims), nil
}
// ── Reads ─────────────────────────────────────────────────────────────────
const summaryCols = `id, ts, received, app, build, uid, page, route, cid, title, note, status, triage_note, updated, log_lines, bytes`
func scanSummary(sc interface{ Scan(...any) error }, extra ...any) (Summary, error) {
var r Summary
dst := append([]any{&r.ID, &r.TS, &r.Received, &r.App, &r.Build, &r.UID, &r.Page, &r.Route,
&r.CID, &r.Title, &r.Note, &r.Status, &r.TriageNote, &r.Updated, &r.LogLines, &r.Bytes}, extra...)
return r, sc.Scan(dst...)
}
// Get returns one report with its body.
func (s *Store) Get(id int64) (Report, error) {
ctx, cancel := opCtx()
defer cancel()
var body string
row := s.db.DB().QueryRowContext(ctx, `SELECT `+summaryCols+`, body FROM report WHERE id = ?`, id)
sum, err := scanSummary(row, &body)
if errors.Is(err, sql.ErrNoRows) {
return Report{}, ErrNotFound
}
if err != nil {
return Report{}, err
}
return Report{Summary: sum, Body: json.RawMessage(body)}, nil
}
// List returns one page of summaries, newest first, with the counts the
// inbox's filter bar needs.
func (s *Store) List(f Filter) (Page, error) {
if f.Limit <= 0 || f.Limit > 200 {
f.Limit = 50
}
ctx, cancel := opCtx()
defer cancel()
db := s.db.DB()
var where []string
var args []any
if f.App != "" {
where, args = append(where, "app = ?"), append(args, f.App)
}
appWhere, appArgs := append([]string(nil), where...), append([]any(nil), args...)
if f.Status != "" {
where, args = append(where, "status = ?"), append(args, f.Status)
}
if q := strings.TrimSpace(f.Q); q != "" {
like := "%" + escapeLike(q) + "%"
where = append(where, `(note LIKE ? ESCAPE '\' OR title LIKE ? ESCAPE '\' OR cid LIKE ? ESCAPE '\' OR uid LIKE ? ESCAPE '\')`)
args = append(args, like, like, like, like)
}
if f.Before > 0 {
where, args = append(where, "id < ?"), append(args, f.Before)
}
clause := ""
if len(where) > 0 {
clause = " WHERE " + strings.Join(where, " AND ")
}
page := Page{Reports: []Summary{}, Apps: map[string]int64{}, Counts: map[string]int64{}}
rows, err := db.QueryContext(ctx, `SELECT `+summaryCols+` FROM report`+clause+` ORDER BY id DESC LIMIT ?`,
append(args, f.Limit+1)...)
if err != nil {
return page, err
}
for rows.Next() {
sum, err := scanSummary(rows)
if err != nil {
rows.Close()
return page, err
}
page.Reports = append(page.Reports, sum)
}
rows.Close()
if len(page.Reports) > f.Limit {
page.Reports, page.More = page.Reports[:f.Limit], true
}
if err := groupCount(ctx, db, `SELECT app, count(*) FROM report GROUP BY app`, nil, page.Apps); err != nil {
return page, err
}
statusQ := `SELECT status, count(*) FROM report`
if len(appWhere) > 0 {
statusQ += " WHERE " + strings.Join(appWhere, " AND ")
}
if err := groupCount(ctx, db, statusQ+` GROUP BY status`, appArgs, page.Counts); err != nil {
return page, err
}
return page, nil
}
func groupCount(ctx context.Context, db *sql.DB, q string, args []any, into map[string]int64) error {
rows, err := db.QueryContext(ctx, q, args...)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var k string
var n int64
if err := rows.Scan(&k, &n); err != nil {
return err
}
into[k] = n
}
return rows.Err()
}
func escapeLike(s string) string {
return strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(s)
}
+227
View File
@@ -0,0 +1,227 @@
package reports
import (
"encoding/json"
"path/filepath"
"strings"
"testing"
"time"
)
func open(t *testing.T, maxRows int, maxBytes int64) *Store {
t.Helper()
s, err := Open(filepath.Join(t.TempDir(), "db", "reports.db"), maxRows, maxBytes)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = s.Close() })
return s
}
func envelope(t *testing.T, v map[string]any) map[string]json.RawMessage {
t.Helper()
b, _ := json.Marshal(v)
var out map[string]json.RawMessage
if err := json.Unmarshal(b, &out); err != nil {
t.Fatal(err)
}
return out
}
func file(t *testing.T, s *Store, v map[string]any) int64 {
t.Helper()
r, err := Normalize(envelope(t, v), time.Now())
if err != nil {
t.Fatal(err)
}
id, err := s.Insert(r)
if err != nil {
t.Fatal(err)
}
return id
}
func TestNormalizeRequiresApp(t *testing.T) {
if _, err := Normalize(envelope(t, map[string]any{"note": "x"}), time.Now()); err == nil {
t.Fatal("a report without app must be refused")
}
}
func TestNormalizeClampsAndBounds(t *testing.T) {
log := make([]string, 250)
for i := range log {
log[i] = strings.Repeat("é", 700)
}
now := time.Now()
r, err := Normalize(envelope(t, map[string]any{
"app": "meta-read",
"note": strings.Repeat("n", 1500),
"title": strings.Repeat("t", 300),
"log": log,
"context": map[string]string{"big": strings.Repeat("x", MaxBlockBytes)},
"reader": map[string]int{"page": 4},
"device": nil,
"filedAt": now.Add(-time.Hour).UnixMilli(),
}), now)
if err != nil {
t.Fatal(err)
}
if len(r.Note) != maxNote || len(r.Title) != maxTitle {
t.Fatalf("note/title not clamped: %d %d", len(r.Note), len(r.Title))
}
if r.TS != now.Add(-time.Hour).UnixMilli() || r.Received != now.UnixMilli() {
t.Fatalf("filedAt should set ts, received stays now: %d %d", r.TS, r.Received)
}
var body map[string]json.RawMessage
_ = json.Unmarshal(r.Body, &body)
var lines []string
_ = json.Unmarshal(body["log"], &lines)
if len(lines) != MaxLogLines || r.LogLines != MaxLogLines {
t.Fatalf("log tail: %d lines (col %d)", len(lines), r.LogLines)
}
if n := len([]rune(lines[0])); n != MaxLogLineChars {
t.Fatalf("log line clamped by rune: %d", n)
}
if !strings.Contains(string(body["context"]), `"truncated":true`) {
t.Fatalf("oversized block not replaced by a marker: %.80s", body["context"])
}
if string(body["reader"]) != `{"page":4}` {
t.Fatalf("app block not kept verbatim: %s", body["reader"])
}
if _, ok := body["device"]; ok {
t.Fatal("null block should be dropped")
}
if _, ok := body["filedAt"]; ok {
t.Fatal("filedAt belongs in ts, not the body")
}
}
func TestNormalizeIgnoresImplausibleFiledAt(t *testing.T) {
now := time.Now()
for _, at := range []time.Time{now.Add(-60 * 24 * time.Hour), now.Add(time.Hour)} {
r, _ := Normalize(envelope(t, map[string]any{"app": "a", "filedAt": at.UnixMilli()}), now)
if r.TS != now.UnixMilli() {
t.Fatalf("filedAt %v should be ignored", at)
}
}
}
func TestInsertGetListFilters(t *testing.T) {
s := open(t, 0, 0)
a := file(t, s, map[string]any{"app": "meta-watch", "note": "stall at 3%", "cid": "bafyA"})
b := file(t, s, map[string]any{"app": "meta-read", "note": "blank page", "uid": "u1"})
c := file(t, s, map[string]any{"app": "meta-read", "note": "wrong cover 100%"})
got, err := s.Get(a)
if err != nil || got.App != "meta-watch" || got.CID != "bafyA" || got.Status != StatusOpen {
t.Fatalf("get: %+v %v", got.Summary, err)
}
if _, err := s.Get(999); err != ErrNotFound {
t.Fatalf("missing id: %v", err)
}
p, err := s.List(Filter{})
if err != nil || len(p.Reports) != 3 || p.Reports[0].ID != c {
t.Fatalf("list newest first: %+v %v", p.Reports, err)
}
if p.Apps["meta-read"] != 2 || p.Apps["meta-watch"] != 1 || p.Counts[StatusOpen] != 3 {
t.Fatalf("counts: %+v %+v", p.Apps, p.Counts)
}
p, _ = s.List(Filter{App: "meta-read"})
if len(p.Reports) != 2 || p.Counts[StatusOpen] != 2 || p.Apps["meta-watch"] != 1 {
t.Fatalf("app filter: %+v counts %+v apps %+v", p.Reports, p.Counts, p.Apps)
}
p, _ = s.List(Filter{Q: "100%"})
if len(p.Reports) != 1 || p.Reports[0].ID != c {
t.Fatalf("q is a literal substring (%% escaped): %+v", p.Reports)
}
p, _ = s.List(Filter{Q: "u1"})
if len(p.Reports) != 1 || p.Reports[0].ID != b {
t.Fatalf("q matches uid: %+v", p.Reports)
}
p, _ = s.List(Filter{Limit: 2})
if len(p.Reports) != 2 || !p.More {
t.Fatalf("page 1: %d more=%v", len(p.Reports), p.More)
}
p, _ = s.List(Filter{Limit: 2, Before: p.Reports[1].ID})
if len(p.Reports) != 1 || p.More || p.Reports[0].ID != a {
t.Fatalf("page 2: %+v more=%v", p.Reports, p.More)
}
}
func TestUpdateAndDelete(t *testing.T) {
s := open(t, 0, 0)
id := file(t, s, map[string]any{"app": "a"})
st, note := StatusTriaged, " Deck #900 "
r, err := s.Update(id, &st, &note)
if err != nil || r.Status != StatusTriaged || r.TriageNote != "Deck #900" || r.Updated == 0 {
t.Fatalf("update: %+v %v", r.Summary, err)
}
bad := "done"
if _, err := s.Update(id, &bad, nil); err == nil {
t.Fatal("invalid status accepted")
}
if _, err := s.Update(42, &st, nil); err != ErrNotFound {
t.Fatalf("update missing: %v", err)
}
p, _ := s.List(Filter{Status: StatusTriaged})
if len(p.Reports) != 1 {
t.Fatalf("status filter: %+v", p.Reports)
}
if err := s.Delete(id); err != nil {
t.Fatal(err)
}
if err := s.Delete(id); err != ErrNotFound {
t.Fatalf("second delete: %v", err)
}
}
func TestPruneEvictsClosedThenOldest(t *testing.T) {
s := open(t, 3, 0)
ids := []int64{
file(t, s, map[string]any{"app": "a", "note": "1"}),
file(t, s, map[string]any{"app": "a", "note": "2"}),
file(t, s, map[string]any{"app": "a", "note": "3"}),
}
closed := StatusClosed
if _, err := s.Update(ids[2], &closed, nil); err != nil {
t.Fatal(err)
}
// Fourth report: the closed one (newest of the three) goes first.
file(t, s, map[string]any{"app": "a", "note": "4"})
if _, err := s.Get(ids[2]); err != ErrNotFound {
t.Fatal("closed report should be evicted before older open ones")
}
// Fifth: no closed left, so the oldest open goes.
file(t, s, map[string]any{"app": "a", "note": "5"})
if _, err := s.Get(ids[0]); err != ErrNotFound {
t.Fatal("oldest open report should be evicted next")
}
p, _ := s.List(Filter{})
if len(p.Reports) != 3 {
t.Fatalf("store over its row bound: %d", len(p.Reports))
}
}
func TestPruneByBytes(t *testing.T) {
s := open(t, 0, 3000)
for i := 0; i < 5; i++ {
file(t, s, map[string]any{"app": "a", "note": strings.Repeat("x", 900)})
}
p, _ := s.List(Filter{})
var total int64
for _, r := range p.Reports {
total += r.Bytes
}
if total > 3000 || len(p.Reports) == 0 {
t.Fatalf("byte bound not held: %d bytes in %d reports", total, len(p.Reports))
}
}
func TestClampKeepsRunes(t *testing.T) {
if got := clamp("héllo", 2); got != "hé" {
t.Fatalf("clamp: %q", got)
}
}
+49 -8
View File
@@ -61,6 +61,9 @@ type stats struct {
CIDGroupRows int CIDGroupRows int
CIDConflicts int // cids/<c> on a rid while cid:<c> names another rid CIDConflicts int // cids/<c> on a rid while cid:<c> names another rid
CIDShared int // cids/<c> (no cid: row) already kept by an earlier record
RecordsFolded int // records whose every cid an earlier record kept: merged into it
DupClaims int // (cid, key) already claimed by the legacy signer (folded records)
OrphanCIDRows int // cid:<c> → rid with no file: rows OrphanCIDRows int // cid:<c> → rid with no file: rows
UDLMigrated counts UDLMigrated counts
@@ -74,6 +77,8 @@ type stats struct {
Principals int Principals int
ClaimsWritten int ClaimsWritten int
Elapsed time.Duration Elapsed time.Duration
canon map[string]string // rid → the canonical cid its claims sit on
} }
func newStats() *stats { func newStats() *stats {
@@ -81,6 +86,7 @@ func newStats() *stats {
WatcherFacts: counts{}, RefusedFields: counts{}, WatcherFacts: counts{}, RefusedFields: counts{},
UDLMigrated: counts{}, UDLDropped: counts{}, UDLMigrated: counts{}, UDLDropped: counts{},
Scratch: counts{}, Derived: counts{}, Scratch: counts{}, Derived: counts{},
canon: map[string]string{},
} }
} }
@@ -239,6 +245,7 @@ func migrate(o options) (st *stats, err error) {
st.CIDGroupRows = len(groups) st.CIDGroupRows = len(groups)
st.Principals = len(w.pids) st.Principals = len(w.pids)
st.ClaimsWritten = int(w.seq) st.ClaimsWritten = int(w.seq)
st.DupClaims = w.dups
if !o.dryRun { if !o.dryRun {
if err = w.finish(groups); err != nil { if err = w.finish(groups); err != nil {
return nil, err return nil, err
@@ -439,14 +446,40 @@ func emitRecord(o options, st *stats, legacy *identity.Identity, index map[strin
st.RecordsNoCID++ st.RecordsNoCID++
return nil return nil
} }
canonical := usable[0] // A cid with no cid: index row can sit in two records' cids/* (the same
for _, c := range usable[1:] { // content stored twice). The first record keeps it; a record left with none
canonical = cid.Better(canonical, c) // folds into the record owning its first shared cid, and its fields become
} // claims there (the legacy signer's first claim per (cid, key) wins).
owner, shared := "", ""
var own []string
for _, c := range usable { for _, c := range usable {
groups[c] = rid if r, ok := groups[c]; ok && r != rid {
st.CIDShared++
if owner == "" {
owner, shared = r, c
}
continue
}
own = append(own, c)
}
var canonical string
if len(own) == 0 {
st.RecordsFolded++
rid = owner
if canonical = st.canon[owner]; canonical == "" {
canonical = shared
}
} else {
canonical = own[0]
for _, c := range own[1:] {
canonical = cid.Better(canonical, c)
}
for _, c := range own {
groups[c] = rid
}
st.canon[rid] = canonical
st.Rids++
} }
st.Rids++
emitted := 0 emitted := 0
mk := func(key, value string) error { mk := func(key, value string) error {
emitted++ emitted++
@@ -685,6 +718,7 @@ type writer struct {
db *sql.DB db *sql.DB
pids map[string]int64 pids map[string]int64
seq int64 seq int64
dups int
} }
func (w *writer) write(batch []claim.Claim) error { func (w *writer) write(batch []claim.Claim) error {
@@ -702,7 +736,7 @@ func (w *writer) write(batch []claim.Claim) error {
return err return err
} }
defer tx.Rollback() defer tx.Rollback()
ins, err := tx.Prepare(`INSERT INTO claim(cid, key, pid, value, ts, sig, seq) VALUES(?, ?, ?, ?, ?, ?, ?)`) ins, err := tx.Prepare(`INSERT OR IGNORE INTO claim(cid, key, pid, value, ts, sig, seq) VALUES(?, ?, ?, ?, ?, ?, ?)`)
if err != nil { if err != nil {
return err return err
} }
@@ -716,9 +750,14 @@ func (w *writer) write(batch []claim.Claim) error {
w.pids[cl.UID] = pid w.pids[cl.UID] = pid
} }
w.seq++ w.seq++
if _, err := ins.Exec(cl.CID, cl.Key, pid, valueArg(cl.Value), int64(cl.TS), cl.Sig, w.seq); err != nil { res, err := ins.Exec(cl.CID, cl.Key, pid, valueArg(cl.Value), int64(cl.TS), cl.Sig, w.seq)
if err != nil {
return fmt.Errorf("claim (%s, %s, %s): %w", cl.CID, cl.Key, cl.UID, err) return fmt.Errorf("claim (%s, %s, %s): %w", cl.CID, cl.Key, cl.UID, err)
} }
if n, _ := res.RowsAffected(); n == 0 { // a folded record's duplicate
w.seq--
w.dups++
}
} }
return tx.Commit() return tx.Commit()
} }
@@ -875,6 +914,8 @@ func printSummary(w io.Writer, st *stats, dryRun bool) {
fmt.Fprintf(w, "principals %d\n", st.Principals) fmt.Fprintf(w, "principals %d\n", st.Principals)
fmt.Fprintf(w, "cid_group rows %d (%d minted for UDL-only cids)\n", st.CIDGroupRows, st.GroupsMinted) fmt.Fprintf(w, "cid_group rows %d (%d minted for UDL-only cids)\n", st.CIDGroupRows, st.GroupsMinted)
fmt.Fprintf(w, "cid conflicts %d (cids/<c> on a rid while cid:<c> names another; index wins)\n", st.CIDConflicts) fmt.Fprintf(w, "cid conflicts %d (cids/<c> on a rid while cid:<c> names another; index wins)\n", st.CIDConflicts)
fmt.Fprintf(w, "shared cids %d (cids/<c> kept by an earlier record; %d records folded into it)\n", st.CIDShared, st.RecordsFolded)
fmt.Fprintf(w, "duplicate claims skipped %d (folded records' (cid, key) already claimed; not counted in dry run)\n", st.DupClaims)
fmt.Fprintf(w, "orphan cid: rows dropped %d (rid has no file: rows)\n", st.OrphanCIDRows) fmt.Fprintf(w, "orphan cid: rows dropped %d (rid has no file: rows)\n", st.OrphanCIDRows)
fmt.Fprintf(w, "watcher facts skipped %d\n", st.WatcherFacts.total()) fmt.Fprintf(w, "watcher facts skipped %d\n", st.WatcherFacts.total())
printCounts(w, " ", st.WatcherFacts) printCounts(w, " ", st.WatcherFacts)
+68
View File
@@ -373,3 +373,71 @@ func TestHalfFloat(t *testing.T) {
} }
} }
} }
// Two records listing the same cid in cids/* with no cid: index row (the same
// content stored twice): the first keeps it, a record left with no cid of its
// own folds into it, and the duplicate (cid, key) claims are skipped.
func TestSharedUnindexedCIDFolds(t *testing.T) {
dir := t.TempDir()
oldPath := filepath.Join(dir, "v1.db")
db, err := sql.Open("sqlite", "file:"+oldPath)
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec(`CREATE TABLE kv(k TEXT PRIMARY KEY, v TEXT NOT NULL) WITHOUT ROWID`); err != nil {
t.Fatal(err)
}
for k, v := range map[string]string{
"meta-core:schema-version": "2",
"file:" + ridA + "/cids/" + cidSHA: "true",
"file:" + ridA + "/title": "A",
"file:" + ridB + "/cids/" + cidSHA: "true",
"file:" + ridB + "/title": "B",
"file:" + ridB + "/genres/Drama": "true",
"file:" + ridC + "/cids/" + cidSHA: "true",
"file:" + ridC + "/cids/" + cidB: "true",
"file:" + ridC + "/title": "C",
} {
if _, err := db.Exec(`INSERT INTO kv(k, v) VALUES(?, ?)`, k, v); err != nil {
t.Fatal(err)
}
}
db.Close()
truthPath := filepath.Join(dir, "truth.db")
st, err := migrate(options{oldPath: oldPath, truthPath: truthPath, workers: 2})
if err != nil {
t.Fatalf("migrate: %v", err)
}
if st.Rids != 2 || st.RecordsFolded != 1 || st.CIDShared != 2 || st.DupClaims != 1 {
t.Errorf("rids %d folded %d shared %d dups %d", st.Rids, st.RecordsFolded, st.CIDShared, st.DupClaims)
}
out, err := sql.Open("sqlite", "file:"+truthPath+"?mode=ro")
if err != nil {
t.Fatal(err)
}
defer out.Close()
groups := map[string]string{}
rows, err := out.Query(`SELECT cid, rid FROM cid_group`)
if err != nil {
t.Fatal(err)
}
for rows.Next() {
var c, r string
if err := rows.Scan(&c, &r); err != nil {
t.Fatal(err)
}
groups[c] = r
}
rows.Close()
if groups[cidSHA] != ridA || groups[cidB] != ridC || len(groups) != 2 {
t.Errorf("cid_group %v", groups)
}
var n int
if err := out.QueryRow(`SELECT count(*) FROM claim WHERE cid = ? AND key IN ('title', 'genres/Drama')`, cidSHA).Scan(&n); err != nil || n != 2 {
t.Errorf("claims on the shared cid: %d (%v)", n, err)
}
var title string
if err := out.QueryRow(`SELECT value FROM claim WHERE cid = ? AND key = 'title'`, cidB).Scan(&title); err != nil || title != "C" {
t.Errorf("record C title on its own cid: %q (%v)", title, err)
}
}