improved search
Publish image / publish (push) Failing after 13s

This commit is contained in:
Worph
2026-09-12 16:37:22 +02:00
parent e9bf9d3f34
commit aed0db3cda
12 changed files with 2598 additions and 72 deletions
+43 -17
View File
@@ -1,12 +1,23 @@
# meta-feeder-book
Book **feeder sidecar** for [MetaMesh](https://github.com/worph/meta-gateway) —
surfaces public-domain books from **Project Gutenberg** (via the
[Gutendex](https://gutendex.com) API) into a meta-gateway.
Book **feeder sidecar** for [MetaMesh](https://github.com/worph/meta-gateway).
One binary, `book-feeder`, hosts two upstreams:
It is the proof-of-concept feeder: a single upstream, no preview-seeding, no
BitTorrent — just enough to exercise the `meta-feeder-sdk` + feeder HTTP
contract end to end.
| Upstream | Serves | What it is |
|----------|--------|------------|
| `gutenberg` | `document` × `book` | Public-domain **editions** from Project Gutenberg (via the [Gutendex](https://gutendex.com) API), with an epub url-locator per record |
| `openlibrary` | `card` × `book` | **Work cards** from [Open Library](https://openlibrary.org) — the book identity tier (`TMDB : video :: AniList : comics :: Open Library : books`) |
The two meet through `openlibraryid`. A Gutenberg edition carries it only when
Open Library itself records the link: exactly one Open Library work lists the
etext's Project Gutenberg id (`search.json?q=id_project_gutenberg:<id>`). The
edition then also carries the transient trust claim `anchored=true`,
`anchorSource=openlibrary`, which the gateway stores as
`anchoredBy/openlibrary:OL…W`. There is no title or author matching.
An Open Library card is addressed `compute_card_cid("openlibrary", "OL…W")`,
the bare work id, byte-identical to meta-read's `card_cid::openlibrary`. A work
with no cover or no description gets no card.
## Role in MetaMesh
@@ -32,9 +43,16 @@ drives the contract:
|---------|---------|-------|
| `META_FEEDER_HTTP_LISTEN` | `0.0.0.0:8080` | HTTP listen address |
| `META_FEEDER_STATE_DIR` | `/data/meta-feeder` | redb cache + state |
| `OPENLIBRARY_CONTACT` | — | e-mail or URL sent in the Open Library User-Agent. First-boot seed for the `contact` config field; a value saved in the config UI wins |
| `OPENLIBRARY_TOP_N` | `12` | cards per search |
| `OPENLIBRARY_DISCOVERY_N` | `20` | cards per browse row (`popular:true` → Open Library trending, `fresh:true` → recently added) |
| `RUST_LOG` | `info` | tracing filter |
No upstream API key is required — Gutendex is open.
No upstream API key is required. Open Library asks API clients to identify
themselves, so set a contact in production: an identified client gets a larger
request budget. Both upstreams share one Open Library rate budget. The config
page edits the `gutenberg` upstream, and the contact saved there is read by
both.
## Image
@@ -47,24 +65,32 @@ tag) and on `v*` tags (semver tags).
## Build locally
The build context is the **repo root** (the Cargo workspace) so the vendored
`meta-feeder-sdk` path dependency resolves:
The build context is the **repo root** (the Cargo workspace).
`meta-feeder-sdk` is a git dependency pinned by tag, so the builder needs
network access to github.com:
```bash
docker build -f feeder-plugin/book-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-book:dev .
```
Tests (`cargo` is not needed on the host; wiremock stands in for every upstream):
```bash
docker run --rm -v "$PWD":/src -w /src -e CARGO_TARGET_DIR=/target rust:1.89 \
cargo test -p book-feeder
```
## Repo layout
This repo is a self-contained Cargo workspace vendored out of the
`meta-gateway` monorepo:
```
Cargo.toml # workspace: members = crates/*, feeder-plugin/*
crates/meta-feeder-sdk/ # vendored shared feeder SDK
Cargo.toml # workspace: members = feeder-plugin/*
feeder-plugin/book-feeder/ # this feeder's crate + Dockerfile
src/gutenberg.rs # `gutenberg` upstream
src/openlibrary.rs # `openlibrary` upstream (cards)
src/openlibrary_client.rs # Open Library client (budget, cache, binding lookup)
crates/meta-feeder-sdk/ # old vendored SDK copy — no longer built (excluded)
```
Upstream source of truth for the SDK and the feeder crate is
[`worph/meta-gateway`](https://github.com/worph/meta-gateway); changes there are
vendored back into this repo.
`meta-feeder-sdk` comes from [`worph/meta-feeder-sdk`](https://github.com/worph/meta-feeder-sdk)
at the tag in `feeder-plugin/book-feeder/Cargo.toml`. An SDK change reaches
this feeder only when it is tagged and the pin is bumped.