12 Commits
Author SHA1 Message Date
Worph e6a9ab9e45 CI: push to ipcr-push-85-17-246-67.ip.nsl.sh (the box's cert no longer covers nip.io) [skip ci] 2026-11-04 14:31:35 +00:00
Manielle 02bff131ff CI accepts four-part tags (v1.2.3.1) for fixes
Publish image / publish (push) Successful in 1m50s
Test / cargo test (push) Successful in 5m59s
2026-10-25 14:22:06 +01:00
Manielle 560effe226 clippy: a char array for the subtitle separators in main_title (manual_pattern_char_comparison)
Test / cargo test (push) Successful in 5m29s
Publish image / publish (push) Failing after 15m37s
2026-10-19 20:49:39 +00:00
Manielle e14797e57b ci: no debug info and two jobs at most, so the test link fits in the runner's memory
Test / cargo test (push) Failing after 3m48s
The test job's linker was killed with signal 9 twice (run 705): cargo test links
several test binaries at once, each with full debug info. CARGO_PROFILE_DEV_DEBUG=0
and CARGO_BUILD_JOBS=2 keep the link within the job's memory; the tests are the
same.
2026-10-19 20:45:18 +00:00
Manielle efba2f2825 gutenberg: one bind() for the search and redeem paths
Test / cargo test (push) Failing after 2m47s
compute_outcomes still passed the identifier-only lookup's Option<String> to
into_discovery_record, which now takes the binding (CI: E0308). Both paths now
call bind(): the identifier, else the title match for an etext Open Library
lists under no work, so a redeemed record is bound exactly as its search record
was.
2026-10-19 19:38:22 +00:00
Manielle d177e7b710 gutenberg: bind by title when Open Library lists no work, and answer work pages by searching
Test / cargo test (push) Failing after 2m35s
Two reasons a book's page in meta-read showed no Gutenberg edition, both found
on the live stack with Frankenstein (OL450063W, etext 84):

- An edition was bound to its work only when Open Library's search lists the
  etext under exactly one work (id_project_gutenberg). For many classics it
  lists it under none: no edition of OL450063W carries etext 84. Such an etext
  is now bound by title and author: same main title (main_title), an author
  with the same surname, and the most published of those works, since Open
  Library holds many small "Frankenstein" works by Mary Shelley. The record
  carries the same claim plus anchorMethod=title, so the gateway records
  matchedBy/openlibrary:OL...W and meta-read grades it Matched, below an
  identifier binding. An etext several works list (an unmerged duplicate)
  still gets no binding: gutenberg_etext keeps "several" apart from "none".
- A work page asks for openlibraryid:OL...W with no text, which fell through to
  the catalogue browse (the 32 most downloaded books), so a book outside them
  never reached its own page. Such a query now searches Gutendex for the work's
  main title and its first author's surname (work_search_text); the gateway
  still keeps only the records bound to the work.

Tests: title binding graded Matched while an identifier binding stays
Anchored; a work query searches "frankenstein shelley"; main_title, surname,
the most-published choice, ties and non-matches; Etext one/several/unlisted
and its cache form.
2026-10-19 19:01:28 +00:00
Manielle 6bfe36edb5 take meta-feeder-sdk from gitea, and run the tests on every push
github.com/worph/meta-feeder-sdk answers 404, so a clean build could not fetch
the dependency; the same tag lives on gitea-watch at the same commit (v1.7.0 =
027917c8), so only the source URL changes, in Cargo.toml and Cargo.lock.

The repo had only the image workflow; test.yml runs cargo test --locked and
clippy -D warnings, like meta-read's.
2026-10-19 19:01:28 +00:00
Worph 8bf94cb186 release (first build on gitea-watch / IPCR)
Publish image / publish (push) Successful in 2m11s
2026-10-13 11:03:04 +00:00
Worph 9d20e3ae15 gitea workflow: push to IPCR's direct host (Cloudflare 100 MB cap) [skip ci] 2026-10-13 10:02:33 +00:00
Worph 87df5ef104 gitea workflow: publish to IPCR (metadec.eth) instead of ghcr
Publish image / publish (push) Successful in 2m12s
2026-10-12 18:30:48 +00:00
Worph e87afa68c8 SDK v1.7.0: advertises over beacon v2 as metamesh.feeder/<upstream>
Publish image / publish (push) Failing after 14s
2026-10-04 17:35:16 +00:00
Worph aed0db3cda improved search
Publish image / publish (push) Failing after 13s
2026-09-12 16:37:22 +02:00
14 changed files with 3198 additions and 74 deletions
+88
View File
@@ -0,0 +1,88 @@
# Gitea twin of .github/workflows/publish.yml. Gitea runs only .gitea/workflows/ once this
# folder exists, and GitHub never reads it.
#
# Publishes to IPCR on watch.nsl.sh instead of ghcr.io:
# push ipcr-push-85-17-246-67.ip.nsl.sh/metadec.eth/<repo>:<tag> docker login as metadec.eth with the
# IPCR_PUSH_TOKEN secret (the metadec.eth namespace token from the IPCR dashboard)
# pull ipcr-watch.nsl.sh/ipns/metadec.eth/metadec.eth/<repo>:<tag> anonymous
# The push goes to ipcr-push-85-17-246-67.ip.nsl.sh, the box's own address, not ipcr-push-watch.nsl.sh: Cloudflare fronts
# *.watch.nsl.sh and refuses any request over 100 MB (413), which a large image layer is.
# That host's certificate is signed by the box's private Mesh Router CA, given to buildkit from
# the IPCR_CA_PEM repo variable. docker login would make the job's Docker daemon check the
# certificate without that CA, so the login is written straight into ~/.docker/config.json.
# The runner's job image (node:22-bookworm) has no docker CLI, so the static one is
# installed first; buildx installs itself. No `type=gha` cache: the runner has none.
name: Publish image
# Build and push the feeder image to the IPCR (ipcr-push-watch.nsl.sh):
# * every push to main -> the moving `main` tag
# * every `v*` tag -> semver tags (e.g. v1.0.0 -> 1.0.0, 1.0, 1)
# The build context is the repo root (the workspace); `meta-feeder-sdk` is a
# tag-pinned git dependency fetched during the build. The Dockerfile lives under
# feeder-plugin/book-feeder/.
on:
push:
branches: [main]
tags: ['v*']
env:
REGISTRY: ipcr-push-85-17-246-67.ip.nsl.sh
IMAGE_NAME: metadec.eth/${{ github.event.repository.name }}
jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Docker CLI
run: curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-28.5.1.tgz | tar -xz -C /usr/local/bin --strip-components=1 docker/docker
- name: Trust the Mesh Router CA
run: printf '%s\n' "$IPCR_CA_PEM" > /tmp/ipcr-ca.pem
env:
IPCR_CA_PEM: ${{ vars.IPCR_CA_PEM }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
version: latest
buildkitd-config-inline: |
[registry."${{ env.REGISTRY }}"]
ca = ["/tmp/ipcr-ca.pem"]
- name: Log in to IPCR
env:
IPCR_PUSH_TOKEN: ${{ secrets.IPCR_PUSH_TOKEN }}
run: |
mkdir -p ~/.docker
printf '{"auths":{"%s":{"auth":"%s"}}}\n' "$REGISTRY" "$(printf 'metadec.eth:%s' "$IPCR_PUSH_TOKEN" | base64 -w0)" > ~/.docker/config.json
- name: Extract image metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=semver,pattern={{version}}
# A fix to release 1.2.3 is tagged v1.2.3.1: four numbers are not semver, so
# the pattern above makes no tag for it; this one does (image tag 1.2.3.1).
type=match,pattern=v(\d+\.\d+\.\d+\.\d+),group=1
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha,format=short
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
file: ./feeder-plugin/book-feeder/Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
+31
View File
@@ -0,0 +1,31 @@
# Runs the feeder's tests on every push. The repo had only the image workflow,
# so nothing checked a change before it shipped.
name: Test
on:
push:
branches: ["**"]
jobs:
rust:
name: cargo test
runs-on: ubuntu-latest
# The job ran out of memory linking the test binaries (ld killed with
# signal 9, run 705 twice): no debug info, and at most two links at once.
# What is tested does not change.
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CARGO_BUILD_JOBS: "2"
steps:
- uses: actions/checkout@v4
# rust-toolchain.toml pins the channel (with clippy); rustup installs it on
# first use, so the action only has to provide rustup itself.
- uses: dtolnay/rust-toolchain@stable
with:
components: clippy
- uses: Swatinem/rust-cache@v2
- name: cargo test
run: cargo test --locked
- name: cargo clippy
run: cargo clippy --all-targets -- -D warnings
+5 -2
View File
@@ -3,8 +3,8 @@ name: Publish image
# Build and push the feeder image to the GitHub Container Registry (ghcr.io): # Build and push the feeder image to the GitHub Container Registry (ghcr.io):
# * every push to main -> the moving `main` tag # * every push to main -> the moving `main` tag
# * every `v*` tag -> semver tags (e.g. v1.0.0 -> 1.0.0, 1.0, 1) # * every `v*` tag -> semver tags (e.g. v1.0.0 -> 1.0.0, 1.0, 1)
# The build context is the repo root (the workspace) so the vendored # The build context is the repo root (the workspace); `meta-feeder-sdk` is a
# `meta-feeder-sdk` path dependency resolves; the Dockerfile lives under # tag-pinned git dependency fetched during the build. The Dockerfile lives under
# feeder-plugin/book-feeder/. # feeder-plugin/book-feeder/.
on: on:
push: push:
@@ -43,6 +43,9 @@ jobs:
tags: | tags: |
type=ref,event=branch type=ref,event=branch
type=semver,pattern={{version}} type=semver,pattern={{version}}
# A fix to release 1.2.3 is tagged v1.2.3.1: four numbers are not semver, so
# the pattern above makes no tag for it; this one does (image tag 1.2.3.1).
type=match,pattern=v(\d+\.\d+\.\d+\.\d+),group=1
type=semver,pattern={{major}}.{{minor}} type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}} type=semver,pattern={{major}}
type=sha,format=short type=sha,format=short
Generated
+3 -2
View File
@@ -791,7 +791,8 @@ checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
[[package]] [[package]]
name = "meta-feeder-sdk" name = "meta-feeder-sdk"
version = "0.1.0" version = "1.7.0"
source = "git+https://gitea-watch.nsl.sh/metastar/meta-feeder-sdk?tag=v1.7.0#027917c8cffe7ca43dadcf3b426cc5f69ecc443a"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-trait", "async-trait",
@@ -800,13 +801,13 @@ dependencies = [
"bytes", "bytes",
"futures", "futures",
"hyper", "hyper",
"libc",
"redb", "redb",
"regex", "regex",
"reqwest", "reqwest",
"serde", "serde",
"serde_json", "serde_json",
"sha2", "sha2",
"tempfile",
"thiserror 1.0.69", "thiserror 1.0.69",
"tokio", "tokio",
"tracing", "tracing",
+5 -1
View File
@@ -1,6 +1,10 @@
[workspace] [workspace]
resolver = "2" resolver = "2"
members = ["crates/*", "feeder-plugin/*"] members = ["feeder-plugin/*"]
# The vendored SDK copy is no longer built: book-feeder pins meta-feeder-sdk by
# git tag. Excluded (not deleted) so a stray `cargo` run inside it does not try
# to join this workspace.
exclude = ["crates/meta-feeder-sdk"]
[workspace.package] [workspace.package]
edition = "2021" edition = "2021"
+43 -17
View File
@@ -1,12 +1,23 @@
# meta-feeder-book # meta-feeder-book
Book **feeder sidecar** for [MetaMesh](https://github.com/worph/meta-gateway) — Book **feeder sidecar** for [MetaMesh](https://github.com/worph/meta-gateway).
surfaces public-domain books from **Project Gutenberg** (via the One binary, `book-feeder`, hosts two upstreams:
[Gutendex](https://gutendex.com) API) into a meta-gateway.
It is the proof-of-concept feeder: a single upstream, no preview-seeding, no | Upstream | Serves | What it is |
BitTorrent — just enough to exercise the `meta-feeder-sdk` + feeder HTTP |----------|--------|------------|
contract end to end. | `gutenberg` | `document` × `book` | Public-domain **editions** from Project Gutenberg (via the [Gutendex](https://gutendex.com) API), with an epub url-locator per record |
| `openlibrary` | `card` × `book` | **Work cards** from [Open Library](https://openlibrary.org) — the book identity tier (`TMDB : video :: AniList : comics :: Open Library : books`) |
The two meet through `openlibraryid`. A Gutenberg edition carries it only when
Open Library itself records the link: exactly one Open Library work lists the
etext's Project Gutenberg id (`search.json?q=id_project_gutenberg:<id>`). The
edition then also carries the transient trust claim `anchored=true`,
`anchorSource=openlibrary`, which the gateway stores as
`anchoredBy/openlibrary:OL…W`. There is no title or author matching.
An Open Library card is addressed `compute_card_cid("openlibrary", "OL…W")`,
the bare work id, byte-identical to meta-read's `card_cid::openlibrary`. A work
with no cover or no description gets no card.
## Role in MetaMesh ## Role in MetaMesh
@@ -32,9 +43,16 @@ drives the contract:
|---------|---------|-------| |---------|---------|-------|
| `META_FEEDER_HTTP_LISTEN` | `0.0.0.0:8080` | HTTP listen address | | `META_FEEDER_HTTP_LISTEN` | `0.0.0.0:8080` | HTTP listen address |
| `META_FEEDER_STATE_DIR` | `/data/meta-feeder` | redb cache + state | | `META_FEEDER_STATE_DIR` | `/data/meta-feeder` | redb cache + state |
| `OPENLIBRARY_CONTACT` | — | e-mail or URL sent in the Open Library User-Agent. First-boot seed for the `contact` config field; a value saved in the config UI wins |
| `OPENLIBRARY_TOP_N` | `12` | cards per search |
| `OPENLIBRARY_DISCOVERY_N` | `20` | cards per browse row (`popular:true` → Open Library trending, `fresh:true` → recently added) |
| `RUST_LOG` | `info` | tracing filter | | `RUST_LOG` | `info` | tracing filter |
No upstream API key is required — Gutendex is open. No upstream API key is required. Open Library asks API clients to identify
themselves, so set a contact in production: an identified client gets a larger
request budget. Both upstreams share one Open Library rate budget. The config
page edits the `gutenberg` upstream, and the contact saved there is read by
both.
## Image ## Image
@@ -47,24 +65,32 @@ tag) and on `v*` tags (semver tags).
## Build locally ## Build locally
The build context is the **repo root** (the Cargo workspace) so the vendored The build context is the **repo root** (the Cargo workspace).
`meta-feeder-sdk` path dependency resolves: `meta-feeder-sdk` is a git dependency pinned by tag, so the builder needs
network access to github.com:
```bash ```bash
docker build -f feeder-plugin/book-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-book:dev . docker build -f feeder-plugin/book-feeder/Dockerfile -t ghcr.io/worph/meta-feeder-book:dev .
``` ```
Tests (`cargo` is not needed on the host; wiremock stands in for every upstream):
```bash
docker run --rm -v "$PWD":/src -w /src -e CARGO_TARGET_DIR=/target rust:1.89 \
cargo test -p book-feeder
```
## Repo layout ## Repo layout
This repo is a self-contained Cargo workspace vendored out of the
`meta-gateway` monorepo:
``` ```
Cargo.toml # workspace: members = crates/*, feeder-plugin/* Cargo.toml # workspace: members = feeder-plugin/*
crates/meta-feeder-sdk/ # vendored shared feeder SDK
feeder-plugin/book-feeder/ # this feeder's crate + Dockerfile feeder-plugin/book-feeder/ # this feeder's crate + Dockerfile
src/gutenberg.rs # `gutenberg` upstream
src/openlibrary.rs # `openlibrary` upstream (cards)
src/openlibrary_client.rs # Open Library client (budget, cache, binding lookup)
crates/meta-feeder-sdk/ # old vendored SDK copy — no longer built (excluded)
``` ```
Upstream source of truth for the SDK and the feeder crate is `meta-feeder-sdk` comes from [`worph/meta-feeder-sdk`](https://github.com/worph/meta-feeder-sdk)
[`worph/meta-gateway`](https://github.com/worph/meta-gateway); changes there are at the tag in `feeder-plugin/book-feeder/Cargo.toml`. An SDK change reaches
vendored back into this repo. this feeder only when it is tagged and the pin is bumped.
+9 -4
View File
@@ -6,9 +6,10 @@ rust-version.workspace = true
license.workspace = true license.workspace = true
publish = false publish = false
# Book feeder sidecar: Project Gutenberg (via Gutendex). The proof-of-concept # Book feeder sidecar, two upstreams:
# feeder — single upstream, no preview-seeding, no BitTorrent — that validates # - `gutenberg` — Project Gutenberg editions (via Gutendex), the byte tier.
# the meta-feeder-sdk + feeder HTTP contract end to end. # - `openlibrary` — Open Library work cards, the book identity tier
# (`TMDB : video :: AniList : comics :: Open Library : books`).
[lib] [lib]
name = "book_feeder" name = "book_feeder"
@@ -19,7 +20,11 @@ name = "book-feeder"
path = "src/main.rs" path = "src/main.rs"
[dependencies] [dependencies]
meta-feeder-sdk = { path = "../../crates/meta-feeder-sdk" } # Pinned by TAG, never by path. A path dep resolves OUTSIDE the crate, and the
# vendored copy this used to point at had already drifted behind the shared SDK
# (no `RateBudget`, a capped card cid). The rest of the fleet pins the same
# way; see meta-feeder-anilist's manifest.
meta-feeder-sdk = { git = "https://gitea-watch.nsl.sh/metastar/meta-feeder-sdk", tag = "v1.7.0" }
tokio = { version = "1.40", features = ["full"] } tokio = { version = "1.40", features = ["full"] }
axum = { version = "0.7", features = ["json"] } axum = { version = "0.7", features = ["json"] }
+7 -7
View File
@@ -1,12 +1,13 @@
# syntax=docker/dockerfile:1 # syntax=docker/dockerfile:1
# ============================================================================= # =============================================================================
# book-feeder — Project Gutenberg feeder sidecar. Multi-stage build. # book-feeder — Project Gutenberg + Open Library feeder sidecar. Multi-stage
# build; build context is this repo's root (the Cargo workspace).
# #
# The build context must be `packages/meta-gateway/` (the workspace root) so # meta-feeder-sdk arrives as a git dependency (tag-pinned), so the vendored
# the `meta-feeder-sdk` path dependency (../../crates/meta-feeder-sdk) resolves. # `crates/` copy is not copied in — `git` must be present in the builder for
# Same BuildKit cache-mount strategy as the gateway Dockerfile: the cargo # cargo to fetch it. Same BuildKit cache-mount strategy as the gateway
# registry/git and `target/` persist across builds. The release binary is # Dockerfile: the cargo registry/git and `target/` persist across builds. The
# copied OUT of the target cache mount within the same RUN. # release binary is copied OUT of the target cache mount within the same RUN.
# ============================================================================= # =============================================================================
FROM rust:1.89-slim-bookworm AS builder FROM rust:1.89-slim-bookworm AS builder
@@ -20,7 +21,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./ COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
COPY crates/ crates/
COPY feeder-plugin/ feeder-plugin/ COPY feeder-plugin/ feeder-plugin/
RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \ RUN --mount=type=cache,id=cargo-registry,target=/usr/local/cargo/registry \
+674 -12
View File
@@ -6,17 +6,24 @@
//! don't expose an epub variant fail `compute_outcomes` with a permanent //! don't expose an epub variant fail `compute_outcomes` with a permanent
//! error (operator-visible; very rare in practice). //! error (operator-visible; very rare in practice).
//! //!
//! No auth, no API key, no rate-limit configuration. Moved verbatim from the //! - Per-edition metadata: `GET {rdf_base}/cache/epub/{id}/pg{id}.rdf` for the
//! gateway crate's `plugins/gutenberg.rs`; only the import paths changed //! fields Gutendex does not carry (summary, release date, credits).
//! (`crate::*`/`super::common` → `meta_feeder_sdk::*`, `GatewayPlugin` → //! - Work binding: Open Library's `search.json?q=id_project_gutenberg:{id}` —
//! `FeederPlugin`). //! see [`OpenLibraryClient::gutenberg_work`] and reading-model §2.4.
//!
//! No auth, no API key. Originally moved verbatim from the gateway crate's
//! `plugins/gutenberg.rs`.
use std::collections::{BTreeMap, HashMap}; use std::collections::{BTreeMap, HashMap};
use std::path::Path; use std::path::Path;
use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use meta_feeder_sdk::budget::RateBudget;
use meta_feeder_sdk::cache::MidhashCache; use meta_feeder_sdk::cache::MidhashCache;
use meta_feeder_sdk::common; use meta_feeder_sdk::common;
use meta_feeder_sdk::config::ConfigSchema;
use meta_feeder_sdk::lang::normalize_lang_code;
#[cfg(test)] #[cfg(test)]
use meta_feeder_sdk::plugin::HashKind; use meta_feeder_sdk::plugin::HashKind;
use meta_feeder_sdk::plugin::{upstream_id_field, ConfigError, FeederPlugin, HashOutcome}; use meta_feeder_sdk::plugin::{upstream_id_field, ConfigError, FeederPlugin, HashOutcome};
@@ -25,10 +32,52 @@ use meta_feeder_sdk::types::{DiscoveryRecord, GatewayError, PluginHealth};
use serde::Deserialize; use serde::Deserialize;
use tracing::{debug, warn}; use tracing::{debug, warn};
use crate::openlibrary::first_filter;
use crate::openlibrary_client::{self, Etext, OpenLibraryClient};
/// An edition's Open Library work, and how it was found.
#[derive(Debug, Clone)]
struct Binding {
work: String,
/// Bound by title and author ([`OpenLibraryClient::title_work`]) rather than
/// by an identifier Open Library records: graded Matched, not Anchored.
by_title: bool,
}
/// One etext's binding: the identifier Open Library records, else — only for an
/// etext it lists under no work — the title match. Shared by the search path and
/// the redeem path so the two cannot bind the same book differently.
async fn bind(
client: &OpenLibraryClient,
id: u64,
title: &str,
author: &str,
deadline: std::time::Duration,
) -> Result<Option<Binding>, GatewayError> {
Ok(match client.gutenberg_etext(id, deadline).await? {
Etext::One(work) => Some(Binding { work, by_title: false }),
Etext::Several => None,
Etext::Unlisted => client
.title_work(title, author, deadline)
.await?
.map(|work| Binding { work, by_title: true }),
})
}
/// Canonical Gutendex base URL. Overridable via [`GutenbergPlugin::with_base_url`] /// Canonical Gutendex base URL. Overridable via [`GutenbergPlugin::with_base_url`]
/// for tests + private mirrors. /// for tests + private mirrors.
const DEFAULT_BASE_URL: &str = "https://gutendex.com"; const DEFAULT_BASE_URL: &str = "https://gutendex.com";
/// Canonical host for the per-edition RDF. Overridable via
/// [`GutenbergPlugin::with_rdf_base_url`] — until it was, every unit test that
/// ran a query also fetched live RDF from gutenberg.org.
const DEFAULT_RDF_BASE_URL: &str = "https://www.gutenberg.org";
/// Upper bound on one binding lookup on the live search path, on top of the
/// budget deadline. A slow Open Library must not hold a Gutenberg result that
/// is already in hand.
const BINDING_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(8);
/// Format we treat as the canonical hashable bytes for a Gutenberg record. /// Format we treat as the canonical hashable bytes for a Gutenberg record.
const EPUB_FORMAT: &str = "application/epub+zip"; const EPUB_FORMAT: &str = "application/epub+zip";
@@ -64,7 +113,18 @@ const RETRY_BACKOFF_MS: u64 = 250;
pub struct GutenbergPlugin { pub struct GutenbergPlugin {
http: reqwest::Client, http: reqwest::Client,
base_url: String, base_url: String,
rdf_base_url: String,
cache: Option<MidhashCache>, cache: Option<MidhashCache>,
/// The Open Library binding client, built at `configure()`. `None` before
/// that — records are then simply unbound.
openlibrary: Option<OpenLibraryClient>,
/// Test hook for the Open Library API base. Never set in production.
openlibrary_base: Option<String>,
/// Shared with the `openlibrary` plugin by `main.rs`; built at
/// `configure()` when absent.
openlibrary_budget: Option<Arc<RateBudget>>,
/// The Open Library contact (see `openlibrary_client::load_contact`).
contact: Option<String>,
} }
impl GutenbergPlugin { impl GutenbergPlugin {
@@ -80,10 +140,33 @@ impl GutenbergPlugin {
Self { Self {
http, http,
base_url, base_url,
rdf_base_url: DEFAULT_RDF_BASE_URL.to_string(),
cache: None, cache: None,
openlibrary: None,
openlibrary_base: None,
openlibrary_budget: None,
contact: openlibrary_client::env_contact(),
} }
} }
/// Fetch the per-edition RDF from `rdf_base_url` instead of gutenberg.org.
pub fn with_rdf_base_url(mut self, rdf_base_url: String) -> Self {
self.rdf_base_url = rdf_base_url;
self
}
/// Send the Open Library binding lookup to `base` — tests only.
pub fn with_openlibrary_base_url(mut self, base: String) -> Self {
self.openlibrary_base = Some(base);
self
}
/// Draw binding lookups from a budget shared with the `openlibrary` plugin.
pub fn with_openlibrary_budget(mut self, budget: Arc<RateBudget>) -> Self {
self.openlibrary_budget = Some(budget);
self
}
fn cache(&self) -> Result<&MidhashCache, GatewayError> { fn cache(&self) -> Result<&MidhashCache, GatewayError> {
common::require_cache(self.cache.as_ref(), "gutenberg") common::require_cache(self.cache.as_ref(), "gutenberg")
} }
@@ -151,11 +234,13 @@ impl GutenbergPlugin {
const BIBREC_CONCURRENCY: usize = 6; const BIBREC_CONCURRENCY: usize = 6;
let cache = self.cache.clone(); let cache = self.cache.clone();
let http = self.http.clone(); let http = self.http.clone();
let rdf_base = self.rdf_base_url.trim_end_matches('/').to_string();
let ids: Vec<u64> = books.iter().map(|b| b.id).collect(); let ids: Vec<u64> = books.iter().map(|b| b.id).collect();
stream::iter(ids.into_iter()) stream::iter(ids.into_iter())
.map(|id| { .map(|id| {
let cache = cache.clone(); let cache = cache.clone();
let http = http.clone(); let http = http.clone();
let rdf_base = rdf_base.clone();
async move { async move {
let record_id = id.to_string(); let record_id = id.to_string();
if let Some(cached) = cache if let Some(cached) = cache
@@ -164,7 +249,7 @@ impl GutenbergPlugin {
{ {
return cached; return cached;
} }
let url = format!("https://www.gutenberg.org/cache/epub/{id}/pg{id}.rdf"); let url = format!("{rdf_base}/cache/epub/{id}/pg{id}.rdf");
let resp = tokio::time::timeout( let resp = tokio::time::timeout(
std::time::Duration::from_secs(8), std::time::Duration::from_secs(8),
http.get(&url).send(), http.get(&url).send(),
@@ -334,6 +419,86 @@ impl GutenbergPlugin {
.collect() .collect()
.await .await
} }
/// The Gutendex search a work page's query stands for.
///
/// meta-read's book page asks for `openlibraryid:OL…W` with no free text.
/// That used to fall through to the catalogue browse — the 32 most
/// downloaded books — and the gateway kept those bound to the work, so a
/// book outside that list never showed an edition on its own page. Searching
/// for the work's main title and its first author's surname finds it
/// wherever it ranks; the gateway still keeps only the records bound to the
/// work. `None` (no work named, Open Library off or failing) keeps the browse.
async fn work_search_text(&self, query: &GatewayQuery) -> Option<String> {
let work = first_filter(query, "openlibraryid")?.to_ascii_uppercase();
if !openlibrary_client::is_work_id(&work) {
return None;
}
let client = self.openlibrary.as_ref()?;
let lookup = client.doc_by_work_id(&work, openlibrary_client::BINDING_DEADLINE);
let doc = tokio::time::timeout(BINDING_TIMEOUT, lookup).await.ok()?.ok()??;
let title = openlibrary_client::main_title(doc.title.as_deref()?);
if title.is_empty() {
return None;
}
let author = doc.author_name.first().map(|a| openlibrary_client::surname(a)).unwrap_or_default();
Some(format!("{title} {author}").trim().to_string())
}
/// Each result's Open Library work, through the Project Gutenberg id Open
/// Library records (reading-model §2.4), or, for an etext Open Library lists
/// under no work, by title and author (`by_title`, graded Matched). Aligned
/// with the input slice; `None` means no binding — ambiguous, no title match,
/// not configured, over budget, or the lookup failed.
///
/// ⚠ **A failure never blocks or drops a record.** The binding upgrades an
/// edition that is already readable. Every failure is left uncached, so the
/// next query that surfaces the book tries again; the budget deadline is
/// short on purpose (`BINDING_DEADLINE`), so a cold browse row binds the
/// first handful of books and the cache fills the rest over later queries.
async fn fetch_openlibrary_works(&self, books: &[GutendexBook]) -> Vec<Option<Binding>> {
use futures::stream::{self, StreamExt};
const BINDING_CONCURRENCY: usize = 4;
let Some(client) = self.openlibrary.as_ref() else {
return vec![None; books.len()];
};
// Owned values, not `books.iter().map(..)`: a borrowing closure inside the
// stream makes `handle_query`'s future fail the `async_trait` lifetime
// check ("FnOnce is not general enough").
let wanted: Vec<(u64, String, String)> = books
.iter()
.map(|b| (b.id, b.title.clone(), b.authors.first().map(|a| a.name.clone()).unwrap_or_default()))
.collect();
stream::iter(wanted)
.map(|(id, title, author)| async move {
let lookup = bind(client, id, &title, &author, openlibrary_client::BINDING_DEADLINE);
match tokio::time::timeout(BINDING_TIMEOUT, lookup).await {
Ok(Ok(binding)) => binding,
Ok(Err(e)) => {
debug!(
target: "meta-share::gateway",
upstream = "gutenberg",
id,
error = %e,
"open library binding lookup failed; record left unbound"
);
None
}
Err(_) => {
debug!(
target: "meta-share::gateway",
upstream = "gutenberg",
id,
"open library binding lookup timed out; record left unbound"
);
None
}
}
})
.buffered(BINDING_CONCURRENCY)
.collect()
.await
}
} }
impl Default for GutenbergPlugin { impl Default for GutenbergPlugin {
@@ -349,7 +514,22 @@ impl FeederPlugin for GutenbergPlugin {
} }
fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError> { fn configure(&mut self, cache_dir: &Path) -> Result<(), ConfigError> {
self.cache = Some(common::open_midhash_cache(cache_dir, "gutenberg")?); let cache = common::open_midhash_cache(cache_dir, "gutenberg")?;
// The binding cache rides this plugin's own redb (the `misc` table), so
// a binding survives a restart without touching the card plugin's store.
let contact = openlibrary_client::load_contact(cache_dir);
let budget = self
.openlibrary_budget
.clone()
.unwrap_or_else(|| openlibrary_client::budget(contact.is_some()));
self.openlibrary = Some(OpenLibraryClient::new(
budget,
Some(cache.clone()),
self.openlibrary_base.clone(),
contact.as_deref(),
));
self.contact = contact;
self.cache = Some(cache);
Ok(()) Ok(())
} }
@@ -386,7 +566,16 @@ impl FeederPlugin for GutenbergPlugin {
// everything, already ordered by download count (identical to // everything, already ordered by download count (identical to
// `?sort=popular`), so the row leads with Pride and Prejudice rather than // `?sort=popular`), so the row leads with Pride and Prejudice rather than
// with whatever happens to contain punctuation. // with whatever happens to contain punctuation.
let q = query.free_text.trim(); // A work page's query names its Open Library work and no text: search
// for that work's title and author instead of browsing. See
// `work_search_text`.
let free = query.free_text.trim();
let work_text = if free.is_empty() {
self.work_search_text(query).await
} else {
None
};
let q = work_text.as_deref().unwrap_or(free);
let params: Vec<(&str, &str)> = if q.is_empty() { let params: Vec<(&str, &str)> = if q.is_empty() {
Vec::new() Vec::new()
} else { } else {
@@ -405,13 +594,19 @@ impl FeederPlugin for GutenbergPlugin {
"search returned" "search returned"
); );
let raw: Vec<GutendexBook> = body.results.into_iter().take(max_results).collect(); let raw: Vec<GutendexBook> = body.results.into_iter().take(max_results).collect();
let (cover_cids, bibrecs) = let (cover_cids, bibrecs, works) = tokio::join!(
tokio::join!(self.fetch_cover_cids(&raw), self.fetch_bibrec_fields(&raw),); self.fetch_cover_cids(&raw),
self.fetch_bibrec_fields(&raw),
self.fetch_openlibrary_works(&raw),
);
Ok(raw Ok(raw
.into_iter() .into_iter()
.zip(cover_cids) .zip(cover_cids)
.zip(bibrecs) .zip(bibrecs)
.map(|((book, cover_cid), bibrec)| into_discovery_record(book, cover_cid, bibrec)) .zip(works)
.map(|(((book, cover_cid), bibrec), work)| {
into_discovery_record(book, cover_cid, bibrec, work)
})
.collect()) .collect())
} }
@@ -438,7 +633,26 @@ impl FeederPlugin for GutenbergPlugin {
.ok() .ok()
.flatten() .flatten()
.unwrap_or_default(); .unwrap_or_default();
let record = into_discovery_record(book, cover_cid, bibrec); // Not user-facing, so the binding lookup may queue for the full search
// deadline. A failure still leaves the record unbound, never fails the
// compute.
let author = book.authors.first().map(|a| a.name.clone()).unwrap_or_default();
let work = match self.openlibrary.as_ref() {
Some(ol) => bind(ol, book.id, &book.title, &author, openlibrary_client::SEARCH_DEADLINE)
.await
.unwrap_or_else(|e| {
debug!(
target: "meta-share::gateway",
upstream = "gutenberg",
record_id,
error = %e,
"open library binding lookup failed; record left unbound"
);
None
}),
None => None,
};
let record = into_discovery_record(book, cover_cid, bibrec, work);
Ok(common::single_outcome( Ok(common::single_outcome(
cid, cid,
@@ -481,13 +695,30 @@ impl FeederPlugin for GutenbergPlugin {
fn served_content_kinds(&self) -> &'static [&'static str] { fn served_content_kinds(&self) -> &'static [&'static str] {
&["book"] &["book"]
} }
/// ⚠ Declared HERE, not only on `openlibrary`. The SDK's `/config` page
/// edits the lowest hosted upstream id, which is this one — so this is where
/// the operator sets the contact both upstreams send to Open Library.
fn config_schema(&self) -> ConfigSchema {
ConfigSchema {
fields: vec![openlibrary_client::contact_field()],
}
}
fn config_values(&self) -> serde_json::Value {
serde_json::json!({ "contact": self.contact.clone().unwrap_or_default() })
}
} }
/// Convert a `GutendexBook` into our wire-level `DiscoveryRecord`. /// Convert a `GutendexBook` into our wire-level `DiscoveryRecord`.
///
/// `openlibrary_work` is the binding from [`bind`]: the one work Open Library
/// records for this etext, or, when it records none, the title match.
fn into_discovery_record( fn into_discovery_record(
book: GutendexBook, book: GutendexBook,
cover_cid: Option<String>, cover_cid: Option<String>,
bibrec: BTreeMap<String, String>, bibrec: BTreeMap<String, String>,
openlibrary_work: Option<Binding>,
) -> DiscoveryRecord { ) -> DiscoveryRecord {
let GutendexBook { let GutendexBook {
id, id,
@@ -520,6 +751,8 @@ fn into_discovery_record(
fields.insert(upstream_id_field("gutenberg"), id.to_string()); fields.insert(upstream_id_field("gutenberg"), id.to_string());
if !authors.is_empty() { if !authors.is_empty() {
// Legacy scalar, kept verbatim for existing readers: Gutenberg's own
// "Last, First" spelling, comma-joined.
fields.insert( fields.insert(
"author".into(), "author".into(),
authors authors
@@ -529,9 +762,28 @@ fn into_discovery_record(
.join(", "), .join(", "),
); );
} }
// `authors/{name}` key-set (reading-model §3.2): one member per person, in
// display order, so one author compares equal across sources — Open Library
// and AniList both write "Jane Austen", never "Austen, Jane".
for a in &authors {
if let Some(name) = display_name(&a.name) {
fields.insert(format!("authors/{name}"), "true".into());
}
}
if let Some(lang) = languages.first() { if let Some(lang) = languages.first() {
fields.insert("language".into(), iso639_1_to_2(lang).to_string()); fields.insert("language".into(), iso639_1_to_2(lang).to_string());
} }
// `languages/{lang3}` key-set for EVERY language, in the stack's shared
// vocabulary (`fre`, where the legacy scalar above says `fra`). Two-letter
// codes only the local table knows (`la`, `ca`, `is`) are widened first;
// anything still not three letters is dropped rather than written as a
// member no `languages:` filter can match.
for lang in &languages {
let code = normalize_lang_code(iso639_1_to_2(&lang.trim().to_ascii_lowercase()));
if code.len() == 3 && code.bytes().all(|b| b.is_ascii_lowercase()) {
fields.insert(format!("languages/{code}"), "true".into());
}
}
if !subjects.is_empty() { if !subjects.is_empty() {
fields.insert("subjects".into(), subjects.join(", ")); fields.insert("subjects".into(), subjects.join(", "));
} }
@@ -619,10 +871,46 @@ fn into_discovery_record(
let _ = download_count; let _ = download_count;
fields.insert("fileName".into(), format!("gutenberg-{id}.epub")); fields.insert("fileName".into(), format!("gutenberg-{id}.epub"));
// The work binding and its trust claim (reading-model §2.4). On an
// identifier link Open Library itself records, the claim is the plain
// Anchored form: `anchored` + `anchorSource`, and NO `anchorMethod` — that
// marks a title match. The
// gateway transcribes the claim into `anchoredBy/openlibrary:OL…W` and
// strips the transient keys before storing.
//
// A title binding (an etext Open Library lists under no work) carries the
// same claim plus `anchorMethod=title`: the gateway then records
// `matchedBy/openlibrary:OL…W` instead of `anchoredBy/`, and meta-read grades
// the edition Matched — listed on its work's page, ranked after any
// identifier-bound edition.
if let Some(binding) = openlibrary_work.filter(|b| openlibrary_client::is_work_id(&b.work)) {
fields.insert("openlibraryid".into(), binding.work);
fields.insert("anchored".into(), "true".into());
fields.insert("anchorSource".into(), "openlibrary".into());
if binding.by_title {
fields.insert("anchorMethod".into(), "title".into());
}
}
for (k, v) in bibrec { for (k, v) in bibrec {
fields.entry(k).or_insert(v); fields.entry(k).or_insert(v);
} }
// Registry spellings beside the legacy RDF keys, derived after the merge so
// a bibrec cached before these keys existed produces them too.
//
// `description/eng` from `pgterms:marc520` (kept as `summary`): Gutenberg's
// summaries are written in English whatever the book's language, and
// `description/{lang3}` is the form every card and client reads.
if let Some(summary) = fields.get("summary").cloned() {
fields.entry("description/eng".into()).or_insert(summary);
}
// `releasedate` (iso-date) beside the legacy camelCase `releaseDate`, only
// when the RDF value really is one.
if let Some(date) = fields.get("releaseDate").and_then(|d| iso_date(d)) {
fields.entry("releasedate".into()).or_insert(date);
}
DiscoveryRecord { DiscoveryRecord {
upstream_id: "gutenberg".into(), upstream_id: "gutenberg".into(),
record_id, record_id,
@@ -630,6 +918,38 @@ fn into_discovery_record(
} }
} }
/// Gutenberg's "Last, First" → "First Last", the display order every other
/// writer of `authors/{name}` uses.
///
/// Only when there is **exactly one** comma. "King, Martin Luther, Jr." or a
/// corporate name has no safe reordering, and a wrong one is worse than
/// Gutenberg's own spelling, which is kept as is.
fn display_name(raw: &str) -> Option<String> {
let name = raw.trim();
if name.is_empty() {
return None;
}
match name.split(',').map(str::trim).collect::<Vec<_>>().as_slice() {
[last, first] if !last.is_empty() && !first.is_empty() => Some(format!("{first} {last}")),
_ => Some(name.to_string()),
}
}
/// `raw` when it is an `iso-date` (`YYYY-MM-DD`, or a bare `YYYY`), else `None`.
///
/// Never padded: "1998" does not become "1998-01-01", which would claim a
/// precision the RDF never had.
fn iso_date(raw: &str) -> Option<String> {
let digits = |s: &str, n: usize| s.len() == n && s.bytes().all(|b| b.is_ascii_digit());
let t = raw.trim();
let ok = match t.split('-').collect::<Vec<_>>().as_slice() {
[y] => digits(y, 4),
[y, m, d] => digits(y, 4) && digits(m, 2) && digits(d, 2),
_ => false,
};
ok.then(|| t.to_string())
}
/// Parse a Project Gutenberg `pg{id}.rdf` document for per-edition metadata. /// Parse a Project Gutenberg `pg{id}.rdf` document for per-edition metadata.
fn parse_pgterms_rdf(xml: &str) -> BTreeMap<String, String> { fn parse_pgterms_rdf(xml: &str) -> BTreeMap<String, String> {
let mut out = BTreeMap::new(); let mut out = BTreeMap::new();
@@ -762,6 +1082,28 @@ mod rdf_parser_tests {
let f = parse_pgterms_rdf("<html><body>not rdf</body></html>"); let f = parse_pgterms_rdf("<html><body>not rdf</body></html>");
assert!(f.is_empty()); assert!(f.is_empty());
} }
/// The legacy RDF keys stay; the registry spellings are derived beside them
/// on the record — including from a bibrec cached before those keys
/// existed, which holds only the legacy ones.
#[test]
fn summary_and_issued_date_also_arrive_under_registry_keys() {
let xml = r#"<rdf:RDF>
<dcterms:issued rdf:datatype="...">1998-06-01</dcterms:issued>
<pgterms:marc520>A classic novel of manners.</pgterms:marc520>
</rdf:RDF>"#;
let book: GutendexBook = serde_json::from_value(serde_json::json!({
"id": 1342,
"title": "Pride and Prejudice"
}))
.unwrap();
let r = into_discovery_record(book, None, parse_pgterms_rdf(xml), None);
let get = |k: &str| r.fields.get(k).map(String::as_str);
assert_eq!(get("summary"), Some("A classic novel of manners."));
assert_eq!(get("description/eng"), Some("A classic novel of manners."));
assert_eq!(get("releaseDate"), Some("1998-06-01"));
assert_eq!(get("releasedate"), Some("1998-06-01"));
}
} }
/// ISO 639-1 (two-letter) → ISO 639-2/T (three-letter) for the ~20 most common /// ISO 639-1 (two-letter) → ISO 639-2/T (three-letter) for the ~20 most common
@@ -877,13 +1219,27 @@ mod tests {
}) })
} }
/// Every upstream — Gutendex, the RDF host and Open Library — on the one mock
/// server, so no test reaches the internet. An unmounted path 404s, which
/// every optional lookup treats as "nothing there".
fn configured_plugin_against(server: &MockServer) -> (GutenbergPlugin, tempfile::TempDir) { fn configured_plugin_against(server: &MockServer) -> (GutenbergPlugin, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let mut plugin = GutenbergPlugin::with_base_url(server.uri()); let mut plugin = GutenbergPlugin::with_base_url(server.uri())
.with_rdf_base_url(server.uri())
.with_openlibrary_base_url(server.uri())
.with_openlibrary_budget(RateBudget::new(100.0, 100.0));
plugin.configure(dir.path()).expect("configure"); plugin.configure(dir.path()).expect("configure");
(plugin, dir) (plugin, dir)
} }
/// Open Library's answer to `q=id_project_gutenberg:<id>`.
fn binding(id: u64, body: serde_json::Value) -> Mock {
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", format!("id_project_gutenberg:{id}")))
.respond_with(ResponseTemplate::new(200).set_body_json(body))
}
/// ⚠ The search path must ask for the CANONICAL `/books/`. /// ⚠ The search path must ask for the CANONICAL `/books/`.
/// ///
/// Gutendex 301s `/books` → `/books/`. Following a redirect is free in /// Gutendex 301s `/books` → `/books/`. Following a redirect is free in
@@ -1086,7 +1442,16 @@ mod tests {
Some("https://www.gutenberg.org/ebooks/1342") Some("https://www.gutenberg.org/ebooks/1342")
); );
assert_eq!(r0.fields.get("author").map(String::as_str), Some("Austen, Jane")); assert_eq!(r0.fields.get("author").map(String::as_str), Some("Austen, Jane"));
assert_eq!(
r0.fields.get("authors/Jane Austen").map(String::as_str),
Some("true")
);
assert!(!r0.fields.contains_key("authors/Austen, Jane"));
assert_eq!(r0.fields.get("language").map(String::as_str), Some("eng")); assert_eq!(r0.fields.get("language").map(String::as_str), Some("eng"));
assert_eq!(r0.fields.get("languages/eng").map(String::as_str), Some("true"));
// No Open Library answer mounted: the record is returned, unbound.
assert!(!r0.fields.contains_key("openlibraryid"));
assert!(!r0.fields.contains_key("anchored"));
assert_eq!(r0.fields.get("format").map(String::as_str), Some("epub")); assert_eq!(r0.fields.get("format").map(String::as_str), Some("epub"));
assert_eq!( assert_eq!(
r0.fields.get("fileName").map(String::as_str), r0.fields.get("fileName").map(String::as_str),
@@ -1261,6 +1626,303 @@ mod tests {
} }
} }
#[test]
fn display_name_reorders_only_a_single_comma() {
assert_eq!(display_name("Austen, Jane").as_deref(), Some("Jane Austen"));
assert_eq!(
display_name("Shelley, Mary Wollstonecraft").as_deref(),
Some("Mary Wollstonecraft Shelley")
);
assert_eq!(
display_name("King, Martin Luther, Jr.").as_deref(),
Some("King, Martin Luther, Jr.")
);
assert_eq!(display_name("Homer").as_deref(), Some("Homer"));
assert_eq!(display_name(" "), None);
}
#[test]
fn every_language_becomes_a_lang3_key_set_member() {
let book: GutendexBook = serde_json::from_value(serde_json::json!({
"id": 1,
"title": "Polyglot",
"languages": ["en", "fr", "la", "xx"]
}))
.unwrap();
let r = into_discovery_record(book, None, BTreeMap::new(), None);
let members: Vec<&str> = r
.fields
.keys()
.filter_map(|k| k.strip_prefix("languages/"))
.collect();
// The SDK vocabulary (`fre`), every language, and nothing that is not
// three letters.
assert_eq!(members, vec!["eng", "fre", "lat"]);
// The legacy scalar is untouched.
assert_eq!(r.fields.get("language").map(String::as_str), Some("eng"));
}
#[test]
fn releasedate_is_written_only_for_an_iso_date() {
assert_eq!(iso_date("2008-06-27").as_deref(), Some("2008-06-27"));
assert_eq!(iso_date("1998").as_deref(), Some("1998"));
assert_eq!(iso_date("1998-06"), None);
assert_eq!(iso_date("June 1998"), None);
}
/// The RDF host is configurable, so the per-edition fields are testable
/// without gutenberg.org.
#[tokio::test]
async fn rdf_summary_and_issue_date_arrive_through_the_configured_rdf_host() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&server)
.await;
Mock::given(method("GET"))
.and(path("/cache/epub/1342/pg1342.rdf"))
.respond_with(ResponseTemplate::new(200).set_body_string(
r#"<rdf:RDF><pgterms:ebook rdf:about="ebooks/1342">
<dcterms:issued rdf:datatype="http://www.w3.org/2001/XMLSchema#date">1998-06-01</dcterms:issued>
<pgterms:marc520>"Pride and Prejudice" is a classic novel.</pgterms:marc520>
</pgterms:ebook></rdf:RDF>"#,
))
.expect(1)
.mount(&server)
.await;
let (plugin, _dir) = configured_plugin_against(&server);
let records = plugin
.handle_query(&GatewayQuery::from_free_text("alice"), 50)
.await
.expect("handle_query");
let pride = &records[0].fields;
assert!(pride["description/eng"].contains("classic novel"));
assert_eq!(pride.get("releasedate").map(String::as_str), Some("1998-06-01"));
// No RDF mounted for 84: neither key, and the record still arrives.
assert!(!records[1].fields.contains_key("description/eng"));
assert!(!records[1].fields.contains_key("releasedate"));
server.verify().await;
}
#[tokio::test]
async fn exactly_one_open_library_work_binds_the_edition_with_an_anchored_claim() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&server)
.await;
binding(
1342,
serde_json::json!({ "numFound": 1, "docs": [
{ "key": "/works/OL66554W", "id_project_gutenberg": ["1342", "42671", "45186"] }
]}),
)
.mount(&server)
.await;
// Two works list etext 84: an unmerged duplicate, so no binding at all.
binding(
84,
serde_json::json!({ "numFound": 2, "docs": [
{ "key": "/works/OL450063W", "id_project_gutenberg": ["84"] },
{ "key": "/works/OL18388W", "id_project_gutenberg": ["84"] }
]}),
)
.mount(&server)
.await;
let (plugin, _dir) = configured_plugin_against(&server);
let records = plugin
.handle_query(&GatewayQuery::from_free_text("alice"), 50)
.await
.expect("handle_query");
assert_eq!(records.len(), 2);
let pride = &records[0].fields;
assert_eq!(pride.get("openlibraryid").map(String::as_str), Some("OL66554W"));
assert_eq!(pride.get("anchored").map(String::as_str), Some("true"));
assert_eq!(pride.get("anchorSource").map(String::as_str), Some("openlibrary"));
assert!(
!pride.contains_key("anchorMethod"),
"an identifier link Open Library records is not a title match"
);
let frankenstein = &records[1].fields;
for k in ["openlibraryid", "anchored", "anchorSource", "anchorMethod"] {
assert!(!frankenstein.contains_key(k), "ambiguous binding must not stamp {k}");
}
}
/// Open Library's answer to `title=…&author=…` for Frankenstein: several
/// works by Mary Shelley share the title; the most published is the novel.
fn frankenstein_by_title() -> Mock {
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("title", "frankenstein"))
.and(query_param("author", "shelley"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "numFound": 3, "docs": [
{ "key": "/works/OL26860869W", "title": "Frankenstein", "author_name": ["Mary Shelley"], "edition_count": 84 },
{ "key": "/works/OL450063W", "title": "Frankenstein; or, The Modern Prometheus", "author_name": ["Mary Shelley"], "edition_count": 2187 },
{ "key": "/works/OL25595002W", "title": "Mary Shelley's Frankenstein", "author_name": ["Mary Shelley"], "edition_count": 90 }
]})))
}
/// ⚠ Open Library lists etext 84 under no work, so the identifier rule alone
/// left the most read novel on Gutenberg unbound and its book page empty.
#[tokio::test]
async fn an_etext_no_work_lists_is_bound_by_title_and_graded_matched() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&server)
.await;
binding(
1342,
serde_json::json!({ "numFound": 1, "docs": [
{ "key": "/works/OL66554W", "id_project_gutenberg": ["1342"] }
]}),
)
.mount(&server)
.await;
binding(84, serde_json::json!({ "numFound": 0, "docs": [] }))
.mount(&server)
.await;
frankenstein_by_title().expect(1).mount(&server).await;
let (plugin, _dir) = configured_plugin_against(&server);
let records = plugin
.handle_query(&GatewayQuery::from_free_text("alice"), 50)
.await
.expect("handle_query");
let pride = &records[0].fields;
assert_eq!(pride.get("openlibraryid").map(String::as_str), Some("OL66554W"));
assert!(!pride.contains_key("anchorMethod"), "an identifier binding stays Anchored");
let frankenstein = &records[1].fields;
assert_eq!(frankenstein.get("openlibraryid").map(String::as_str), Some("OL450063W"));
assert_eq!(frankenstein.get("anchored").map(String::as_str), Some("true"));
assert_eq!(frankenstein.get("anchorSource").map(String::as_str), Some("openlibrary"));
assert_eq!(frankenstein.get("anchorMethod").map(String::as_str), Some("title"));
}
/// ⚠ A work page asks for its work and no text. That used to browse the 32
/// most downloaded books, so a book outside them never reached its own page.
#[tokio::test]
async fn a_work_page_query_searches_for_the_works_title_and_author() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", "key:/works/OL450063W"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ "numFound": 1, "docs": [
{ "key": "/works/OL450063W", "title": "Frankenstein; or, The Modern Prometheus", "author_name": ["Mary Shelley"] }
]})))
.mount(&server)
.await;
Mock::given(method("GET"))
.and(path("/books/"))
.and(query_param("search", "frankenstein shelley"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.expect(1)
.mount(&server)
.await;
binding(1342, serde_json::json!({ "numFound": 0, "docs": [] }))
.mount(&server)
.await;
binding(84, serde_json::json!({ "numFound": 0, "docs": [] }))
.mount(&server)
.await;
frankenstein_by_title().mount(&server).await;
let (plugin, _dir) = configured_plugin_against(&server);
let mut query = GatewayQuery::from_free_text("");
query.filters.insert("openlibraryid".into(), vec!["OL450063W".into()]);
query.filters.insert("contentKind".into(), vec!["book".into()]);
let records = plugin.handle_query(&query, 50).await.expect("handle_query");
let frankenstein = records.iter().find(|r| r.record_id == "84").expect("etext 84 is found");
assert_eq!(frankenstein.fields.get("openlibraryid").map(String::as_str), Some("OL450063W"));
assert_eq!(frankenstein.fields.get("anchorMethod").map(String::as_str), Some("title"));
}
/// ⚠ A binding failure never costs the record.
#[tokio::test]
async fn no_work_or_a_failed_lookup_leaves_the_record_unbound_but_present() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&server)
.await;
binding(1342, serde_json::json!({ "numFound": 0, "docs": [] }))
.mount(&server)
.await;
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", "id_project_gutenberg:84"))
.respond_with(ResponseTemplate::new(500))
.mount(&server)
.await;
let (plugin, _dir) = configured_plugin_against(&server);
let records = plugin
.handle_query(&GatewayQuery::from_free_text("alice"), 50)
.await
.expect("an Open Library failure must not fail the query");
assert_eq!(records.len(), 2);
for r in &records {
assert!(!r.fields.contains_key("openlibraryid"));
assert!(!r.fields.contains_key("anchored"));
}
}
/// A book surfaced by every browse must not cost a lookup every browse.
#[tokio::test]
async fn a_binding_answer_is_looked_up_once_per_book() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&server)
.await;
binding(
1342,
serde_json::json!({ "numFound": 1, "docs": [
{ "key": "/works/OL66554W", "id_project_gutenberg": ["1342"] }
]}),
)
.expect(1)
.mount(&server)
.await;
binding(84, serde_json::json!({ "numFound": 0, "docs": [] }))
.expect(1)
.mount(&server)
.await;
let (plugin, _dir) = configured_plugin_against(&server);
for _ in 0..2 {
let records = plugin
.handle_query(&GatewayQuery::from_free_text("alice"), 50)
.await
.expect("handle_query");
assert_eq!(
records[0].fields.get("openlibraryid").map(String::as_str),
Some("OL66554W")
);
}
server.verify().await;
}
/// The SDK's config page shows the lowest upstream id — this one — so the
/// Open Library contact must be settable here.
#[test]
fn the_open_library_contact_is_configurable_on_this_upstream() {
let schema = GutenbergPlugin::new().config_schema();
assert!(schema.fields.iter().any(|f| f.key == "contact"));
}
#[test] #[test]
fn health_reflects_configure_state() { fn health_reflects_configure_state() {
let plugin = GutenbergPlugin::new(); let plugin = GutenbergPlugin::new();
+11 -3
View File
@@ -1,5 +1,13 @@
//! `book-feeder` library surface — exposes the Gutenberg plugin so integration //! `book-feeder` library surface — exposes the plugins so integration tests
//! tests (and, later, other binaries) can construct it. The binary //! (and, later, other binaries) can construct them. The binary (`main.rs`) is a
//! (`main.rs`) is a thin wrapper over [`meta_feeder_sdk::serve_feeders`]. //! thin wrapper over [`meta_feeder_sdk::serve_feeders`].
//!
//! Two upstreams:
//! - [`gutenberg`] — Project Gutenberg editions, the byte tier (`document` × `book`);
//! - [`openlibrary`] — Open Library work cards, the identity tier (`card` × `book`),
//! over the client in [`openlibrary_client`], which `gutenberg` also uses to
//! bind an edition to its work.
pub mod gutenberg; pub mod gutenberg;
pub mod openlibrary;
pub mod openlibrary_client;
+26 -5
View File
@@ -1,17 +1,27 @@
//! `book-feeder` — Project Gutenberg feeder sidecar. //! `book-feeder` — the book tier's feeder sidecar.
//! //!
//! Hosts the [`gutenberg::GutenbergPlugin`] over the feeder HTTP contract via //! Hosts two plugins over the feeder HTTP contract via
//! [`meta_feeder_sdk::serve_feeders`]. The gateway core reaches it server-to- //! [`meta_feeder_sdk::serve_feeders`]:
//! server on the internal docker network. //! - [`GutenbergPlugin`] — Project Gutenberg editions (`document` × `book`);
//! - [`OpenLibraryPlugin`] — Open Library work cards (`card` × `book`).
//!
//! The gateway core reaches it server-to-server on the internal docker network.
//! //!
//! Env: //! Env:
//! - `META_FEEDER_HTTP_LISTEN` — listen addr (default `0.0.0.0:8080`) //! - `META_FEEDER_HTTP_LISTEN` — listen addr (default `0.0.0.0:8080`)
//! - `META_FEEDER_STATE_DIR` — per-plugin cache root (default `/data/meta-feeder`) //! - `META_FEEDER_STATE_DIR` — per-plugin cache root (default `/data/meta-feeder`)
//! - `OPENLIBRARY_CONTACT` — e-mail/URL sent in the Open Library User-Agent;
//! first-boot seed for the `contact` config field (the saved field wins)
//! - `OPENLIBRARY_TOP_N`, `OPENLIBRARY_DISCOVERY_N` — optional card result sizing
//! - `RUST_LOG` — tracing filter (default `info`) //! - `RUST_LOG` — tracing filter (default `info`)
use std::net::SocketAddr; use std::net::SocketAddr;
use std::path::Path;
use book_feeder::gutenberg::GutenbergPlugin; use book_feeder::gutenberg::GutenbergPlugin;
use book_feeder::openlibrary::OpenLibraryPlugin;
use book_feeder::openlibrary_client;
use meta_feeder_sdk::plugin::FeederPlugin;
use meta_feeder_sdk::serve_feeders; use meta_feeder_sdk::serve_feeders;
use tracing_subscriber::EnvFilter; use tracing_subscriber::EnvFilter;
@@ -27,5 +37,16 @@ async fn main() -> anyhow::Result<()> {
let state_dir = let state_dir =
std::env::var("META_FEEDER_STATE_DIR").unwrap_or_else(|_| "/data/meta-feeder".to_string()); std::env::var("META_FEEDER_STATE_DIR").unwrap_or_else(|_| "/data/meta-feeder".to_string());
serve_feeders(vec![Box::new(GutenbergPlugin::new())], state_dir, listen).await // ONE budget for both plugins: the Gutenberg → Open Library binding and the
// card plugin call the same upstream, which rate-limits per client, not per
// plugin. Two buckets would each think they had the whole allowance.
let contact =
openlibrary_client::load_contact(&Path::new(&state_dir).join("gateway").join("openlibrary"));
let budget = openlibrary_client::budget(contact.is_some());
let plugins: Vec<Box<dyn FeederPlugin>> = vec![
Box::new(GutenbergPlugin::new().with_openlibrary_budget(budget.clone())),
Box::new(OpenLibraryPlugin::new().with_budget(budget)),
];
serve_feeders(plugins, state_dir, listen).await
} }
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+165 -21
View File
@@ -1,24 +1,36 @@
//! End-to-end contract test for the feeder HTTP surface: boots the real //! End-to-end contract test for the feeder HTTP surface: boots the real
//! `serve_feeders` router (book-feeder hosting GutenbergPlugin) on an ephemeral //! `serve_feeders` router (book-feeder hosting GutenbergPlugin AND
//! port, pointed at a wiremock Gutendex, and drives `/manifest`, `/query`, and //! OpenLibraryPlugin, as `main.rs` does) on an ephemeral port, pointed at one
//! `/compute` over HTTP. Deterministic — no live Gutendex. //! wiremock server that plays Gutendex, the Gutenberg RDF host and Open Library,
//! and drives `/manifest`, `/query`, and `/compute` over HTTP. Deterministic —
//! no live upstream.
use book_feeder::gutenberg::GutenbergPlugin; use book_feeder::gutenberg::GutenbergPlugin;
use book_feeder::openlibrary::OpenLibraryPlugin;
use book_feeder::openlibrary_client;
use meta_feeder_sdk::{ use meta_feeder_sdk::{
configure_plugins, hash, router, ComputeRequest, ComputeResponse, GatewayQuery, HashKindDto, configure_plugins, hash, router, ComputeRequest, ComputeResponse, FeederPlugin, GatewayQuery,
ManifestResponse, QueryRequest, QueryResponse, HashKindDto, ManifestResponse, QueryRequest, QueryResponse,
}; };
use wiremock::matchers::{method, path}; use wiremock::matchers::{method, path, query_param};
use wiremock::{Mock, MockServer, ResponseTemplate}; use wiremock::{Mock, MockServer, ResponseTemplate};
/// Spawn the feeder router (Gutenberg pointed at `upstream`) on an ephemeral /// Spawn the feeder router (every upstream pointed at `upstream`) on an
/// port; return its base URL. The TempDir is returned so the per-plugin cache /// ephemeral port; return its base URL. The TempDir is returned so the
/// outlives the test. /// per-plugin cache outlives the test.
async fn spawn_feeder(upstream: &MockServer) -> (String, tempfile::TempDir) { async fn spawn_feeder(upstream: &MockServer) -> (String, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("tempdir"); let dir = tempfile::tempdir().expect("tempdir");
let plugin = GutenbergPlugin::with_base_url(upstream.uri()); let budget = openlibrary_client::budget(false);
let plugins = let plugins: Vec<Box<dyn FeederPlugin>> = vec![
configure_plugins(vec![Box::new(plugin)], dir.path()).expect("configure plugins"); Box::new(
GutenbergPlugin::with_base_url(upstream.uri())
.with_rdf_base_url(upstream.uri())
.with_openlibrary_base_url(upstream.uri())
.with_openlibrary_budget(budget.clone()),
),
Box::new(OpenLibraryPlugin::with_api_base(upstream.uri()).with_budget(budget)),
];
let plugins = configure_plugins(plugins, dir.path()).expect("configure plugins");
let app = router(plugins, "book-feeder-test".to_string(), dir.path()); let app = router(plugins, "book-feeder-test".to_string(), dir.path());
let listener = tokio::net::TcpListener::bind("127.0.0.1:0") let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
@@ -44,7 +56,7 @@ fn books_json() -> serde_json::Value {
} }
#[tokio::test] #[tokio::test]
async fn manifest_lists_book_capabilities() { async fn manifest_lists_both_upstreams() {
let upstream = MockServer::start().await; let upstream = MockServer::start().await;
let (base, _dir) = spawn_feeder(&upstream).await; let (base, _dir) = spawn_feeder(&upstream).await;
@@ -57,21 +69,36 @@ async fn manifest_lists_book_capabilities() {
.await .await
.expect("decode manifest"); .expect("decode manifest");
assert_eq!(manifest.plugins.len(), 1); assert_eq!(manifest.plugins.len(), 2);
let p = &manifest.plugins[0]; let gutenberg = manifest.plugins.iter().find(|p| p.id == "gutenberg").expect("gutenberg");
assert_eq!(p.id, "gutenberg"); assert_eq!(gutenberg.served_file_types, vec!["document".to_string()]);
assert_eq!(p.served_file_types, vec!["document".to_string()]); assert_eq!(gutenberg.served_content_kinds, vec!["book".to_string()]);
assert_eq!(p.served_content_kinds, vec!["book".to_string()]); let openlibrary = manifest
.plugins
.iter()
.find(|p| p.id == "openlibrary")
.expect("openlibrary");
assert_eq!(openlibrary.served_file_types, vec!["card".to_string()]);
assert_eq!(openlibrary.served_content_kinds, vec!["book".to_string()]);
} }
#[tokio::test] #[tokio::test]
async fn query_returns_gutenberg_records() { async fn query_returns_gutenberg_records_with_their_open_library_claim() {
let upstream = MockServer::start().await; let upstream = MockServer::start().await;
Mock::given(method("GET")) Mock::given(method("GET"))
.and(path("/books/")) .and(path("/books/"))
.respond_with(ResponseTemplate::new(200).set_body_json(books_json())) .respond_with(ResponseTemplate::new(200).set_body_json(books_json()))
.mount(&upstream) .mount(&upstream)
.await; .await;
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", "id_project_gutenberg:1342"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"numFound": 1,
"docs": [{ "key": "/works/OL66554W", "id_project_gutenberg": ["1342", "42671"] }]
})))
.mount(&upstream)
.await;
let (base, _dir) = spawn_feeder(&upstream).await; let (base, _dir) = spawn_feeder(&upstream).await;
let resp: QueryResponse = reqwest::Client::new() let resp: QueryResponse = reqwest::Client::new()
@@ -90,9 +117,75 @@ async fn query_returns_gutenberg_records() {
assert_eq!(resp.records.len(), 1); assert_eq!(resp.records.len(), 1);
let r = &resp.records[0]; let r = &resp.records[0];
let f = |k: &str| r.fields.get(k).map(String::as_str);
assert_eq!(r.upstream_id, "gutenberg"); assert_eq!(r.upstream_id, "gutenberg");
assert_eq!(r.fields.get("gutenbergid").map(String::as_str), Some("1342")); assert_eq!(f("gutenbergid"), Some("1342"));
assert_eq!(r.fields.get("contentKind").map(String::as_str), Some("book")); assert_eq!(f("contentKind"), Some("book"));
assert_eq!(f("authors/Jane Austen"), Some("true"));
assert_eq!(f("languages/eng"), Some("true"));
// The transient claim the gateway transcribes into
// `anchoredBy/openlibrary:OL66554W`.
assert_eq!(f("openlibraryid"), Some("OL66554W"));
assert_eq!(f("anchored"), Some("true"));
assert_eq!(f("anchorSource"), Some("openlibrary"));
assert_eq!(f("anchorMethod"), None, "an identifier link is not a title match");
}
#[tokio::test]
async fn query_returns_an_open_library_card() {
let upstream = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", "pride"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"numFound": 1,
"docs": [{
"key": "/works/OL66554W",
"title": "Pride and Prejudice",
"author_name": ["Jane Austen"],
"cover_i": 14348537,
"first_publish_year": 1813
}]
})))
.mount(&upstream)
.await;
Mock::given(method("GET"))
.and(path("/works/OL66554W.json"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"key": "/works/OL66554W",
"description": "A novel of manners."
})))
.mount(&upstream)
.await;
let (base, _dir) = spawn_feeder(&upstream).await;
let mut query = GatewayQuery::from_free_text("pride");
query.filters.insert("fileType".into(), vec!["card".into()]);
let resp: QueryResponse = reqwest::Client::new()
.post(format!("{base}/query"))
.json(&QueryRequest {
upstream_id: "openlibrary".to_string(),
query,
max_results: 10,
})
.send()
.await
.expect("POST /query")
.json()
.await
.expect("decode query response");
assert_eq!(resp.records.len(), 1);
let r = &resp.records[0];
assert_eq!(r.upstream_id, "openlibrary");
assert_eq!(r.record_id, "OL66554W");
assert_eq!(r.fields.get("fileType").map(String::as_str), Some("card"));
assert_eq!(
r.fields
.get("cids/bagdsaaaubnxxazlonruwe4tboj4u6tbwgy2tkncx")
.map(String::as_str),
Some("true")
);
} }
#[tokio::test] #[tokio::test]
@@ -151,6 +244,57 @@ async fn compute_returns_outcome_with_inline_epub_bytes() {
assert_eq!(hash::compute_ipfs_cid(&decoded), expected_cid); assert_eq!(hash::compute_ipfs_cid(&decoded), expected_cid);
} }
#[tokio::test]
async fn compute_for_an_open_library_card_is_a_byteless_card_locator() {
let upstream = MockServer::start().await;
Mock::given(method("GET"))
.and(path("/search.json"))
.and(query_param("q", "key:/works/OL45804W"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"numFound": 1,
"docs": [{
"key": "/works/OL45804W",
"title": "Fantastic Mr Fox",
"author_name": ["Roald Dahl"],
"cover_i": 6498519,
"first_publish_year": 1970
}]
})))
.mount(&upstream)
.await;
Mock::given(method("GET"))
.and(path("/works/OL45804W.json"))
.respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({
"key": "/works/OL45804W",
"description": "A clever fox outwits three farmers."
})))
.mount(&upstream)
.await;
let (base, _dir) = spawn_feeder(&upstream).await;
let resp: ComputeResponse = reqwest::Client::new()
.post(format!("{base}/compute"))
.json(&ComputeRequest {
upstream_id: "openlibrary".to_string(),
record_id: "OL45804W".to_string(),
})
.send()
.await
.expect("POST /compute")
.json()
.await
.expect("decode compute response");
assert_eq!(resp.outcomes.len(), 1);
let o = &resp.outcomes[0];
assert_eq!(o.hash, "bagdsaaaubnxxazlonruwe4tboj4u6tbugu4dancx");
assert_eq!(o.hash_kind, HashKindDto::CardLocator);
assert!(o.bytes_b64.is_none());
assert!(o.file_extension.is_none());
let record = o.record.as_ref().expect("the card record rides the outcome");
assert_eq!(record.fields.get("title").map(String::as_str), Some("Fantastic Mr Fox"));
}
#[tokio::test] #[tokio::test]
async fn query_unknown_upstream_is_404() { async fn query_unknown_upstream_is_404() {
let upstream = MockServer::start().await; let upstream = MockServer::start().await;