meta-feeder-arxiv

12 Commits 1 Branch 5 Tags

3 Commits

Author	SHA1	Message	Date
Manielle	dc8a97c53f	CI accepts four-part tags (v1.2.3.1) for fixes Publish image / publish (push) Failing after 53s Details	2026-10-25 14:22:13 +01:00
Manielle	ebc7e1da65	give every record a url-locator CID so the gateway can persist it Publish image / publish (push) Failing after 13s Details A record with no CID cannot be persisted by the gateway ("search-persist: record carries no CID; not persisted"), yet the gateway still stamps the query as covered for an hour (META_GATEWAY_SEARCH_COVERAGE_TTL_SECS) and, inside that window, skips this feeder and answers from what it persisted. So the first query returned papers and every identical one for the next hour returned none. Measured on the live box: 40 hits, then 0 hits in 1.5 s with no fan-out. Each record now carries exactly one `cids/` member: a 0x1006 url-locator over the canonical, versioned PDF URL (https://arxiv.org/pdf/<id>.pdf). This is the fleet convention (Gutenberg, internetarchive, jamendo): it is a pure function of the URL, so it costs no I/O, and it doubles as the byte path that meta-share redeems. The versioned id means v2 of a paper is a different CID. Built from the canonical URL rather than the feed's link, which arXiv emits as http:// without .pdf. Verified live after deploying: 40 hits each carrying a CID; the same query again inside its coverage window returns 40 (was 0); meta-share serves the locator with 206 Partial Content and the bytes are a real PDF (%PDF-1.7, 1,182,901 B).	2026-10-11 17:02:17 +02:00
Manielle	92b2f8aa0d	browse the newest papers when a query has no keywords A query with only filters (a consumer's `domain:science contentKind:paper` row) reached the plugin with no free text, and the plugin substituted `all:`. arXiv answers that with HTTP 500, so every such query returned nothing, however many papers existed. An empty query (or the SDK's `` placeholder) now browses the major subject archives (`cat:<archive>.`, OR-ed) newest first (`sortBy=submittedDate`). Keyword queries are unchanged: same `all:` field, same URL, relevance order. The URL is built by a pure function, so both shapes are tested without the network. Every form was verified against the live API: `all:` -> 500, the browse query -> 200 with 60 entries out of ~2.6M papers. No `licence` is written, on purpose: arXiv declares none per paper, its default licence does not permit redistribution, and the registry says absence means "unknown", never "unrestricted".	2026-10-11 16:34:00 +02:00

Author

SHA1

Message

Date

Manielle

dc8a97c53f

CI accepts four-part tags (v1.2.3.1) for fixes

Publish image / publish (push) Failing after 53s

Details

2026-10-25 14:22:13 +01:00

Manielle

ebc7e1da65

give every record a url-locator CID so the gateway can persist it

Publish image / publish (push) Failing after 13s

Details

A record with no CID cannot be persisted by the gateway ("search-persist: record
carries no CID; not persisted"), yet the gateway still stamps the query as
covered for an hour (META_GATEWAY_SEARCH_COVERAGE_TTL_SECS) and, inside that
window, skips this feeder and answers from what it persisted. So the first query
returned papers and every identical one for the next hour returned none.
Measured on the live box: 40 hits, then 0 hits in 1.5 s with no fan-out.

Each record now carries exactly one `cids/` member: a 0x1006 url-locator over the
canonical, versioned PDF URL (https://arxiv.org/pdf/<id>.pdf). This is the fleet
convention (Gutenberg, internetarchive, jamendo): it is a pure function of the
URL, so it costs no I/O, and it doubles as the byte path that meta-share
redeems. The versioned id means v2 of a paper is a different CID. Built from the
canonical URL rather than the feed's link, which arXiv emits as http:// without
.pdf.

Verified live after deploying: 40 hits each carrying a CID; the same query again
inside its coverage window returns 40 (was 0); meta-share serves the locator
with 206 Partial Content and the bytes are a real PDF (%PDF-1.7, 1,182,901 B).

2026-10-11 17:02:17 +02:00

Manielle

92b2f8aa0d

browse the newest papers when a query has no keywords

A query with only filters (a consumer's `domain:science contentKind:paper` row)
reached the plugin with no free text, and the plugin substituted `all:*`. arXiv
answers that with HTTP 500, so every such query returned nothing, however many
papers existed.

An empty query (or the SDK's `*` placeholder) now browses the major subject
archives (`cat:<archive>.*`, OR-ed) newest first (`sortBy=submittedDate`).
Keyword queries are unchanged: same `all:` field, same URL, relevance order.

The URL is built by a pure function, so both shapes are tested without the
network. Every form was verified against the live API: `all:*` -> 500, the
browse query -> 200 with 60 entries out of ~2.6M papers.

No `licence` is written, on purpose: arXiv declares none per paper, its default
licence does not permit redistribution, and the registry says absence means
"unknown", never "unrestricted".

2026-10-11 16:34:00 +02:00