Skip to content

Proposal: News (the news surface in Search mode)

Status: Superseded by four-modes.md · Last updated: 2026-07-15

2026-07-15 — moved again, to Read. Search itself stopped being a mode, so the news surface moved to where reading lives: Read's Articles and Feeds collections. The v1 reasoning below still holds — news is not a standalone mode, it belongs inside the surface it serves — only the host changed. $lib/news/ itself did not move; kinds 30107/30108 are unchanged.

2026-07-13 — no longer its own mode. News shipped v1 as a sixth mode at /news, then was folded back into Search the same day: Search is web search, so news discovery belongs inside it, not beside it. The /news route and mode entry are gone; the whole surface (Explore/Saved/Feeds tabs, URL bar, reader) renders below Search's hero, deep links moved to /search?a= / /search?url=, and the AI-settings gate no longer blocks reading (news needs no endpoint). Everything else below — data model, kinds 30107/30108, fetch layering, $lib/news/ — still stands as built; read route/mode references through that lens.

Summary

A news reader with three content sources behind one reading surface: nostr long-form articles (kind 30023), RSS/Atom feed subscriptions, and arbitrary news-site URLs. Grows out of Search mode's Explore-Recent-Events feed — article cards open the reader in place.

Motivation

The discover feed already proves demand and mechanics: real kind-30023 content, zero backend. What's missing is the reading: a proper article view instead of a dead-end card, plus the two content sources nostr can't cover (RSS feeds and the open web). SvelteReader's article flow is the UX ancestor; this rebuilds it client-only.

Decisions

  • External fetch: Firecrawl for everything (decided 2026-07-13). RSS feeds and news sites mostly don't serve CORS headers, so the browser can't fetch them directly. Firecrawl (api.firecrawl.dev, CORS *, user's BYO key — already wired for Search) scrapes article pages to markdown (/v2/scrape, formats:["markdown"]) and fetches raw feed XML (formats:["rawHtml"]), which is parsed in-browser. Graceful degradation: no key ⇒ nostr articles fully work; RSS/web surfaces show a configure-Firecrawl empty state.
  • Subscriptions and saved articles sync over nostr as per-item addressable events (kinds 30107/30108, below), reusing Read mode's LWW-by-updatedAt + tombstone machinery. Article content cache and read-state stay device-local — Firecrawl output can be hundreds of KB and relays cap event size; metadata syncs, bytes re-fetch on demand.
  • AI chat on articles is deferred (the search-mode proposal's "reading view with chat" vision). The reader layout keeps a right-column seam for a future chat sidebar; $lib/ai/ is mode-agnostic and bolts on later.
  • The discover feed moves to $lib/news/ and Search embeds it — the feed is News's core surface; the cross-mode import is documented in CLAUDE.md like the other sanctioned ones. Its hardcoded wss://relay.damus.io becomes settings.news.feedRelay.
  • One Firecrawl key — News reuses search.firecrawlApiKey; hoisting to a shared settings section waits for a third consumer (same rule as cross-mode imports).

Design

Nostr events: kinds 30107 / 30108

Per-item addressable events, not a single NIP-51/30078-style list: the existing sync engine already does per-address LWW + tombstones, so per-item events merge item-wise across devices, deletions reuse the tombstone machinery verbatim, and the codec is a copy of the annotation draft. A single list event would need bespoke list-merge logic and resurrects deleted items on stale-list push.

  • 30107 — news feed subscription (one event per feed). d = sha-256 hex of the normalized feed URL (lowercased scheme+host, trailing slash stripped) — deterministic, so two devices subscribing to the same feed pre-sync converge on one address, and the URL stays out of cleartext tags (subscriptions are private reading habits). Content: NIP-44 self-encrypted JSON of the NewsFeed record.
  • 30108 — saved article (read-later). d = sha-256 of the canonical article key (30023:<pubkey>:<d> for nostr; normalized URL for rss/web). Content: NIP-44 self-encrypted metadata only — never article markdown.

Registered in docs/nostr-event-model.md (30109+ remains unassigned).

Data model (DB v5)

export type NewsSource = 'nostr' | 'rss' | 'web';
/** nostr → `30023:<pubkey>:<d>`; rss/web → normalized URL */
export type NewsArticleKey = string;

interface NewsFeed {        // store 'newsFeeds' — synced as 30107
  id: string;               // sha-256 of normalized url = d-tag
  url: string; title: string; siteUrl?: string; description?: string;
  addedAt: number; updatedAt: number;   // updatedAt drives LWW
}
interface SavedArticle {    // store 'newsSaved' — synced as 30108
  id: string;               // sha-256 of key = d-tag
  key: NewsArticleKey; source: NewsSource; title: string;
  url?: string; nostr?: { pubkey: string; dTag: string };
  summary?: string; image?: string; publishedAt?: number;
  savedAt: number; updatedAt: number;
}
interface NewsArticleCache { // store 'newsCache' — device-local, never synced
  key: NewsArticleKey; markdown: string; title: string;
  fetchedAt: number; readAt?: number;   // read-state lives here
}

RSS/Atom parsing

firecrawlScrapeRaw fetches the feed body; DOMParser (text/xml) parses it in-browser. RSS 2.0 (channel>item: guid/link/title/description/pubDate/ enclosure) and Atom (feed>entry: id/link@href/title/summary/updated), namespace-tolerant lookups. Items dedupe by guid→link, sort by date desc. Feed items render title/summary/date from the XML only; opening an item always full-scrapes the linked page (RSS bodies are usually truncated). Fetch layering: plain fetch(url) first (many feeds do serve CORS), Firecrawl rawHtml for the blocked rest — both feed the same parser.

Reader & routing

  • Route /news; a ui.view store switches feed | reader | saved | subscriptions — no subroutes. TopBar stays (not immersive).
  • Deep links /news?a=30023:<pubkey>:<d> and /news?url=<encoded>, mirrored with replaceState, gated on handledInitialUrl (the ?book= pattern).
  • Article load path: discover-feed content → local cache → nip50Search refetch by authors+#d → (web) Firecrawl scrape.
  • Rendering: renderNewsMarkdown() — marked+DOMPurify mirroring $lib/ai/markdown.ts but allowing img and dropping the [sN] cite pass. The AI variant stays image-free.
  • Search's card click navigates goto('/news?a=…'); mode switches naturally since mode derives from URL.

Refresh discipline

All relay/Firecrawl traffic sits behind explicit user actions — no timers, no background polling. v1 has no "refresh all": a subscription's items are fetched when the user opens that feed (one fetch per click, errors isolated per feed by construction). A combined refresh-all river stays a possible later addition; if built, it must fetch sequentially (never Promise.all) to respect Firecrawl 402/429.

Open questions

  • Feed quality/spam: one-per-author cap carries over from the discover feed; WoT filtering remains open (shared with search-mode proposal).
  • OPML import/export for subscriptions — cheap, deferred.
  • AI chat sidebar on the reader — phase 2 of the mode, after this proposal completes.
  • Following nostr authors (a subscriptions-like list of 30023 authors, distinct from kind-3 follows) — deferred; the saved-article path covers the immediate need.

Phases

  • Phase 0 — This proposal; kind registry rows in docs/nostr-event-model.md.
  • Phase 1 — Mode scaffold ('news' mode + /news route), discover feed moved to $lib/news/, ArticleReader with renderNewsMarkdown, ?a= deep links, Search card click-through, news.feedRelay setting, e2e/news.test.ts.
  • Phase 2 — Web + RSS fetching: firecrawlScrapeWithRaw (one call returns rawHtml + markdown — spike verified rawHtml IS the feed XML on hnrss/github.blog/substack, 2026-07-13), rss.ts parser, URL bar (feed-vs-article sniff, plain-fetch-first), feed item list, no-key states, fixture-driven e2e.
  • Phase 3 — Subscriptions + saved: DB v5 stores, subscription/saved/cache stores, $lib/news/nostr/ codec + sync (30107/30108, reusing Read's AddressableDraft/dedupe/tombstone machinery), session hooks, Explore/ Saved/Feeds tabs + Sync button, Save/Subscribe toggles, e2e incl. a full encrypted push → wiped-device → pull-restore roundtrip against mocked pool relays.
  • Phase 4 — Polish: read-state dimming (discover cards + feed items), newsCache LRU eviction (cap 200, saved exempt), MkDocs user docs (docs/news.md), CLAUDE.md gotchas section.
  • Phase 5 — Folded into Search (see status note at top): mode + /news route removed, NewsSection embedded under Search's hero, reader takes over the Search surface, deep links on /search, docs merged into docs/search.md.
  • Phase 6 (2026-07-14) — Feed-item caching + free-fetch layering, to stop every saved-feed click from spending a Firecrawl credit:
  • newsFeedCache store (DB v7), keyed by normalized feed URL: opening a feed URL renders the cached ParsedFeed with zero network; only the explicit Refresh button in the item-list header (or a cache miss) fetches. Header stamps "Updated Xm ago". Eviction: unsubscribed feeds' rows dropped at session start (previews are one-off; only subscriptions earn a persistent cache) and on unsubscribe.
  • Optional news.feedProxy setting (synced in the 30111 slice): a CORS pass-through URL template ({url} placeholder, else appended) tried for feed XML between the direct fetch and Firecrawl. Feed-only — articles still need Firecrawl's markdown extraction. Deliberately opt-in and empty by default: the proxy operator sees which feeds you read, so privacy-conscious users can leave it off and pay Firecrawl instead.
  • Note: Firecrawl batch-scrape is NOT a saver — it charges per URL, so "one credit for N feeds" is impossible on their API. The real long-term saver is the planned project VPS (relay + Blossom + a tiny feed CORS-proxy endpoint, e.g. a Caddy reverse_proxy block) becoming the default feedProxy, making all feed fetching free; Firecrawl then only pays for article extraction.

Each phase gated on pnpm check + pnpm build + full Playwright suite.