Nexus Recall — How It Works Under the Hood

01 — The Big Picture

What is Nexus Recall?

A two-pane app: the Tome (a document viewer) on one side and the Oracle (a streaming RAG chat) on the other. Everything up to the LLM call happens in your browser — parsing, chunking, embedding and vector search — so a document never leaves your machine unless you opt in to cloud embeddings. Only the final question + the retrieved passages are sent to the server to generate an answer.

🔒 Local-first

Embeddings run in a Web Worker via Transformers.js. The vector store is IndexedDB. Documents survive a refresh and work offline once indexed.

⚡ Streaming

Answers stream token-by-token over Server-Sent Events. Reasoning is separated from the reply; citations and cost arrive as metadata.

🧱 Layered

UI, API, and RAG/AI live in separate folders with a one-way import rule. The RAG core is testable with no browser and no network.

The stack

SvelteKit 2 · Svelte 5 TypeScript Vercel AI SDK · streamText Transformers.js · MiniLM LangChain.js · splitters pdfjs-dist idb · IndexedDB Fireworks.ai / Claude Zod Vitest + Playwright Tailwind v4

02 — Architecture

Layers, one rule

Code is organized by responsibility, and dependencies flow one way. The UI knows the API; the API knows the RAG/AI core; the core knows neither HTTP nor Svelte.

routes/ · components/

The split-pane view, ChatPanel (an @ai-sdk/svelte Chat), and DocumentViewer with highlight overlays. Reads streamed metadata for citations + cost.

State

lib/stores/

The ingestion state machine, API keys, theme, reasoning toggle, toasts. Svelte stores — small and single-purpose, no god-files.

RAG / AI

lib/rag/

parser → chunker → embeddings → vector-store. Pure, browser-side, framework-agnostic — unit-tested with a mock worker and fake-indexeddb.

API

routes/api/ · lib/server/

The HTTP boundary: key resolution, Zod validation, provider selection, the reranker singleton, context assembly, and the streaming interceptor — split into focused chat.*.ts modules.

UI → API → RAG/AI · never the reverse

Why it matters: the RAG core runs in Node tests with no browser; the API modules are small enough to read in one sitting; a UI rewrite can't reach into model code. Constants (CHUNK_SIZE, TOP_K, EMBEDDING_DIMS, MODEL_IDS) live in as const objects so logic never depends on raw literals.

03 — Getting Scrolls In

The ingestion pipeline

Drop a file and a per-document state machine carries it from pending → indexing → embedding → ready (or error) — all in the browser. Progress for chunking and embedding is tracked separately so the UI stays granular.

Parse

lib/rag/parser.ts · pdfjs-dist / markdown

PDFs are reconstructed into paragraphs from text-run geometry (Y-gaps, hyphen joins); Markdown has its frontmatter stripped.

Chunk

lib/rag/chunker.ts · LangChain splitters

Markdown-aware recursive split at 800 chars with 120 overlap; the nearest heading is prepended for context. Each chunk gets a deterministic source::pN::cN id.

Embed (Web Worker)

lib/rag/embedding.worker.ts · Transformers.js

A singleton MiniLM pipeline (384-dim) runs off the main thread. First load downloads weights (10–60s) with a progress bar; the model is reused after that.

Store

lib/rag/vector-store.ts · IndexedDB (idb)

Vectors + chunks are upserted into the nexus-recall DB and rehydrated on startup, so a refresh keeps your library.

04 — Asking the Oracle

The query pipeline

A question is embedded and searched locally, then the candidates go to the server to be reranked and answered. Colors mark the segment: browser, server.

Embed the question & vector-search

ChatPanel → embeddings.embedText → vector-store.similaritySearch

Cosine similarity over IndexedDB returns the top candidates, optionally scoped to the active document (sourceFilter) to avoid cross-doc bleed.

Server pipeline

routes/api/chat/+server.ts

Linear gauntlet: resolve keys → Zod validate → select provider → rerank → assemble context. Any failure short-circuits before a token is generated.

Cross-encoder rerank

lib/server/reranker.ts · ms-marco-MiniLM-L-6-v2

A module-level singleton (warmed by /api/warmup) reorders candidates by relevance; on any error it falls back to vector order. Top 8 become the context.

Assemble context & stream

chat.context.ts + chat.stream.ts · streamText

Chunks become numbered <source n> blocks; the Oracle answers with inline [n] citations. Reasoning is intercepted; cost is computed on finish.

What the Oracle streams back

Reasoning tokens are buffered and forwarded as message-metadata (shown in a separate panel, never in the answer). Citations are derived deterministically from the reranked chunks — no second LLM call.

🔎 vector search · 10 hits0.1s

⚖ rerank → top 80.4s

🧠 reasoning0.6s

✦ channeling the answer…streaming

05 — The Wire Format

The streaming protocol

The server returns an AI SDK v6 UI-message stream (SSE, data: <json>\n\n, header x-vercel-ai-ui-message-stream: v1). A server-side interceptor sits in the middle, reshaping a few chunk types before they reach the client.

Chunk	Handled how	Payload
message-metadata	Citations sent up-front; reasoning + cost merged in later	citations, reasoning?, usage?, costUsd?, truncated?
text-delta	Forwarded as-is — the visible answer	delta, id
reasoning-delta	Buffered, flushed every 6 tokens as metadata — never shown in the bubble	delta
reasoning-end	Final reasoning flush	—
finish	Compute usage + USD cost server-side, log, emit metadata, then forward	finishReason (stop · length · error)
finish · error	Swapped for a safe, in-character message ("the Oracle has gone silent")	—

Why intercept? Letting reasoning events reach the AI SDK client directly wedges it in a streaming state. The interceptor (chat.stream.ts) consumes the raw stream, narrows the chunks it cares about with type guards, and re-emits clean metadata — keeping chain-of-thought out of the answer and attaching cost on the way out.

06 — Safety & Boundaries

Guardrails

🛡 Prompt-injection

Retrieved text is wrapped in <source n> tags and the system prompt treats it as untrusted data. A scroll that says "ignore previous instructions" is disregarded — and there's a unit test that proves the delimiting.

🔑 Key handling

LLM keys travel as request headers, resolved server-side (user → demo env → 401) and never logged. Cloud-embedding keys are an explicit client-side opt-in (documented in an ADR).

🧪 Input boundary

Zod caps the question at 2000 chars and the payload at 200 chunks; null bytes and control characters are stripped before anything reaches the model.

🔀 Provider fallback

resolveProvider expresses primary → fallback → error in code: an explicit pick is honored if its key exists, otherwise Fireworks → Claude → a helpful error.

⏳ Stall timeout

The generation is bounded by AbortSignal.any of the client signal + a 60s AbortSignal.timeout. A provider that accepts the request but never streams can't hang the function to the platform limit — on expiry the stream aborts and the safe "the Oracle has gone silent" message is shown.

🔐 Security headers

vercel.json sets X-Frame-Options, nosniff, Referrer-Policy, a deny-by-default Permissions-Policy, HSTS, and the COOP/COEP the WASM worker needs. The CSP is owned by SvelteKit's kit.csp — it hashes the inline boot script, so script-src stays 'self' 'wasm-unsafe-eval' with no unsafe-inline.

📊 Observability

Every request carries a requestId; structured logs capture the rerank delta, retrieve + latency ms, tokens, and a coarse error category. OpenTelemetry spans (via @vercel/otel) record token usage, latency, and finish reason — with prompt/answer text deliberately off for privacy.

07 — The Tuning Knobs

Key numbers

No magic numbers scattered through the code — every constant lives in one module or an as const object, so a change is a one-line edit (and re-run the evals).

Chunking

size 800 overlap 120 id source::pN::cN

Embeddings

MiniLM 384 MPNet 768 OpenAI 1536

Retrieval

search 10 rerank top-k 8 ctx cap 24k chars

Generation

temperature 0.3 max output 1024 stall timeout 60s reasoning flush 6 question cap 2000

Models

deepseek-v4-flash claude-sonnet-4-6 rerank: ms-marco-MiniLM

Cost /1M tok

Fireworks $0.22 / $0.88 Claude $3 / $15 shown per answer

08 — Does the AI Actually Work?

Evaluation

Unit and E2E tests verify the code works; evals verify the retrieval works. pnpm eval runs offline against 20 ground-truth Q&A pairs and gates the build.

📏 Retrieval (free, in CI)

Recall@k & MRR for three strategies side by side, so the eval mirrors what production retrieves — not a proxy:
BM25 (lexical, deterministic) — the always-on PR gate, no model download
Vector (MiniLM cosine) — the app's similaritySearch
Vector + rerank (cross-encoder) — the full production path; on the same production-faithful chunks (Markdown 800/120 + heading-prepend) it holds recall@1 100% / MRR 100% where lexical BM25 dilutes to 85% / 93%
Gate: recall@3 ≥ 0.80 on both BM25 and the production path

⚖ Generation (LLM-as-judge)

Faithfulness — is every claim grounded in the context? (Gate ≥ 0.80)
Answer similarity — cosine vs. a gold answer (Gate ≥ 0.80)
Answer relevance — RAGAS question-regen (reported, 0.65 floor) — why it sits at ~0.73, a deep dive →
Scores append to evals/scores.json — a flat history is a cheap regression detector

The bug this caught: the eval used to score BM25 only — a proxy, not the shipped retrieval path. When we made it mirror production, it surfaced that the cross-encoder reranker had never actually run: its model id pointed at a repo with no ONNX weights, so the load failed and the try/catch fallback silently returned the un-reranked order. The lesson: a fallback hides a dead component unless an eval exercises the real path. Fix in, the eval now proves the reranker earns its latency.

The discipline: never ship a prompt, chunking, or embedding-model change without re-running evals — they look like refactors but silently move output quality.