11 — Resilience & Error Handling (grag-client Policy)

Purpose. This document is the normative error-handling and resilience specification for packages/grag-client (plan 2.5) and every Knoll-backend feature that calls the GRAG platform (plan 7.4, 7.6). It defines: the master error taxonomy (what each HTTP status / SSE frame means on GRAG and what the client and the German UI must do), the ⚠ missing-tenant hazard, the retry/backoff and circuit-breaker policy (including the ~30 s deploy-churn window on every GRAG merge), SSE stream resilience, model cold-start handling, rate-limit budgets, per-feature degradation ladders, and the observability hooks that make failures diagnosable. An engineer implementing grag-client should be able to code the error paths from this document alone.

Status / verified against: 2026-07-06, repo document-processing-pipelines @ db63a95 (live: https://app.grag.ai). Sibling docs: architecture in 01-architecture.md, request recipes in 02-grag-api-cookbook.md, ID rules in 03-id-conventions.md, chat specifics in 09-chat-integration.md, ingest specifics in 10-document-pipeline.md, open platform gaps in 13-platform-gaps-issues.md, budget mechanics in 14-cost-model.md.

1. Scope & principles

1. All GRAG traffic goes through grag-client. No feature code calls fetch against $GRAG_URL directly. The client enforces headers, taxonomy mapping, retries, and the circuit breaker in one place.
2. Fail loudly on client-side contract violations (missing tenant, missing workspace, invalid ID charset) before a byte leaves the Knoll backend — see §3.
3. Fail soft on platform degradation. GRAG itself is largely fail-open (retrieval gate trims but never blocks, chat stages fail open except generate, budget poller fails open). The Knoll UI mirrors that with explicit degraded states (§9), never with silent wrong data.
4. Knoll DB and the Knoll file store are the source of truth (plan 2.1/2.4, target architecture). Anything lost on the GRAG side (queued ingest, KG edges) must be replayable from Knoll-side state (§5.3, plan 3.6).

Every request example in this doc uses the standard header set. Base pattern https://app.grag.ai/<traefik-prefix>/<internal-path>:

curl -sS "$GRAG_URL/workspaces/api/v1/kbs" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Request-ID: $(uuidgen)"

kg-service and entity-linking additionally require X-Workspace-ID on every non-public route (400 without it).

2. Master error taxonomy

Error envelope everywhere: {"detail": "..."} (FastAPI style); every response carries an X-Request-ID header (echoed if we sent one, else generated). Cross-tenant access always returns 404, never 403 — a 404 therefore means "does not exist in our tenant".

⚠ Exception to the whole table: the BFF lane (/next/api/*) is unauthenticated today (plan 1.8 — blocker). Until 1.8 lands, a 401/403 can never come from the BFF routes; after 1.8 the BFF rows behave like the rest of the table. The verification probe is plan 0.10a / 7.2.

Decision rule for ambiguous 503: route + detail text. Dark-flag 503s are stable (same answer every time → cache the unavailability); safety/admission 503s are transient (retry). When in doubt, retry once; if the second answer is identical detail, treat as dark.

3. ⚠ The missing-tenant hazard (hard-fail before sending)

allow_default_tenant=true is the platform-wide default (pipeline-common settings) and is not overridden in the prod env example: a request without X-Tenant-ID does not error — it is silently served as tenant default. Consequences of one forgotten header:

• Writes (documents, entities, conversations, settings) land in tenant default, invisible to tenant knoll — data loss from Knoll's perspective, orphaned PII from DSGVO's perspective (12-gdpr-compliance.md).
• Reads return empty/foreign results with HTTP 200 — no error to catch downstream.

Policy (plan 2.5, 7.7 — non-negotiable):

1. grag-client constructor requires a tenant ID; every request builder throws GragClientError("tenant missing") if the resolved tenant is empty, default, system, or canary (reserved IDs).
2. X-Workspace-ID is mandatory for every kg-service/entity-linking call; the typed client methods for those services take workspaceId as a required parameter — not an option.
3. ID charset validation (^[a-z0-9][a-z0-9-]{0,62}$) runs client-side for tenant, workspace, KB, conversation and document IDs (03-id-conventions.md), so the 400 row in §2 is unreachable in practice.
4. A CI test asserts that no code path can construct a GRAG request without the two headers (unit test over the client's request middleware).

// packages/grag-client — the only place headers are assembled
function baseHeaders(scope: { tenant: string; workspace?: string }) {
  assertDnsLabel(scope.tenant);
  if (RESERVED_TENANTS.has(scope.tenant)) throw new GragClientError("reserved tenant");
  return {
    Authorization: `Bearer ${env.GRAG_API_KEY}`,
    "X-Tenant-ID": scope.tenant,           // hard-required, never defaulted
    ...(scope.workspace ? { "X-Workspace-ID": scope.workspace } : {}),
    "X-Request-ID": crypto.randomUUID(),
  };
}

4. Retry & backoff policy

4.1 Classes

Timeout budget per attempt: 30 s default; 150 s for LLM completions and groundedness (covers the BFF's own 120 s completion timeout and cold starts, §7); 60 s for search/rerank first-call-after-deploy. Overall deadline for an interactive user-facing operation: 180 s, then fail with the mapped German message.

4.2 What is explicitly NOT retried

• Validation failures (422/400): retrying cannot succeed and pollutes logs; they page the on-call as contract drift instead (plan 1.9).
• Cross-tenant-shaped 404s: retrying leaks time; reconcile grag_refs instead.
• 402 budget: retrying spends nothing but signals wrong intent; hard stop + alert.
• Chat SSE turns: a dropped/failed stream is never auto-resent (the LLM may have partially run; the user resends consciously). The user message is already persisted by the BFF's persist_user stage, so no input is lost.

5. Circuit breaker & deploy-churn handling

5.1 The churn reality

GRAG deploys on every push to main with --force-recreate: all ~90 containers restart, no blue/green, typical unavailability ~30 s, and transient 401/404/422 answers can occur while containers come up behind Traefik (documented platform behavior; single-box SPOF). Knoll must treat this as routine, not incident.

5.2 Breaker spec (one breaker for the whole GRAG origin)

• Open when: ≥3 connection-level failures (refused/reset/TLS/timeout) or 502/503 responses across distinct endpoints within 30 s.
• While open: all non-queued calls fail fast with the "wird gerade aktualisiert" state; ingest/KG writes divert to the replay queue (§5.3). Probe loop:

# public, unauthenticated liveness probe (no headers needed)
curl -fsS -m 3 "$GRAG_URL/workspaces/health"

• Half-open after 2 consecutive healthy probes (5 s interval): send ONE authenticated canary read — this catches the deploy 401-window that /health cannot see:

curl -fsS -m 5 "$GRAG_URL/workspaces/api/v1/kbs" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: knoll-dev"

• Close on 200; while half-open, only the canary flows.
• Alert if the breaker stays open > 5 min (that is no longer a deploy — see §10).
• State deploy-suspected: set for 120 s after any open→close transition; within it, the §2 nuances apply (single extra retry for GET 401/404/422) and cold-start timeouts are used (§7).

5.3 Knoll-side job queue & replay (plan 7.4, risk table)

Ingest and KG writes must survive GRAG being down mid-operation:

1. Uploads: the file lands in the Knoll file store first (plan 4.1/2.4). The upload→convert→ingest sequence runs as a Knoll-side job (dateien + grag_refs rows with sync_status ∈ pending | in_flight | done | failed | pending_replay).
2. On breaker-open or terminal connection failure, the job flips to pending_replay; a worker replays it once the breaker closes. Replay is idempotent end-to-end: deterministic document_id/KB IDs (409 = already there), docfold re-convert is safe, voyager point IDs are chunk-derived, KG upserts/edges are MERGE-idempotent (plan 3.6 reconcile job covers the KG side on schedule as well).
3. In-flight ambiguity (connection died after send): re-poll by ID before re-submitting — GET /workspaces/api/v1/kbs/{kb}/documents / GET /next/api/pipeline/status/{jobId} tell us whether the previous attempt took effect.
4. Chat and other interactive reads are not queued — they fail fast with the deploy state and let the user retry.

6. SSE resilience

Three streams matter (all via the BFF lane, blocked-for-prod by plan 1.8 like all BFF traffic):

6.1 Ingest progress — GET /next/api/orchestrator/events/{jobId}

• Contract: opening event: subscribed, then verbatim orchestrator events (node_start/node_complete/node_fail/completed/failed/cancelled/…), : keepalive comment every 20 s (KEEPALIVE_MS = 20_000), hard 30-min ceiling ending with event: timeout then stream close (STREAM_TIMEOUT_MS = 30 min). Errors as event: error.
• Watchdog: if no bytes (data or keepalive) for 45 s, consider the stream stalled — close it and fall back to polling.
• On timeout: the job may still be running. Fall back to polling; do NOT mark failed.
• Fallback polling: GET /next/api/pipeline/status/{jobId} every 5 s (status hash: {status, progress, error, results}); terminal failed drives the plan 4.2 UX (checklist item stays „Angefordert", visible error, re-upload path).
• Reconnect semantics: the bridge is Redis pub/sub — missed events are NOT replayed. After any reconnect, poll the status endpoint once to resync before trusting the stream again.

6.2 Chat — POST /next/api/chat/stream

• POST-body SSE consumed via fetch().body.getReader() (no EventSource, no auto-reconnect).
• No keepalive frames exist on this stream (verified: no keepalive emission in app/api/chat/stream/route.ts). Long silences are normal: the upstream completion is non-streaming (fake tokenization only starts after the full completion returns), and the BFF's completion timeout is 120 s, groundedness 30 s.
• Watchdog: 150 s since last event → abort, show the chat error state. Do not auto-resend (§4.2); the persist_user stage has already saved the user turn.
• Stage semantics: every trace stage except generate fails open (status: failed on the trace event, stream continues). Map stage failures to the §9 degraded states instead of aborting.
• A dropped stream mid-answer means the assistant message was likely not persisted (assistant persist happens at the end) — on reload, the turn shows the user message only; offer „Erneut senden".
• Remember plan 5.2: the conversation must exist (created via POST /workspaces/api/v1/conversations) before the first stream call — the BFF does NOT create conversations, and persistence fails silently (fail-open) for unknown IDs.

6.3 Orchestrator/agent relay ceilings

• The 30-min ceiling means any pipeline expected to run longer (bulk ingest) must be monitored by polling, not by a single SSE subscription.
• Agent session relay (/next/api/agents/sessions/{id}/stream, only if plan 5.6 spike happens): surface event: error frames; 503 = agent plane dark (profile not running).

6.4 Proxy buffering caveat

The documented Plesk-nginx snippet does not set proxy_buffering off for the SSE routes. Symptom if buffered: all events arrive in one burst at stream end. TODO-VERIFY: live SSE probe through https://app.grag.ai for both streams (plan 0.10b; test steps in 05-verification-runbook.md). If buffering is confirmed: ingest progress falls back to polling permanently (acceptable), chat becomes an answer-at-once UX (not acceptable) → escalate as a platform issue in 13-platform-gaps-issues.md.

7. Cold starts & preloads

Model-serving services lazy-load weights on first call:

Policy:

1. Plan 1.4 asks the operator to set the preload flags. TODO-VERIFY: live values of RERANK_FUSION_PRELOAD / GROUNDEDNESS_PRELOAD on app.grag.ai (flag dump via host-diagnose.yml, plan 0.9).
2. Even with preloads, every merge recreates the containers — so after each breaker open→close cycle, grag-client enters deploy-suspected (§5.2) and uses the long first-call timeouts (60 s rerank, 150 s groundedness/LLM).
3. Optional warm-up (recommended for the Gutachten pipeline, plan 5.3): after the breaker closes, fire one background warm-up pair with generous timeouts:

# warm rerank-fusion via the compound route (tiny query, kb-methodik)
curl -sS -m 150 "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/kb-methodik/search/rerank" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"query_text": "warmup", "top_k": 1}'

# warm groundedness
curl -sS -m 150 "$GRAG_URL/groundedness/api/v1/score" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"response_text": "warmup", "chunks": [{"chunk_id": "w1", "text": "warmup"}]}'

4. UI: during a known warm-up window, the chat input shows „Die KI-Dienste starten gerade — die erste Antwort kann bis zu zwei Minuten dauern."

8. Rate limits & client-side throttling

⚠ Note the caller-attribution asymmetry (plan 0.5): BFF-lane traffic never presents Knoll's key (the BFF injects the platform key server-side), so the per-caller gateway buckets only govern grag-client's direct service calls until plan 1.8 adds BFF auth.

Client-side throttling policy (grag-client):

• Global concurrency cap: 8 concurrent GRAG requests (single-box backend; be a polite tenant), separate semaphore of 2 for ingest pipelines.
• Token bucket at 50 req/s sustained (half of the Traefik ceiling, leaving headroom for the platform's own UI traffic from other IPs — Traefik keys on our IP alone, but bursts of parallel Gutachten steps plus polling can approach it).
• Retrieval calls throttled to ≤100/min (headroom under the 120 rpm bucket); the Gutachten pipeline step 2 (per-document search/rerank fan-out, plan 5.3) processes documents sequentially with at most 2 in flight.
• Polling intervals: job status 5 s, never below 2 s; back off polling ×2 while a 429 is hot.
• Bulk migration/backfill (e.g. initial kb-methodik ingest, plan 3.3): run at ≤30 rpm and outside business hours; if a real bulk load is planned, raise limits with the operator first (plan 1.15).

9. Degradation ladder per feature

Principle: every degraded mode is visible but non-blocking, and Knoll SQL is the fallback for anything whose source of truth is Knoll DB anyway (plan 3.7/5.4). UI copy in German.

10. Observability hooks

10.1 Correlation

• X-Request-ID: grag-client sets a fresh UUIDv4 on every request; the platform echoes it on the response (or generates one). Log it on both send and receive; store it in ai_runs.refs and grag_refs.last_error context so a Knoll error can be joined to GRAG-side logs by the operator.
• X-Pipeline-Id: stamped once per Gutachten run / per Analyse batch (plan 7.3) on every ai-gateway call; this is the join key for per-Analyse cost: GET /ledger/api/v1/ledger/totals?pipeline_id=<run-id> (⚠ exact path — there is no per-pipeline grouping on /spend). Tenant-level budget display uses GET /ledger/api/v1/ledger/spend?group_by=service|provider (the only two groupings).

curl -sS "$GRAG_URL/ledger/api/v1/ledger/totals?pipeline_id=gutachten-a-042-run-7" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT"

10.2 Structured logging (grag-client, one JSON line per attempt)

Fields: ts, level, request_id, tenant, workspace_id?, kb_id?, analyse_id?, ai_run_id?, grag_service, method, path (templated, no IDs with PII), status, error_class (taxonomy row from §2), attempt, retry_after_s?, latency_ms, circuit_state, pipeline_id?. Never log request/response bodies (Mandanten content, PII) — lengths and sha256 only, mirroring the platform's own discipline.

10.3 Client-side metrics (Knoll backend)

• grag_client_requests_total{service,error_class,retried}
• grag_client_request_duration_seconds{service} (histogram)
• grag_client_circuit_state (0 closed / 1 half-open / 2 open) + grag_client_circuit_open_seconds_total
• knoll_replay_queue_depth{kind=ingest|kg} and knoll_replay_failures_total
• knoll_sse_fallback_total{stream=chat|orchestrator}

10.4 Alert conditions (plan 7.6)

10.5 Canary tie-in

Plan 7.2's knoll-flow canary (create KB → ingest fixture → search → chat → purge) plus the BFF-auth probe run on the GRAG side after every deploy; grag-client's circuit/deploy telemetry is the Knoll-side mirror. When both fire simultaneously, it is a platform regression, not a Knoll bug — file it against 13-platform-gaps-issues.md.

Appendix A — Taxonomy quick map for implementers

connection error ─────────▶ circuit breaker (§5) ─▶ queue/replay or fail-fast
401/403 (auth) ───────────▶ no retry, ops alert (deploy-window nuance §2)
403 (admin gate) ─────────▶ operator runbook (04-provisioning-runbook.md)
400 tenant/workspace ─────▶ unreachable if §3 enforced (client bug)
402 ──────────────────────▶ hard stop, "KI pausiert", page
404 ──────────────────────▶ our-tenant-miss; reconcile grag_refs
409 create ───────────────▶ SUCCESS (already provisioned)     [plan 3.1]
409 docfold result ───────▶ keep polling
413 ──────────────────────▶ split batch (≤256 points)
422 ──────────────────────▶ contract bug, never retry          [plan 1.9]
429 ──────────────────────▶ Retry-After + jitter (class R3)
502 ──────────────────────▶ retry idempotent (R1), else surface + re-upload path
503 dark flag ────────────▶ feature-unavailable cache, hide feature
503 fail-closed safety ───▶ one retry, then fail
503 + Retry-After (WAL) ──▶ honor Retry-After / re-queue ingest
SSE error ────────────────▶ chat: fail turn; jobs: fall back to polling
SSE timeout (30 min) ─────▶ switch to polling, job not failed

Appendix B — Open verification items in this document

• TODO-VERIFY: does Traefik's rate-limit@file 429 carry a Retry-After header? (Live probe; grag-client currently assumes it may be absent and uses a 2 s floor.)
• TODO-VERIFY: SSE pass-through vs. buffering through Plesk nginx for /next/api/chat/stream and /next/api/orchestrator/events/{id} (plan 0.10b; procedure in 05-verification-runbook.md).
• TODO-VERIFY: live values of RERANK_FUSION_PRELOAD, GROUNDEDNESS_PRELOAD (and the plan 0.9 flag dump generally) on app.grag.ai — cold-start timeouts stay in force until confirmed.