02 — GRAG API Cookbook

Purpose. This is the hands-on API reference an engineer keeps open while implementing packages/grag-client (plan 2.5) and every feature that talks to the GRAG platform. It is organized by capability: each section gives the exact external route, a complete curl example with the mandatory headers, the success-response shape, and the error cases the client must handle. Architecture context lives in 01-architecture.md; id rules in 03-id-conventions.md; how the calls compose into flows in 04-provisioning-runbook.md, 08-gutachten-pipeline-spec.md, 09-chat-integration.md, and 10-document-pipeline.md; error-handling policy in 11-resilience-and-errors.md.

Status / verified against: 2026-07-06, repo document-processing-pipelines @ db63a95 (schemas read from service source: workspaces/src/workspaces/schemas.py, ai-gateway/src/ai_gateway/api/schemas/*, kg-service/src/kg_service/api/schemas/*, ledger/src/ledger/models.py, frontend-next/apps/web/app/api/*). Live flag state on app.grag.ai is not assumed — see plan 0.9 and 05-verification-runbook.md.

0. Conventions used in every example

Base URL pattern. Every service is reachable externally as https://app.grag.ai/<traefik-prefix>/<internal-path> — Traefik strips the first path segment before the service sees the request. Examples use env placeholders:

export GRAG_URL="https://app.grag.ai"
export GRAG_API_KEY="sk-knoll-..."       # Knoll runtime key (plan 1.1: PIPELINE_API_KEY entry + GATEWAY_CALLERS)
export GRAG_ADMIN_KEY="sk-admin-..."     # operator-held admin key (plan 1.2), NEVER in the Knoll runtime
export GRAG_TENANT="knoll"               # dev examples use knoll-dev

Mandatory headers. Every direct service call sends:

Authorization: Bearer $GRAG_API_KEY
X-Tenant-ID: $GRAG_TENANT

grag-client must hard-fail before sending if X-Tenant-ID is unset — the platform runs with allow_default_tenant=true everywhere, so a missing header silently reads/writes tenant default (plan 2.5, 7.7). kg-service additionally requires X-Workspace-ID on every non-public route (400 without it). Full cheat-sheet in §O.

Two lanes, two auth realities.

1. Direct service lane (/workspaces, /ai-gateway, /kg-service, /groundedness, /ledger, /lineage): bearer-authenticated as above.
2. BFF lane (/next/api/* — chat stream, upload/ingest, orchestrator events): ⚠ as deployed today this surface is publicly reachable with NO authentication — the routes inject the platform key server-side and trust the inbound X-Tenant-ID. Plan 1.8 is a hard blocker: no real Mandanten data before the BFF authenticates callers. Write client code to send Authorization: Bearer $GRAG_API_KEY on BFF calls anyway, so nothing changes when 1.8 lands.

Common response conventions. Errors are {"detail": "..."}. List endpoints on the workspaces service return {"items": [...]}. Request bodies are extra="forbid" — an unknown field is a 422, not silently ignored. Every response carries X-Request-ID (echoed if you send one). Cross-tenant probes return 404, never 403.

Idempotency idiom (everywhere). There is no Idempotency-Key header anywhere in the platform (review-verified; it would be silently ignored). Idempotency = deterministic client-chosen ids + treat 409 as already-provisioned success (plan 3.1). All ids are DNS-label constrained: ^[a-z0-9][a-z0-9-]{0,62}$ — slugify German umlauts first (see 03-id-conventions.md).

A. Tenants, workspaces, KBs (workspaces service)

Maps to plan 1.3 (tenant), 3.1 (Mandant workspace + Analyse KB), 3.3 (kb-methodik). Hierarchy: tenant → workspace (≥1, "general" auto-created, undeletable) → KB. No project layer is used (plan D1). Provisioning details in 04-provisioning-runbook.md.

A.1 Create tenant (operator action, admin key)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/tenants" \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"id": "knoll", "name": "Kanzlei Knoll", "metadata": {}}'

Success 201 (abbreviated):

{"id": "knoll", "name": "Kanzlei Knoll", "default_workspace_id": "general", "metadata": {}}

Auto-creates the undeletable general workspace. Errors: 403 when called with the runtime key (this is the post-deploy check for the plan-1.2 custody split — the Knoll runtime key must get 403 here), 409 duplicate id (= already provisioned, fine), 422 id not a DNS label. GET/PATCH/DELETE /workspaces/api/v1/tenants/{id} exist; DELETE is soft-only — there is no composite tenant purge (plan 3.2).

A.2 Create workspace (one per Mandant)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/workspaces" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "client-hartmann-x7k2",
    "name": "Hartmann Maschinenbau GmbH",
    "description": "Client m-01",
    "system_prompt": "",
    "metadata": {"knoll_client_id": "m-01"}
  }'

Success 201 echoes the workspace row. 409 = already exists = success (record the id in grag_refs either way). Other routes: GET /workspaces/api/v1/workspaces (list), GET/PATCH/DELETE /workspaces/api/v1/workspaces/{id}. DELETE returns 409 while live children (KBs) exist unless ?cascade=true; deleting general is always 409.

A.3 Create KB (one per Analyse; voyager collection = kb id)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/workspaces/client-hartmann-x7k2/kbs" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "kb-analysis-a01",
    "name": "Analyse a-01 Hartmann",
    "description": "Document corpus of the analysis",
    "status": "indexing",
    "languages": ["de"],
    "pii": true,
    "metadata": {"knoll_analysis_id": "a-01"}
  }'

KBCreate fields (all verified): id, name, description, icon?, color?, status ∈ live|beta|indexing (default indexing), languages[], embedding (string, informational), chunker (string, informational), pii (bool), project_id? (unused by Knoll), metadata{}. Errors: 404 unknown workspace, 409 duplicate, 422 bad id.

Lists: GET /workspaces/api/v1/kbs?workspace_id= (flat) or GET /workspaces/api/v1/workspaces/{ws}/kbs. GET/PATCH/DELETE /workspaces/api/v1/kbs/{id} — DELETE soft-deletes and fans out a voyager/kg/lineage purge only when WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4; verify per plan 0.9).

The KB's chunks/documents/nodes counters are eventually consistent (worker-driven from lineage events + 5-min resync) — never assert exact counts right after ingest.

A.4 The 409-as-success idiom (grag-client contract)

code=$(curl -s -o /tmp/resp.json -w "%{http_code}" -X POST "$GRAG_URL/workspaces/api/v1/workspaces" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"id":"client-hartmann-x7k2","name":"Hartmann Maschinenbau GmbH"}')
case "$code" in
  201|409) echo "provisioned";;   # 409 == someone (or a retry) got there first — same end state
  *) echo "FAIL $code"; cat /tmp/resp.json; exit 1;;
esac

Because ids are deterministic (03-id-conventions.md), a retried provisioning run converges instead of duplicating.

B. Document inventory (workspaces service)

The documents table is the user-facing inventory per KB — metadata only. GRAG never retains the original binary (docfold output is markdown; job payloads expire): the master copy of every file lives in the Knoll file store (plan 2.4), and the inventory is a cache, not duplicated into Knoll's DB (plan 4.4). See 10-document-pipeline.md.

B.1 Create a document row (pre-registration; the BFF upload lane does this for you)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/kbs/kb-analysis-a01/documents" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "kb_id": "kb-analysis-a01",
    "name": "jahresabschluss-2025.pdf",
    "mime": "application/pdf",
    "size_bytes": 4194304,
    "status": "queued",
    "source_kind": "upload",
    "metadata": {"knoll_checklist_item_id": "ci-07"}
  }'

DocumentCreate (verified): id? (server-generates when absent — prefer letting the BFF/server generate here and store the returned id in grag_refs), workspace_id? (derived from KB), kb_id, name, mime, size_bytes, pages, status ∈ queued|processing|indexed|failed|superseded (default queued), language, source_kind ∈ upload|web|sharepoint|gdrive|zendesk|api, source_uri?, docfold_artifact_id?, lineage_pipeline_id?, metadata{}. Success 201 → {"id": "..."} plus the row.

B.2 List documents per KB (the Analyse document list, plan 4.4 / 6.3)

curl -sS "$GRAG_URL/workspaces/api/v1/kbs/kb-analysis-a01/documents?status=indexed" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Success: {"items": [{"id": "...", "name": "...", "status": "indexed", "chunks": 42, "mime": "application/pdf", "supersedes_document_id": null, ...}]}. Also: GET /workspaces/api/v1/documents?workspace_id=&kb_id= (flat) and GET /workspaces/api/v1/documents/{id} (?include_deleted=true supported).

B.3 Update / supersede / reingest / delete

# PATCH metadata or status (fields per DocumentUpdate: name, mime, size_bytes, pages,
# chunks, nodes, status, language, source_uri, docfold_artifact_id, lineage_pipeline_id,
# error, metadata)
curl -sS -X PATCH "$GRAG_URL/workspaces/api/v1/documents/doc-123" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"metadata": {"knoll_checklist_item_id": "ci-07", "confirmed": true}}'

# Replace flow (plan 4.4/4.5): upload the NEW document first (section C), then:
curl -sS -X POST "$GRAG_URL/workspaces/api/v1/documents/doc-123/supersede" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"superseded_by": "doc-456"}'
# 400 on self-supersede; both rows must be live; old row purged only when purge flag on.

# Re-ingest signal (v1 = trigger-only, records intent, flips status to queued):
curl -sS -X POST "$GRAG_URL/workspaces/api/v1/documents/doc-123/reingest" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" -d '{"reason": "korrigierte Fassung"}'
# 409 if the document is superseded.

# Delete (soft + purge fan-out when WORKSPACES_DOCUMENT_PURGE_ENABLED=true):
curl -sS -X DELETE "$GRAG_URL/workspaces/api/v1/documents/doc-123" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Delete GRAG-side and delete the original from the Knoll file store (plan 3.2/7.5) — they are separate stores.

C. The ingest lane (7 steps, BFF submission)

The orchestrator has no HTTP submit — from outside, ingest is only reachable through the BFF (/next/api/upload/start + /next/api/ingest/start), which converts, ensures the voyager collection, signs the DAG envelope, and enqueues it (plan D4). ⚠ Unauthenticated until plan 1.8 — dev data only until then. Full flow spec in 10-document-pipeline.md.

C.1 Upload (convert-only step)

curl -sS -X POST "$GRAG_URL/next/api/upload/start" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -F "kb_id=kb-analysis-a01" \
  -F "files=@jahresabschluss-2025.pdf"
# optional replace flow: -F "replaces_document_id=doc-123"

Form fields (verified in upload/start/route.ts): kb_id, files (repeatable), replaces_document_id?. Success:

{
  "kb_id": "kb-analysis-a01",
  "jobs": [
    {
      "filename": "jahresabschluss-2025.pdf",
      "size": 4194304,
      "docfoldJobId": "df-job-8f3a...",
      "documentId": "doc-123",
      "replacesDocumentId": null
    }
  ]
}

Per-file failures appear as error on the job entry instead of failing the whole call. Supported input formats include pdf, docx/doc, xlsx/xls, pptx, images (OCR), eml/msg, html, md, csv — everything on the Knoll 46-item Checkliste.

C.2 Start ingest (chunk → index)

curl -sS -X POST "$GRAG_URL/next/api/ingest/start" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "kb_id": "kb-analysis-a01",
    "document_id": "doc-123",
    "docfold_job_id": "df-job-8f3a...",
    "filename": "jahresabschluss-2025.pdf",
    "workspace_id": "client-hartmann-x7k2"
  }'

Success: {"jobId": "job-...", "tenantId": "knoll", "status": "queued"}. Always pass workspace_id — it rides the KG ingest envelope and determines which workspace's segments/spans the document lands under (needed for §I.4 and chat full-text expansion). Errors: 502 when docfold conversion failed or the collection could not be ensured (surface + offer re-upload; do not retry blindly — the docfold job may legitimately still be running, the BFF handles the 409-while-running poll internally). The ingest DAG honors the tenant setting ingest.dag_template (plan 1.7 adds an enrichment node — default DAG is chunking→index only).

C.3 Progress via SSE (orchestrator-events bridge)

curl -N "$GRAG_URL/next/api/orchestrator/events/job-abc123" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Accept: text/event-stream"

Wire contract (verified against orchestrator/events/[jobId]/route.ts + orchestrator SSE event types):

event: subscribed   data: {"channel": "...", "jobId": "job-abc123", "tenant": "knoll"}
event: node_start   data: {...}                # one per DAG node
event: node_complete data: {...}
event: batch        data: {...}
event: node_fail    data: {...}
event: completed | failed | cancelled          # terminal — close the stream
: keepalive                                    # comment frames every 20 s
event: timeout                                 # hard 30-minute ceiling
event: error        data: {"message": "..."}

Other possible event names forwarded verbatim: status, budget_exceeded, fallback_triggered. Terminal failed → keep the checklist item at "Angefordert", surface the error, offer re-upload (DLQ replay is operator-facing, plan 4.2). Note plan 0.10b: SSE buffering through the Plesk nginx must be live-verified before relying on real-time progress.

C.4 Fallback poll

curl -sS "$GRAG_URL/next/api/pipeline/status/job-abc123" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Returns {status, progress, error, results}; on COMPLETED, results.<chunk-node-id>.chunks carries the chunk count.

D. Retrieval: the compound /search/rerank route (ai-gateway)

One call does semantic search → BM25+RRF+cross-encoder rerank → evidence gate → per-doc cap. This is the retrieval primitive for the Gutachten pipeline step 2 (plan 5.3) and Dienstleister discovery context. German queries auto-route to the multilingual bge-reranker-v2-m3.

D.1 Request

curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/kb-analysis-a01/search/rerank" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Pipeline-Id: run-a01-gutachten-001" \
  -H "Content-Type: application/json" \
  -d '{
    "query_text": "Wie ist die Marktposition des Unternehmens einzuschätzen?",
    "top_k": 10,
    "rerank_top_n": 25,
    "rrf_k": 60,
    "language": "multi"
  }'

Body fields (all verified): query_text (required for this lane), top_k (default 10, ≤200), rerank_top_n (default 25), rrf_k (default 60), language? (en|multi — send multi for German or omit and let langdetect route), score_threshold? (0–1, overrides tenant/env gate), min_results? (0–50), max_per_document? (0–50).

D.2 Response

Every /api/v1/retrieval/* response is wrapped in a RetrievalEnvelope:

{
  "endpoint": "search/rerank",
  "status_code": 200,
  "processing_time_ms": 412,
  "data": {
    "hits": [
      {
        "id": "chunk-...",
        "score": 3.72,
        "rerank_score": 3.72,
        "rerank_score_norm": 0.87,
        "rank": 1, "bm25_rank": 3, "dense_rank": 1,
        "payload": {
          "doc_id": "doc-123",
          "ordinal": 17,
          "text": "…max. 800-char preview…",
          "vector_upsert_id": "CAS_v1:vector_upsert:..."
        }
      }
    ],
    "rerank": {
      "stages": {"bm25_ms": 4, "rrf_ms": 1, "rerank_ms": 210},
      "model_language": "multi",
      "used_dense_scores": true,
      "elapsed_ms": 240,
      "retrieval_confidence": "strong",
      "dropped_below_threshold": 2,
      "score_threshold": 0.15,
      "max_per_document": 0,
      "dropped_for_per_document_cap": 0
    }
  },
  "raw": {}
}

Notes an implementer needs:

• The hit list rides under whichever key voyager used (hits/points/results) — the route preserves it, falling back to hits. Read defensively.
• retrieval_confidence ∈ strong|weak|none|unknown (ADR 0032). Use it to caveat or withhold answers; the gate trims sources but never blocks.
• On rerank-fusion outage the route degrades (never fails): data.rerank = {"degraded": true, "reason": "...", "retrieval_confidence": "unknown"} with plain voyager hits.
• payload.text is a ≤800-char preview — resolve full passages via §I.4.
• page_numbers is never populated by any chunking strategy — do not promise page-level citations (plan 5.2 renders doc name + section hierarchy instead).
• Errors: voyager 4xx forwarded verbatim; voyager 5xx → 502; 404 also means the collection does not exist yet (no ingest ran).

D.3 Collection info / existence check

curl -sS "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/kb-analysis-a01/info" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
# 404 → collection missing. The BFF ingest lane creates it; manual create:
curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/kb-analysis-a01" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"engine": "dense", "dim": 1536}'      # 409 = already exists = OK

With TENANT_KEYS_ENABLED=false (prod example) collection names are not tenant-prefixed — isolation rests on the KB-id naming discipline; the gateway rewrites names transparently once the flag flips (plan 1.5).

E. Conversations (workspaces service)

⚠ Create-first idiom (plan 5.2, review finding): the chat BFF does not create conversations. If you stream against a made-up conversation_id, the stream works but every transcript persist silently fails (the BFF treats history-404 as "new conversation" and swallows persist errors). Therefore: create the conversation with a client-chosen id first, store it in grag_refs, then chat.

E.1 Create (before the first turn)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/conversations" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "conv-analysis-a01-file-001",
    "kb_id": "kb-analysis-a01",
    "workspace_id": "client-hartmann-x7k2",
    "title": "Frag die Akte — Analyse a-01",
    "metadata": {"knoll_user_id": "u-42"}
  }'

ConversationCreate (verified): id? (auto-gen conv-<8 urlsafe chars> when absent — Knoll always sets it), workspace_id? (defaults to the KB's workspace; mismatch → 400), project_id?, kb_id (required), title (default "New conversation"), system_prompt_override (default ""), metadata{}. 409 duplicate id = already created = OK. Per-user attribution goes into metadata — GRAG has no user model.

Lists/CRUD: GET /workspaces/api/v1/conversations?workspace_id=&kb_id=, GET/PATCH/DELETE /workspaces/api/v1/conversations/{id}.

E.2 Messages: list (history) and append

# Newest-N window, the same read the chat BFF uses:
curl -sS "$GRAG_URL/workspaces/api/v1/conversations/conv-analysis-a01-file-001/messages?order=desc&limit=20" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
# → {"items":[{"id":"...","role":"assistant","content":"...","sources":[...],
#              "trace":[...],"groundedness":{...},"cost":{...}}, ...]}   404 if conversation missing

# Manual append (the Gutachten pipeline can persist its own turns this way):
curl -sS -X POST "$GRAG_URL/workspaces/api/v1/conversations/conv-analysis-a01-file-001/messages" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"role": "user", "content": "Wie ist die Marktposition?", "sources": [], "trace": []}'

MessageCreate (verified): id?, role ∈ user|assistant|system, content, sources[] (free-shape dicts), trace[], groundedness?, cost?. limit is 1–1000.

E.3 Feedback (thumbs on assistant messages)

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/conversations/conv-analysis-a01-file-001/messages/msg-9/feedback" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"rating": "down", "reason": "ungrounded", "comment": "Zahl stimmt nicht mit dem Jahresabschluss überein"}'

rating ∈ up|down; reason? ∈ incorrect|incomplete|ungrounded|outdated|off_topic|other; comment? ≤2000 chars (the only free text). Re-POST flips the rating idempotently. DELETE the same path retracts; GET /conversations/{cid}/feedback returns the map for reload. Feedback feeds the quality rollups (GET /workspaces/api/v1/quality/summary).

F. Chat BFF: POST /next/api/chat/stream (SSE)

The one blessed conversational lane (plan D3) for KB-Chat (5.2), Akquise chat (5.1a) and KIU chat (5.1b). It composes history → condense → retrieve → expand (KG full text) → generate → ground, and persists both turns. ⚠ Unauthenticated until plan 1.8. Full integration guidance in 09-chat-integration.md.

F.1 Request

curl -N -X POST "$GRAG_URL/next/api/chat/stream" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "conversation_id": "conv-analysis-a01-file-001",
    "kb_id": "kb-analysis-a01",
    "user_message": "Welche Risiken nennt der Jahresabschluss?",
    "model": "openai/gpt-4o-mini",
    "top_k": 10,
    "system_prompt": "Du bist der KIU-Assistent der Kanzlei Knoll...",
    "workspace_id": "client-hartmann-x7k2"
  }'

Body (verified in route.ts): conversation_id (must exist — §E.1), kb_id, user_message, model?, top_k?, system_prompt?, workspace_id?. Always send workspace_id — without it the KG full-text expand stage degrades to 800-char previews. The response is an SSE-framed stream over a POST fetch — consume with fetch(...).body.getReader(), not EventSource (which cannot POST).

F.2 Complete SSE event sequence (in order)

event: subscribed    data: {"conversationId": "...", "model": "openai/gpt-4o-mini"}

event: trace         data: {"stage": "history",      "status": "done", "ms": 12}
event: trace         data: {"stage": "persist_user", "status": "done"}
event: trace         data: {"stage": "condense",     "status": "done", "tokens": 96}
event: trace         data: {"stage": "retrieve",     "status": "done", "hits": 10, "confidence": "strong"}
event: trace         data: {"stage": "expand",       "status": "done"}
event: trace         data: {"stage": "generate",     "status": "running"}
                     # status ∈ running|done|failed|skipped; every stage except
                     # generate fails OPEN (chat continues without that stage)

event: sources       data: [{"chunkId": "...", "docId": "doc-123", "ordinal": 17,
                             "span": "…full chunk text…", "page": null, "score": 0.87,
                             "rank": 1, "bm25Rank": 3, "denseRank": 1, "rerankScore": 3.72,
                             "sectionHierarchy": ["2 Lagebericht", "2.3 Risiken"],
                             "docName": "jahresabschluss-2025.pdf", "docDate": null,
                             "contextBefore": "…", "contextAfter": "…",
                             "vectorUpsertId": "CAS_v1:vector_upsert:..."}]

event: retrieval     data: {"confidence": "strong"}          # strong|weak|none|unknown

event: token         data: {"delta": "Die "}                 # ×N — fake-tokenised ~30 ms/frame;
                                                             # upstream completion is NON-streaming

event: groundedness  data: {"score": 0.83, "band": "high", "coverage": 0.7,
                            "unused": 0.2, "nliGroundedness": 0.91}
                     # NB: the BFF band vocabulary is high|medium|low|unknown; the
                     # groundedness SERVICE (§H) returns green|amber|red|unknown —
                     # do not conflate the two.

event: sentences     data: [ ...PerSentence... ]             # when available
event: spans         data: [ ...ChunkSpans... ]              # English-only; absent for German
event: enforcement   data: {"action": "banner", "band": "low"}   # only if tenant setting
                                                             # groundedness.enforcement != off

event: error         data: {"message": "..."}                # FATAL — only the generate step
event: done          data: {"messageId": "msg-...", "cost": {"tokens": 512, "ms": 4100}}

Persistence: the user message is stored before any LLM spend; the assistant message is stored with sources/trace/groundedness/cost nested. page is never populated. Timeouts: completion 120 s, groundedness 30 s inside the BFF.

F.3 Full chunk text for the sources drawer (plan 6.5)

curl -sS "$GRAG_URL/next/api/chat/chunk/doc-123/17?workspace=client-hartmann-x7k2" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

(BFF proxy onto kg-service segment text — §I.4 is the direct equivalent.) The workspace query param is mandatory — 400 when missing; it becomes the kg-service X-Workspace-ID and must be the workspace the document was ingested under (§C.2).

G. LLM: chat/completions, embed, models (ai-gateway)

The single LLM egress for all Gutachten-pipeline steps (plan 5.3), lead scoring (5.1a), Projektgenerator (5.5). Governed by keyvault (§K) and recorded in the ledger (§L) — stamp X-Pipeline-Id on every call (plan 7.3).

G.1 Chat completion (non-streaming; with fallback chain)

curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/chat/completions" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Pipeline-Id: run-a01-gutachten-001" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [
      {"role": "system", "content": "Du bist Analyst der Kanzlei Knoll."},
      {"role": "user", "content": "Fasse die SWOT-Erkenntnisse zusammen: ..."}
    ],
    "temperature": 0.2,
    "max_tokens": 2000,
    "fallback": ["anthropic/claude-sonnet-4-5", "openai/gpt-4o-mini"],
    "fallback_policy": "retriable_only"
  }'

CompletionRequest (verified, complete field list): model?, messages (required, min 1), temperature? (0–2), max_tokens?, top_p?, stop?, BYOK api_key?/api_base?/ api_version?, cache?, fallback? (≤5 models), fallback_policy ∈ any_error|retriable_only|rate_limit_only (default retriable_only).

⚠ There is NO structured output: no response_format, no tools, no JSON mode, no stream (platform gap, plan 1.14; the /api/v1/responses shim adds nothing). Success:

{
  "provider": "openai",
  "model": "gpt-4o",
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 812, "completion_tokens": 496, "total_tokens": 1308},
  "processing_time_ms": 3120,
  "fallback_triggered": false,
  "attempted_models": [],
  "fallback_reason": null
}

fallback_triggered/attempted_models/fallback_reason are populated only when the chain ran. Errors: 402/429/403 keyvault governance (§K), 400 guardrail block (only if enabled — off in prod example), 503 fail-closed safety outage.

G.2 The prompt-JSON + validation idiom (until plan 1.14 lands)

Every "structured output" step (checklist classification 4.3, lead scoring 5.1a, Scoring 5.3.4, Projektsteckbrief 5.5) uses this pattern in grag-client/ai_runs:

1. Prompt for pure JSON. System prompt ends with an explicit contract, e.g.: Antworte AUSSCHLIESSLICH mit einem JSON-Objekt nach diesem Schema, ohne Markdown, ohne Codeblock: {"hebel": string, "score": number (1.0-5.0), "begruendung": string} (Hebel-Skala 1,0–5,0, 5 = beste Bewertung — ADR-013). Keep temperature low (0–0.2).
2. Parse defensively. Strip accidental code fences, then JSON.parse.
3. Validate with Zod against the step's schema (ranges, enums, required keys).
4. Bounded retry (≤2). On parse/validation failure, re-call with the error appended: {"role":"user","content":"Deine letzte Antwort war kein gültiges JSON nach Schema. Fehler: <zod message>. Antworte erneut, NUR JSON."} — then fail the ai_runs step loudly (status failed, error persisted) rather than looping.
5. Never trust numbers for money/scores — deterministic caps and Ampel mapping happen in Knoll code after validation (plan 5.3.4, 5.5).

G.3 Embeddings

curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/embed" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"input": ["Erfolgshebel Marktposition: ..."], "model": "openai/text-embedding-3-small"}'

EmbedRequest (verified): input (string or list, required), model?, provider?, dimensions?, input_type?, task?, encoding_format?, truncate (default true), BYOK fields, cache?. Response: {provider, model, dimensions, embeddings: [[...]], usage: {prompt_tokens, total_tokens}, processing_time_ms, source_content_hash}. Also /embed/batch ({items:[EmbedRequest]}) and /embed/async → poll /embed/jobs/{id}. Note: KB ingest embeds server-side automatically — Knoll only calls /embed directly for its own vectors (rare).

G.4 Model catalog (feeds the Einstellungen model picker, plan 6.6)

curl -sS "$GRAG_URL/ai-gateway/api/v1/models" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Returns the LiteLLM catalog with pricing, context windows, capabilities, and provider availability (derived from which provider keys are configured). GET /providers lists providers. Defaults when model omitted: completion openai/gpt-4o-mini, embedding openai/text-embedding-3-small.

H. Groundedness: /score and /relevance

Answer-QA for every chat turn (the BFF calls it for you) and every Gutachten pipeline step (plan 5.3 calls it directly). Multilingual except evidence spans.

H.1 Score an (answer, chunks) pair

curl -sS -X POST "$GRAG_URL/groundedness/api/v1/score" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Pipeline-Id: run-a01-gutachten-001" \
  -H "Content-Type: application/json" \
  -d '{
    "response_text": "Die Marktposition ist mit 2,5 zu bewerten, weil ...",
    "chunks": [
      {"chunk_id": "chunk-1", "text": "…voller Passagentext aus dem Retrieval…"},
      {"chunk_id": "chunk-2", "text": "…"}
    ],
    "question": "Wie ist die Marktposition zu bewerten?",
    "include_per_sentence": true,
    "include_nli": true,
    "include_spans": false
  }'

For German always include_spans: false — spans are English-only (GROUNDEDNESS_SPAN_LANGS=en); German input returns spans: null + note spans_unavailable_lang. include_nli and include_per_sentence ARE multilingual and are Knoll's standard opt-ins (plan 5.3). Success (abbreviated):

{
  "score": 0.82,
  "band": "green",
  "nli_groundedness_score": 0.91,
  "context_coverage_ratio": 0.7,
  "context_unused_ratio": 0.2,
  "per_chunk": [{"chunk_id": "chunk-1", "coverage": 0.8, "maxsim": 0.74, "used": true}],
  "per_sentence": [{"text": "...", "band": "green", "supporting_chunk_ids": ["chunk-1"]}],
  "spans": null,
  "revision": "…",
  "elapsed_ms": 640,
  "notes": ["spans_unavailable_lang"]
}

band ∈ green|amber|red|unknown (service vocabulary; the chat BFF re-emits high|medium|low|unknown — see §F.2). Empty chunks → band: "unknown". Band cutoffs (0.8/0.5) are uncalibrated placeholders — no user-facing enforcement before the German calibration (plan 7.1). First call after a deploy can take 60–120 s unless GROUNDEDNESS_PRELOAD=true (plan 1.4). Batch: POST /groundedness/api/v1/score/batch {items:[ScoreRequest]}.

H.2 Relevance pruning (multilingual — the German substitute for spans)

curl -sS -X POST "$GRAG_URL/groundedness/api/v1/relevance" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"question": "Wie ist die Marktposition?", "passages": ["…", "…"], "threshold": 0.0}'

Returns per-passage {maxsim, band, kept} — scores and flags only, never drops; the caller prunes. threshold default 0.0 = everything kept. Per-sentence variant: POST /groundedness/api/v1/relevance/sentences (or include_sentences on /relevance). Capabilities probe before relying on opt-ins: GET /groundedness/api/v1/capabilities.

I. kg-service: schemas, entities, segments, entity-match, intents

⚠ Every non-public kg-service route requires X-Workspace-ID (400 without it), on top of bearer + tenant. Entities are tenant-wide (visible across all Mandant workspaces of the Kanzlei — documented trade-off, plan D1). Knoll's KG design lives in 07-kg-schema-knoll-advisory.md.

I.1 Register + activate the knoll-advisory schema (plan 3.4)

curl -sS -X POST "$GRAG_URL/kg-service/api/v1/schemas" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: general" \
  -H "Content-Type: application/json" \
  -d '{
    "schema_version": "1.0",
    "name": "knoll-advisory",
    "version": 1,
    "description": "Knoll Analyzer advisory ontology v1 (no relations, see D7)",
    "based_on": "",
    "license_attribution": "",
    "ner_labels": ["Client", "SuccessLever", "Recommendation", "ServiceProvider", "Project", "Metric"],
    "ner_thresholds": {"default": 0.5, "per_label": {}},
    "entity_types": {
      "SuccessLever": {
        "description": "One of the 11 success levers of the Knoll methodology, e.g. MarketPosition",
        "embedding_template": "SuccessLever: {label}. {description}",
        "candidate_search": {"fuzzy": true, "min_similarity": 0.75}
      }
    },
    "relationships": [],
    "templates": []
  }'

⚠ v1 ships with an empty relationships block — that is the enforcement mechanism keeping prod's KG_RELATION_EXTRACTION_ENABLED=true dormant for Knoll workspaces (plan D7/3.4): relation auto-extraction only fires when the active schema declares relationships. Versions are immutable — changes = register version 2 + re-activate. Errors: 201 + Location header on success, 409 duplicate version, 422 internal inconsistency (entity_type not in ner_labels, unknown relationship endpoints).

Activate per workspace (on every Mandant provisioning, plan 3.1):

curl -sS -X POST "$GRAG_URL/kg-service/api/v1/schemas/knoll-advisory/v1/activate" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: client-hartmann-x7k2" \
  -H "Content-Type: application/json" \
  -d '{"workspace": "client-hartmann-x7k2"}'

Workspace existence is NOT validated. Read back: GET /kg-service/api/v1/workspaces/{ws}/schema.

I.2 Entity upsert (seed taxonomies, plan 3.5; deterministic facts, plan 3.6)

curl -sS -X POST "$GRAG_URL/kg-service/api/v1/entities/upsert" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: general" \
  -H "Content-Type: application/json" \
  -d '{
    "entities": [
      {
        "entity_id": "lever:market-position",
        "entity_type": "SuccessLever",
        "label": "MarketPosition",
        "aliases": ["Market", "Competitive position"],
        "description": "Assessment of the company's position in the market (1,0–5,0)",
        "properties": {"order": 1, "status": "active"}
      }
    ]
  }'

1–1000 items per call → {"inserted": n, "updated": m}. Upserts are idempotent — the plan-3.6 reconcile job simply re-upserts everything for an Analyse.

⚠ There is NO entity delete route (verified: routes are upsert/get/neighborhood/ documents/search only). Removal = tombstone: upsert with "properties": {"status": "inactive"} and filter status != "inactive" in every consumer (plan 3.5). ⚠ Also verified: the HTTP POST /kg-service/api/v1/documents/ingest body has no entities[]/relationships[] fields (extra="forbid") — deterministic edge writes over HTTP need the new platform route from plan 1.11; until then only entities (not edges) are writable via HTTP.

I.3 Entity read routes

# Hybrid label+alias search (candidate lookup):
curl -sS -X POST "$GRAG_URL/kg-service/api/v1/entities/search" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: client-hartmann-x7k2" \
  -H "Content-Type: application/json" \
  -d '{"entity_type": "ServiceProvider", "query": "Vertriebsberatung", "max_candidates": 20, "fuzzy": true, "min_similarity": 0.7}'

# One entity / graph neighborhood / reverse document index:
curl -sS "$GRAG_URL/kg-service/api/v1/entities/lever:market-position" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" -H "X-Workspace-ID: general"
curl -sS "$GRAG_URL/kg-service/api/v1/entities/lever:market-position/neighborhood?depth=2" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" -H "X-Workspace-ID: general"
curl -sS "$GRAG_URL/kg-service/api/v1/entities/lever:market-position/documents" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" -H "X-Workspace-ID: general"

depth is clamped to 5; /documents caps at ≤200 docs / ≤50 ordinals each with a truncated flag.

I.4 Segment full text (batch) — the citation resolver

Resolves voyager's ≤800-char preview into the complete passage by (document_id, ordinal). Used by Gutachten step 2 (plan 5.3) and the sources drawer.

curl -sS -X POST "$GRAG_URL/kg-service/api/v1/segments/text:batch" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: client-hartmann-x7k2" \
  -H "Content-Type: application/json" \
  -d '{"keys": [{"document_id": "doc-123", "ordinal": 17}, {"document_id": "doc-123", "ordinal": 18}]}'

≤64 keys; found-only response — missing keys are silently absent, not errors. ⚠ The X-Workspace-ID must match the workspace the document was ingested under (the workspace_id you passed in §C.2). 400 without the header; single-segment variant: GET /kg-service/api/v1/documents/{id}/segments/{ordinal}/text (404 no-leak on miss).

I.5 Semantic entity match (pgvector) — Dienstleister discovery (plan 5.4)

curl -sS -X POST "$GRAG_URL/kg-service/api/v1/search/entity-match" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: client-hartmann-x7k2" \
  -H "Content-Type: application/json" \
  -d '{"query": "Beratung für Vertriebsaufbau im Maschinenbau", "entity_type": "ServiceProvider", "top_k": 10, "include_neighbors": false}'

query ≤2000 chars, top_k ≤50. Success: {"matches": [{"entity_id": "service-provider:...", "label": "...", "score": 0.81, "neighbors": null}], "model": "openai/text-embedding-3-small", "dimensions": 1536}. 503 unless KG_EMBEDDINGS_ENABLED=true (true in the prod env example — verify live per plan 0.9). Filter tombstoned candidates (properties.status == "inactive") client-side.

I.6 Cypher intents — register with UNTYPED params, execute with canonical ids

⚠ Params declared with an entity_type are always resolved through entity-linking (free text → entity), whose schemas are image-baked — that path 502s/422s for Knoll until plan 1.12. Knoll registers all params untyped (entity_type: null) and passes canonical ids from its own DB (plan 3.7). Register only when a real graph consumer exists (3.8 Explorer).

# Register (once, per schema version):
curl -sS -X POST "$GRAG_URL/kg-service/api/v1/intents" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: general" \
  -H "Content-Type: application/json" \
  -d '{
    "schema_name": "knoll-advisory",
    "schema_version": 1,
    "name": "recommendations_for_client",
    "version": 1,
    "description": "All recommendations for a client (1-hop)",
    "cypher": "MATCH (m {entity_id: $p1})<-[:CONCERNS]-(e) RETURN e",
    "parameters": [
      {"name": "client_id", "position": 1, "entity_type": null, "required": true}
    ],
    "returns": {"e": "Recommendation"},
    "max_rows": 100,
    "max_depth": 3
  }'
# 422 if the Cypher contains write keywords (CREATE/DELETE/SET/REMOVE/MERGE); 409 on version conflict.

# Execute:
curl -sS -X POST "$GRAG_URL/kg-service/api/v1/intents/recommendations_for_client/execute" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: client-hartmann-x7k2" \
  -H "Content-Type: application/json" \
  -d '{"params": {"client_id": "client:hartmann-maschinenbau"}}'

Execute success: {"intent": "...", "resolved_params": {...}, "rows": [...], "row_count": n, "max_rows_applied": 100}. max_rows_applied is int or null — the intent's max_rows when the service auto-appended a LIMIT, null when the template already carried one. Caps: 5 s statement timeout, LIMIT auto-appended, ≤1000 rows / depth ≤5 service-wide. Errors: 422 unresolvable typed param (should never happen with untyped params), 502 entity-linking down (ditto), 404 unknown intent.

J. Tenant settings (workspaces service, ADR 0030)

Per-tenant behaviour knobs (plan 1.7); read-through in the Einstellungen KI tab (plan 6.6). 503 on every route while TENANT_SETTINGS_ENABLED=false (true in the prod example).

# Current overrides for this tenant:
curl -sS "$GRAG_URL/workspaces/api/v1/settings" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

# Typed catalog of every known key (validation source of truth):
curl -sS "$GRAG_URL/workspaces/api/v1/settings/registry" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

# Set one key:
curl -sS -X PUT "$GRAG_URL/workspaces/api/v1/settings/chat.default_model" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"value": "openai/gpt-4o-mini"}'

# Revert to platform default:
curl -sS -X DELETE "$GRAG_URL/workspaces/api/v1/settings/chat.default_model" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

# Audit trail:
curl -sS "$GRAG_URL/workspaces/api/v1/settings/audit?key=chat.default_model" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Keys Knoll sets at baseline (plan 1.7): chat.default_model, chat.default_top_k, chunking.default_*, ingest.dag_template, retrieval.score_threshold, frontend.default_locale. Errors: 400 invalid value (registry-validated), 403 on security-sensitive keys (guardrail/firewall/anonymization — admin key required, i.e. operator action after the 1.2 custody split), 503 feature dark. Absence of an override = env default. Note frontend.modules is cosmetic nav-hiding, NOT authorization.

K. BYOK keyvault (ai-gateway) + governance errors

Stores the Kanzlei's LLM provider keys with hard budget/rate governance (plan 1.6). 503 {"detail": "key vault disabled"} while GATEWAY_KEYVAULT_ENABLED=false (true in prod example).

# Create a governed provider key:
curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/keys" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "openai",
    "secret": "sk-proj-...",
    "label": "Kanzlei Knoll OpenAI",
    "allowed_services": [],
    "scope_level": "tenant",
    "monthly_budget_usd": 200,
    "rate_limit_rpm": 60,
    "enabled": true
  }'

KeyCreate (verified): provider (must be in GET /keys/catalog), secret (write-only, never returned), label, allowed_services[], scope_level ∈ tenant|workspace|project, workspace_id?, project_id?, enabled, monthly_budget_usd?, rate_limit_rpm?, expires_at?. Success 201 returns a KeySummary (id, provider, label, last4 hint, governance fields, live usage: {month, usd, requests}).

Other routes: GET /ai-gateway/api/v1/keys (list, never secrets), GET /ai-gateway/api/v1/keys/catalog (provider/service picker), GET/PATCH /keys/{id} (governance updates), POST /keys/{id}/rotate {"secret": "sk-..."}, DELETE /keys/{id} → 204.

Governance enforcement happens on /chat/completions, /embed, /rerank whenever the request carries no per-request api_key and a stored key matches (provider/service/scope — scope matching reads X-Workspace-ID/X-Project-ID if forwarded):

Knoll treats 402 as "pause all AI runs for the tenant + alert" (plan 7.3: alert at 80 % via the ledger, before the hard 402 ever triggers).

L. Ledger: spend and per-run totals

⚠ Paths (review-corrected): the service's internal prefix is /api/v1/ledger/... behind the Traefik prefix /ledger, so externally the segment appears twice.

# Tenant-level spend rollup (Einstellungen budget display, plan 6.6).
# group_by accepts ONLY service | provider — there is no per-pipeline grouping here:
curl -sS "$GRAG_URL/ledger/api/v1/ledger/spend?group_by=service" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

{
  "tenant": "knoll",
  "group_by": "service",
  "since": null,
  "until": null,
  "rows": [
    {"dimension": "ai-gateway", "calls": 412, "units": 913000, "usd": 12.41, "failed_calls": 3}
  ]
}

# Per-run totals — pairs with the X-Pipeline-Id header you stamp on every
# ai-gateway/groundedness call of a run (plan 7.3: one id per Analyse-Run):
curl -sS "$GRAG_URL/ledger/api/v1/ledger/totals?pipeline_id=run-a01-gutachten-001" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

{
  "pipeline_id": "run-a01-gutachten-001",
  "calls": 18,
  "failed_calls": 0,
  "total_tokens": 96500,
  "total_cost_usd": 0.83,
  "max_latency_ms": 9100,
  "avg_latency_ms": 2900
}

Raw entries with filters: GET /ledger/api/v1/ledger?pipeline_id=&job_id=&service=&provider=&since=&until=&limit= (epoch-seconds floats, limit ≤1000). ?tenant= cross-tenant queries require the ledger admin key (403 otherwise) — not a Knoll-runtime capability. Attribution rule for grag-client: every LLM/retrieval/scoring call inside an ai_runs step sends X-Pipeline-Id: <run-id> (and optionally X-Job-Id); calls without it are only visible in the tenant rollup.

M. Lineage: provenance + GDPR cascade delete

Provenance backbone (source → docfold → chunk → vector_upsert per document) and the forward-delete lane for DSGVO-Löschung (plan 3.2, 7.5, 12-gdpr-compliance.md).

# Where did this chunk come from? (walk up from a citation's vector_upsert_id):
curl -sS "$GRAG_URL/lineage/api/v1/artifact/CAS_v1:vector_upsert:8f3a.../ancestors?depth=5" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

# What was derived from this source? (forward impact before deleting):
curl -sS "$GRAG_URL/lineage/api/v1/artifact/CAS_v1:source:1b2c.../descendants?depth=10" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

# GDPR cascade delete (forward-delete of the artifact and everything derived):
curl -sS -X DELETE "$GRAG_URL/lineage/api/v1/artifact/CAS_v1:source:1b2c...?cascade=true" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Traversal caps: ≤10 hops / ≤5000 rows. Ids are content-addressed (CAS_v1:{type}:{hash}); the bridge from the workspaces inventory is documents.docfold_artifact_id, from retrieval hits payload.vector_upsert_id.

⚠ The cascade is soft-mark unless the purge flags are on: physical removal of voyager points and document rows needs LINEAGE_CLEANUP_ENABLED=true + WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4 — blocking for real data; the plan-7.5 probe verifies voyager points actually disappear). A complete Löschung additionally deletes the Knoll file-store original and Knoll-DB rows — GRAG deletion alone is never sufficient (see 12-gdpr-compliance.md).

N. Master error taxonomy (status → meaning → client behavior)

grag-client maps every non-2xx onto this table (plan 2.5; policy detail in 11-resilience-and-errors.md). Bodies are {"detail": "..."}.

Additional non-error semantics to code for: evidence gate trims, never blocks; rerank outage degrades (data.rerank.degraded=true), never 5xx; segments/text:batch returns found-only (missing ≠ error); first groundedness/rerank call after deploy can take 60–120 s (cold start) unless preloads are set (plan 1.4).

O. Headers cheat-sheet

Cross-references: 01-architecture.md (lane diagram), 03-id-conventions.md (slugs, shortids), 04-provisioning-runbook.md (A applied end-to-end), 05-verification-runbook.md (live probes incl. the 1.8 auth probe), 08-gutachten-pipeline-spec.md (D+G+H composed), 09-chat-integration.md (E+F composed), 10-document-pipeline.md (B+C composed), 11-resilience-and-errors.md (N as policy), 12-gdpr-compliance.md (M applied), 13-platform-gaps-issues.md (1.8 BFF auth, 1.11 KG edge writes, 1.14 structured output), 14-cost-model.md (L applied).