05 — Phase 0 Live-State Verification Runbook (app.grag.ai)

Purpose. Before Knoll writes any real Mandanten data through GRAG, an operator must prove that the live platform on https://app.grag.ai is in the state the integration plan assumes. Compose defaults, the .env.production.example baseline, and the actual live container environment are three different things (the plan calls this out in 0.9). This runbook is the numbered, copy-pasteable checklist that turns every "we assume X" in the plan into an observed fact — feature flags (plan 0.9), the unauthenticated-BFF blocker (plan 0.10a / 1.8), SSE buffering through Plesk nginx (plan 0.10b), the retrieval profile, model cold-start latency, canary sanity, and docs reachability. Each item states what to check, the exact command, the GOOD output, and what a BAD output means for the plan.

Status / verified against. 2026-07-06, repo document-processing-pipelines @ db63a95. All endpoint paths, flag names, and compose defaults below were read from that revision. Items that cannot be settled from source alone (i.e. the live value) carry an explicit TODO-VERIFY: marker — those are exactly the reason this runbook exists.

Sibling docs. Provisioning steps live in 04-provisioning-runbook.md; the raw request shapes in 02-grag-api-cookbook.md; id conventions in 03-id-conventions.md; the known platform gaps this runbook probes for in 13-platform-gaps-issues.md; error handling in 11-resilience-and-errors.md; GDPR preconditions in 12-gdpr-compliance.md.

Conventions for every command in this runbook

External base-URL pattern (Plesk TLS → Traefik strip-prefix → service):

https://app.grag.ai/<traefik-prefix>/<service-native-path>

Set these once in your shell. Use the operator/admin key only where a step says so; everywhere else use the Knoll runtime key.

export GRAG_URL="https://app.grag.ai"
export GRAG_API_KEY="<knoll-runtime-key>"        # entry in PIPELINE_API_KEY / GATEWAY_CALLERS
export GRAG_ADMIN_KEY="<operator-admin-key>"     # WORKSPACES_ADMIN_API_KEY holder (plan 1.2)
export GRAG_TENANT="knoll"                        # prod tenant; use knoll-dev for dev checks

Every GRAG request MUST carry both headers below (kg-service additionally needs X-Workspace-ID; the plan makes the missing-tenant case hard-fail, because the platform silently falls back to tenant default — CLAUDE.md "Multi-tenancy"):

Authorization: Bearer $GRAG_API_KEY
X-Tenant-ID: $GRAG_TENANT

Two of the checks below run on the host (worker flags, container presence), not over HTTP. Workers (kg-service-worker, orchestrator-worker, …) expose no HTTP surface, so a healthy-but-misconfigured worker is invisible from outside — that is precisely why host-diagnose.yml exists (it bit the platform team twice on 2026-07-05). Run it with:

gh workflow run host-diagnose.yml \
  -f container=<name-substring> -f tail=120 \
  --repo MR-Eder/document-processing-pipelines
# then read the run log:
gh run watch --repo MR-Eder/document-processing-pipelines

host-diagnose.yml prints, per matching container: docker ps status, tailed logs, and a filtered env dump (only names matching _(ENABLED|MODE|BACKEND|URL|DBS|MAX_*|BATCH_SIZE)=, with anything key/secret/token/password/dsn/master-shaped excluded). It cannot leak secrets and cannot mutate the host. An operator with shell on the box can equivalently run docker compose ps and docker exec <c> env | grep _ENABLED=.

Section A — Feature-flag state (plan 0.9)

The plan's contract needs specific flags in specific states. The prod .env example values are known (see the "example" column); the live values are not — verify each. Method per row: HTTP probe where a flag has an observable 503/behaviour, otherwise the host-diagnose env dump on the owning container.

A0 — Flag summary (what the plan requires vs. the committed example)

⚠ The four flagged rows are the review-corrected traps: relation-extraction is ON in the example (A6), tenant-keys is OFF (A11), and the two GDPR purge/cleanup flags are unset in the example so they fall to the compose default false (A12/A13 — verified at docker-compose.yml:3731 and :5177).

A1 — Tenant settings store is live

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/workspaces/api/v1/settings/registry"

• GOOD: 200. TENANT_SETTINGS_ENABLED=true live; plan 1.7 baseline writes will work.
• BAD: 503 ("tenant settings disabled", verified settings.py:30) → the flag is off live. Blocks 1.7, plus per-tenant retrieval/groundedness tuning (7.1). Operator must set TENANT_SETTINGS_ENABLED=true and redeploy.

A2 — BYOK key vault is live

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/ai-gateway/api/v1/keys"

• GOOD: 200 (empty list is fine).
• BAD: 503 ("key vault disabled — set GATEWAY_KEYVAULT_ENABLED=true", verified keys.py:30-32) → blocks 1.6 budget/RPM governance; without it there is no hard 402/429 cost cap. Operator flips GATEWAY_KEYVAULT_ENABLED=true + runs the migrate one-shot.

A3 — Lineage emission on

Host check (no HTTP behaviour distinguishes on/off cheaply):

gh workflow run host-diagnose.yml -f container=lineage-api -f tail=40 \
  --repo MR-Eder/document-processing-pipelines

• GOOD: env dump shows LINEAGE_ENABLED=true.
• BAD: false or absent → provenance edges (source→docfold→chunk→vector_upsert) are not written; breaks GDPR forward-delete (3.2) and citation lineage (5.2). Blocking for real data.

A4/A5/A6 — KG worker extraction flags (one host call)

gh workflow run host-diagnose.yml -f container=kg-service-worker -f tail=60 \
  --repo MR-Eder/document-processing-pipelines

Look in the env dump for the in-container KG_SERVICE_* names (compose maps the operator vars onto them — verified docker-compose.yml:5091/5106/5114):

• GOOD: KG_SERVICE_INGEST_ENABLED=true, KG_SERVICE_ENTITY_EXTRACTION_ENABLED=true, and KG_SERVICE_RELATION_EXTRACTION_ENABLED=false.
• BAD (A6 is the trap): KG_SERVICE_RELATION_EXTRACTION_ENABLED=true → un-harnessed German relation auto-extraction will fire the moment a workspace activates a schema that declares relationships. The plan's mitigation (D7 / plan 3.4) is to ship knoll-advisory v1 without a relationships block so extraction cannot fire even with the flag on; belt-and-braces is to also flip the flag off (plan 1.4). If this reads true live, do NOT activate any relationship-declaring schema until 7.1 calibration.
• BAD (A4/A5): either false → no kg segments (full-text re-fetch 5.2/5.3.2 returns 404) or no document entities. Flip KG_INGEST_ENABLED / KG_ENTITY_EXTRACTION_ENABLED.

A7 — KG entity-match (pgvector) reachable

curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Workspace-ID: general" -H "Content-Type: application/json" \
  -d '{"query":"probe","top_k":1}' \
  "$GRAG_URL/kg-service/api/v1/search/entity-match"

(Body shape verified kg-service/.../schemas/search.py: query required, entity_type optional, top_k 1–50, include_neighbors.)

• GOOD: 200 (empty matches fine) → KG_EMBEDDINGS_ENABLED=true live; 5.4 semantic discovery works.
• BAD: 503 → embeddings off; 5.4 falls back to Knoll-SQL-only matching (acceptable degradation, note it). 400 "X-Workspace-ID required" means you omitted the header, not a flag problem.

A8 — Graph gateway (only needed for 3.8 Explorer)

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/graph-gateway/api/v1/graph/capabilities"

• GOOD: 200 → GRAPH_GATEWAY_ENABLED=true; the Knowledge-Graph Explorer (3.8) has a backend.
• BAD: 503 → dark. Not blocking for pilot (3.8 is "later"); record and defer.

A9 — Agent-control: flag ON and the profile actually running

This is a two-part check — the plan explicitly flags that the deploy workflow enables the flag but does not start the agent-control profile (LAP/hermes are profile-gated, verified docker-compose.yml:6598/6617/6656/6679).

Part 1 (control-plane API up):

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/agent-control/api/v1/agents"

Part 2 (LAP black-boxes present):

gh workflow run host-diagnose.yml -f container=lap -f tail=20 \
  --repo MR-Eder/document-processing-pipelines

• GOOD: Part 1 → 200; Part 2 → docker ps lists lap, hermes, lap-postgres as running/healthy.
• BAD: Part 1 503 → AGENT_CONTROL_ENABLED off. Part 1 200 but Part 2 "no match" → the exact plan trap: the control plane answers but has no runtime behind it, so 5.6's multi-turn agent spike will fail at prompt time. Agents (5.6/5.7) are optional — record the state and keep 5.x on the scripted-BFF chat path.

A10 — MCP gateway (optional, 5.7)

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/mcp-gateway/api/v1/tokens"

(Token routes are admin-gated — use the admin key; see plan 1.2 / 5.7.)

• GOOD: 200 → MCP_GATEWAY_ENABLED=true; scoped tokens can be minted (operator action per 5.7).
• BAD: 503 dark, or 401/403 with the admin key → custody/flag issue. Not blocking for pilot.

A11 — Tenant-keys prefixing (isolation posture, plan 1.5)

gh workflow run host-diagnose.yml -f container=ai-gateway-api -f tail=40 \
  --repo MR-Eder/document-processing-pipelines

• GOOD (today, single tenant): TENANT_KEYS_ENABLED=false is acceptable while knoll is the only real tenant — voyager collections are un-prefixed and isolation is by collection-naming discipline.
• BAD (before a 2nd tenant): still false when a second Kanzlei is onboarded → cross- tenant collection addressing is possible. Plan 1.5 gates the cutover (TENANT_KEYS_ENABLED=true
  • Redis/voyager prefixes + scripts/migrate_keys_to_tenant.py) on the second real tenant. Record the live value; it drives the 1.5 go/no-go.

TODO-VERIFY: the live value of TENANT_KEYS_ENABLED on app.grag.ai (example ships false; an operator may have flipped it). This single value changes the Mandanten- isolation story — confirm before onboarding any second tenant.

A12/A13 — GDPR purge + cleanup workers (one call each)

gh workflow run host-diagnose.yml -f container=workspaces-worker -f tail=40 \
  --repo MR-Eder/document-processing-pipelines
gh workflow run host-diagnose.yml -f container=lineage-worker -f tail=40 \
  --repo MR-Eder/document-processing-pipelines

• GOOD: WORKSPACES_DOCUMENT_PURGE_ENABLED=true (workspaces-worker) and LINEAGE_CLEANUP_ENABLED=true (lineage-worker).
• BAD: either false/absent → deletes are soft-mark only; superseded/deleted chunks stay in voyager and lineage subtrees are only marked, not purged. This is a hard GDPR blocker (plan 0.11 / 1.4 / 12-gdpr-compliance.md). Because these are unset in the example, false is the likely live default — treat this row as high-risk.

TODO-VERIFY: whether the operator has set WORKSPACES_DOCUMENT_PURGE_ENABLED and LINEAGE_CLEANUP_ENABLED to true live. Both are unset in .env.production.example, so the compose default false almost certainly applies unless explicitly overridden.

A14 — Evidence-gate threshold value

gh workflow run host-diagnose.yml -f container=ai-gateway-api -f tail=40 \
  --repo MR-Eder/document-processing-pipelines

• GOOD: GATEWAY_RETRIEVAL_SCORE_THRESHOLD=0.15 (matches the example, :50) — the provisional, uncalibrated production value. GATEWAY_RETRIEVAL_MIN_RESULTS=2 alongside.
• BAD / note: 0.0 → gate drops nothing (compose default; weaker retrieval). Any value is "not wrong" but it is uncalibrated for German until plan 7.1 runs scripts/calibrate_evidence_gate.py. Record the live number so 7.1 has a baseline. Do not turn on user-facing groundedness enforcement (groundedness.enforcement=banner) before calibration.

Section B — The BFF authentication probe (⚠ plan 0.10a / 1.8 — the blocker)

This is the single most important check in the runbook. The plan's #1 blocker (finding 0 and 15 in the review) is that /next/api/* is publicly reachable with no auth: there is no middleware.ts in frontend-next, the Traefik /next router attaches no auth middleware, and every BFF route injects PIPELINE_API_KEY server-side while trusting the inbound X-Tenant-ID. So today an unauthenticated POST is expected to succeed — and that success is the proof the blocker is real.

B1 — Unauthenticated chat stream must SUCCEED today (proves the gap)

Send no Authorization header. Use the knoll-dev tenant and a throwaway/nonexistent kb_id so you never touch real data.

curl -sS -N -X POST \
  -H "X-Tenant-ID: knoll-dev" -H "Content-Type: application/json" \
  -d '{"conversation_id":"probe-unauth","kb_id":"kb-does-not-exist","user_message":"ping"}' \
  "$GRAG_URL/next/api/chat/stream" | head -c 400 ; echo

• GOOD (today, pre-1.8): you get an SSE stream back — lines like event: subscribed / event: trace — with no 401. This is the expected, alarming result: it demonstrates the plan-1.8 blocker exactly. Record "BFF UNAUTHENTICATED — confirmed" and keep it on the 0.11 go/no-go list as a hard blocker.
• AFTER 1.8 is shipped: the same request must return 401 (or 403). Re-run this exact probe as the acceptance test for 1.8 and as the 7.2 canary auth probe. Anything other than 401/403 after 1.8 means the fix did not land on this route.

B2 — Unauthenticated ingest submit must SUCCEED today (same gap, write path)

curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
  -H "X-Tenant-ID: knoll-dev" -H "Content-Type: application/json" \
  -d '{"kb_id":"kb-does-not-exist","document_id":"x","docfold_job_id":"y"}' \
  "$GRAG_URL/next/api/ingest/start"

• GOOD (today): a non-401 status (e.g. 400/422/200 depending on how far the handler gets) — again proving no auth gate exists on the write lane.
• AFTER 1.8: must be 401/403.

The BFF injects the platform key server-side, so this path never presents Knoll's key — D5 caller-attribution does not cover chat/ingest until 1.8 adds a bearer check inside the BFF routes (preferred, preserves attribution) or a Traefik /next/api/* auth middleware.

Section C — SSE buffering through Plesk nginx (plan 0.10b)

The documented Plesk nginx snippet sets proxy_http_version 1.1 + Upgrade headers but does not set proxy_buffering off (report-deployment.md open-Q 4). If nginx buffers, the UI never sees incremental frames — the whole stream arrives at once on completion, which silently breaks the chat typing UX and the live pipeline progress bar. Detect it by timing the gaps between frames.

C1 — Chat stream arrives incrementally

curl -sS -N -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{"conversation_id":"probe-sse","kb_id":"<a-real-scratch-kb>","user_message":"Nenne drei Punkte."}' \
  "$GRAG_URL/next/api/chat/stream" \
  | while IFS= read -r line; do printf '%s  %s\n' "$(date +%H:%M:%S.%3N)" "$line"; done

• GOOD (unbuffered): timestamps advance across successive event: trace and event: token lines — you see subscribed early, then retrieve/generate stages, then a burst of token frames spread over time, then done. Incremental = not buffered.
• BAD (buffered): nothing prints for many seconds, then all lines print with nearly identical timestamps at the very end. That is nginx buffering the whole response. Fix: add proxy_buffering off; (and X-Accel-Buffering: no) to the Plesk vhost for /next/api/. Until fixed, chat and pipeline UIs cannot stream (blocks the UX in 6.5/6.3).

C2 — Orchestrator event stream is incremental

The pipeline progress bar (plan 4.2 / 6.3) consumes GET /next/api/orchestrator/events/{jobId} (SSE bridge over Redis pub/sub; opens with event: subscribed, emits : keepalive comments every 20 s, 30-min ceiling — verified orchestrator/events/[jobId]/route.ts:29-30,71-80). Run a real ingest first (see 04-provisioning-runbook.md / 10-document-pipeline.md) to get a jobId, then:

curl -sS -N \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  "$GRAG_URL/next/api/orchestrator/events/<jobId>" \
  | while IFS= read -r line; do printf '%s  %s\n' "$(date +%H:%M:%S.%3N)" "$line"; done

• GOOD: an early event: subscribed, then NodeStart/NodeComplete frames as the DAG runs, plus periodic : keepalive lines while idle — timestamps spread out.
• BAD: no : keepalive lines ever appear and node events all land at once → buffered (same nginx fix as C1). No frames at all + immediate close → wrong jobId/tenant, or the job already expired (30-day archive TTL).

Section D — Retrieval profile & model cold-start

D1 — Retrieval profile is up (search works on a scratch collection)

Semantic search only works under --profile retrieval (voyager). Prove it end-to-end on a throwaway collection so a 503 cannot be confused with an empty-but-healthy KB. Create, upsert one text point (auto-embedded), search, then delete. (Shapes verified in ai-gateway/.../schemas/retrieval.py: CreateCollectionRequest.engine, UpsertPointsRequest.points[] with text, SearchRequest.query_text+top_k.)

SCRATCH="kb-verify-scratch-$(date +%s)"
# 1. create a dense collection
curl -sS -o /dev/null -w 'create %{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" -d '{"engine":"dense","dim":1536}' \
  "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/$SCRATCH"
# 2. upsert one text point (server auto-embeds)
curl -sS -o /dev/null -w 'upsert %{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"points":[{"id":"p1","text":"Die Marktposition des Mandanten ist stark.","payload":{"doc_id":"probe","ordinal":0}}]}' \
  "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/$SCRATCH/points"
# 3. free-text search
curl -sS -w '\nsearch %{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" -d '{"query_text":"Marktposition","top_k":3}' \
  "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/$SCRATCH/search"
# 4. cleanup
curl -sS -o /dev/null -w 'delete %{http_code}\n' -X DELETE \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/$SCRATCH"

• GOOD: create 200, upsert 200 (or 503 + Retry-After under WAL backpressure — retry), search 200 returning the point p1, delete 200.
• BAD: any step 404/503 on the collection routes → the retrieval profile (voyager) is not running. Blocks all of 4.x/5.x retrieval. Operator must ensure COMPOSE_PROFILES includes retrieval (the deploy workflow sets retrieval per report-deployment.md, but confirm the container is actually up via host-diagnose -f container=voyager). A 413 on upsert means the batch exceeded GATEWAY_VOYAGER_MAX_BATCH_SIZE (256) — not relevant to this 1-point probe.

D2 — Compound rerank route is wired

curl -sS -w '\n%{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"query_text":"Marktposition","top_k":5}' \
  "$GRAG_URL/ai-gateway/api/v1/retrieval/collections/$SCRATCH/search/rerank"

(Re-create the scratch collection from D1 first if you deleted it.)

• GOOD: 200 with a data.rerank block carrying retrieval_confidence + dropped_below_threshold. This is the one call plan 5.2/5.3.2 relies on.
• BAD: data.rerank.degraded=true with reason → GATEWAY_RERANK_FUSION_URL unset or rerank-fusion down; you still get plain hits (retrieval_confidence:"unknown"), so answers degrade rather than fail. Record and have the operator confirm rerank-fusion is up.

D3 — Model cold-start latency probe (plan 7.4)

rerank-fusion (~1.2 GB, two models) and groundedness (~2.5–3.1 GB, three models) lazy-load on first call — the first request can take 60–120 s unless RERANK_FUSION_PRELOAD / GROUNDEDNESS_PRELOAD are set. Time a cold groundedness call:

curl -sS -o /dev/null -w 'groundedness cold: %{time_total}s  http %{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"response_text":"Die Marktposition ist stark.","chunks":[{"text":"Der Mandant hat eine starke Marktposition.","chunk_id":"c1"}],"include_nli":true}' \
  "$GRAG_URL/groundedness/api/v1/score"
# run a SECOND time immediately to get the warm number:
curl -sS -o /dev/null -w 'groundedness warm: %{time_total}s  http %{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"response_text":"Die Marktposition ist stark.","chunks":[{"text":"Der Mandant hat eine starke Marktposition.","chunk_id":"c1"}],"include_nli":true}' \
  "$GRAG_URL/groundedness/api/v1/score"

• GOOD: either the cold call is already fast (<~5 s → preloads are on), or cold is slow (60–120 s) but warm is fast (<~3 s). Both mean the service works; the second confirms it was a load, not a hang.
• BAD: cold call times out at the Traefik/client ceiling, or warm is still 60 s+ → model isn't staying resident or the container is thrashing. Set GROUNDEDNESS_PRELOAD=true / RERANK_FUSION_PRELOAD=true (plan 1.4) so first user request after a deploy isn't a 2-minute stall. Note: every GRAG merge to main force-recreates containers (~30 s churn + re-cold-start) — Knoll must tolerate this (11-resilience-and-errors.md).

TODO-VERIFY: whether RERANK_FUSION_PRELOAD / GROUNDEDNESS_PRELOAD are set live. The D3 cold-vs-warm timing answers it behaviourally; confirm the env with host-diagnose -f container=groundedness-api if the cold number is large.

Section E — Canary tenant sanity (plan 7.2)

The platform runs a synthetic-traffic canary on a dedicated canary tenant (filter tenant!="canary" in dashboards). A green canary means the core RAG flows work end-to-end right now; it is the fastest single signal before Knoll builds on the platform.

E1 — Canary flow status is green

curl -sS \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: canary" \
  "$GRAG_URL/canary/canary/api/v1/status" | head -c 800 ; echo

(Note the doubled /canary/canary — the canary app carries its own /canary prefix and Traefik adds the routing prefix. Status is a public-ish read; run/* is admin-gated via CANARY_ADMIN_API_KEYS — verified canary/src/canary/api.py:256,337.)

• GOOD: JSON listing recent flow runs with passing status (the deploy gate itself runs privacy-path + streaming-pair). Recent + passing = platform healthy.
• BAD: stale timestamps (canary loop not running) or failing flows → the platform has a live regression; do not start Knoll ingest/chat verification until it clears. Escalate to the operator.

E2 — (Optional) trigger a canary run

curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" -H "X-Tenant-ID: canary" \
  "$GRAG_URL/canary/canary/api/v1/run/streaming-pair"

• GOOD: 200/202 and E1 then shows a fresh passing run.
• BAD: 401/403 → your admin key is not in CANARY_ADMIN_API_KEYS (custody issue, plan 1.2). A failing run → same escalation as E1.

Section F — Docs / OpenAPI reachability (contract stability, plan 1.9)

packages/grag-client (plan 2.5) is generated from each service's public /{svc}/openapi.json. These are public (no bearer). Confirm every service Knoll consumes serves a spec, so the client generator has a source and drift is detectable.

F1 — All consumed services expose OpenAPI

for svc in workspaces ai-gateway kg-service groundedness lineage ledger; do
  code=$(curl -sS -o /dev/null -w '%{http_code}' "$GRAG_URL/$svc/openapi.json")
  echo "$svc: $code"
done

• GOOD: every line 200. Feed these into the client generator (2.5) and pin them as the 1.9 contract snapshot.
• BAD: any 404/503 → that service is down or unrouted; the client for it cannot be generated and any Knoll call to it will fail. Cross-check with host-diagnose for that container.

F2 — BFF lanes have no OpenAPI (expected — they are hand-wrapped)

The chat/upload/ingest/orchestrator-events BFF routes live in the Next.js app under /next and are not in any service OpenAPI. That is expected — plan 2.5 hand-writes wrappers for them (SSE reader, upload/ingest, orchestrator-events). Do not look for /next/openapi.json; instead the 1.9 stability signal is the chat route.spec.ts spec test in the repo. Record "BFF = hand-wrapped, spec-test-pinned" as the acceptance note.

Results-recording template

Fill one row per check. Result = GOOD / BAD / N-A; Follow-up = the plan task or operator action a BAD result triggers. Keep this table with the verification run so the 0.11 / 8.5 go/no-go has an auditable record.

Go / no-go gate (plan 0.11 / 8.5)

Real Mandanten data may flow only when all of these are GOOD: B1/B2 re-tested to 401 (1.8 shipped), A12 + A13 purge/cleanup on, backups running (see 04-provisioning-runbook.md / 12-gdpr-compliance.md — the deploy workflow enables neither backup nor pitr), C1/C2 unbuffered, D1 retrieval up, and E1 canary green. A6 must be neutralised (schema-v1-without- relationships and/or flag off). Everything else is degrade-and-note, not blocking.