04 — Provisioning Runbook: Knoll Analyzer on GRAG (Plan Phase 1)

Purpose. Step-by-step operator runbook for preparing the live GRAG platform (https://app.grag.ai) for the Knoll Analyzer integration: credentials, admin custody split, tenants knoll / knoll-dev, platform flags, BYOK budget governance, tenant-settings baseline, preloads, backups, and rate limits. Each step lists preconditions, exact commands, expected output, a verification probe, and a rollback note. It executes plan tasks 1.1–1.7, 1.13, 1.15 (see tasks/todo.md); the Phase-1 engineering gaps (1.8 BFF auth, 1.9 contract tests, 1.10 German PII, 1.11 KG write route, 1.14 structured output) are code changes, not provisioning actions — they are tracked in 13-platform-gaps-issues.md. Nothing in this runbook unblocks real Mandanten data by itself: plan 1.8 (BFF auth) and step 9 (backups) are hard blockers per plan 0.11.

Status / verified against: 2026-07-06, repo document-processing-pipelines @ db63a95, deploy target https://app.grag.ai (single-box Plesk host, deployed by .github/workflows/deploy.yml on every push to main; manual runs via workflow_dispatch are supported — deploy.yml:33).

0. Conventions, roles, shell setup

Two credential roles exist after step 2. Keep them in separate custody:

Shell setup used by every example below:

export GRAG_URL="https://app.grag.ai"
export GRAG_API_KEY="<knoll runtime key, minted in step 1>"
export GRAG_ADMIN_KEY="<operator admin key, minted in step 2>"   # operator shell only
export GRAG_TENANT="knoll"                                        # "knoll-dev" for dev examples
export GH_REPO="MR-Eder/document-processing-pipelines"

Every GRAG request carries Authorization: Bearer … and X-Tenant-ID (never omit it — a missing header silently lands in tenant default, allow_default_tenant=true platform-wide). External URL pattern: https://app.grag.ai/<traefik-prefix>/<internal-path> (Traefik strips the first prefix). None of the provisioning endpoints below require X-Workspace-ID (kg-service does; that is Phase 3, see 07-kg-schema-knoll-advisory.md).

Where configuration lives (matters for steps 1, 2, 4, 8, 9):

Two hard consequences:

1. Manual .env edits on the box do not survive — every deploy recreates .env from the committed example + secrets. Durable flag changes go through a PR against .env.production.example.
2. gh secret set replaces the whole value and secrets cannot be read back. The current live values are only recoverable from the assembled .env in the deploy checkout on the box (a persisted copy dpp-live.env lives one level above the checkout, deploy.yml:331–334). Always reconstruct comma-lists from there before overwriting.

Redeploy = pick up new secrets/flags. Trigger with gh workflow run deploy.yml --repo "$GH_REPO" (or merge any PR to main). The run rebuilds .env, then docker compose up -d --wait --force-recreate — expect ~30 s connection churn on every service. Gate: repo variable PRODUCTION_DEPLOY_ENABLED=true must be set.

Before you start (plan 0.9): the committed .env.production.example may not equal the live box if an operator hand-edited between deploys or brought profiles up manually. Dump the live flag state first via the read-only host-diagnose.yml workflow (gh workflow run host-diagnose.yml --repo "$GH_REPO") or docker compose ps + docker compose exec <svc> env | grep -E 'TENANT_|KG_|LINEAGE_|WORKSPACES_|GATEWAY_' on the box. Record the result in 05-verification-runbook.md's checklist.

Step 1 — Mint the Knoll runtime key (plan 1.1, D5)

Preconditions: operator has gh auth with secrets:write on $GH_REPO; access to the box (or dpp-live.env) to read the current PIPELINE_API_KEY list and GATEWAY_CALLERS value.

Commands:

# 1. Mint a dedicated key (any opaque string; prefix aids log triage)
KNOLL_KEY="knoll-$(openssl rand -hex 32)"

# 2. Read the CURRENT values from the box (secrets are write-only in GitHub)
#    ssh to the host, then in the deploy checkout:
#      grep -E '^PIPELINE_API_KEY=' .env
#      grep -E '^GATEWAY_CALLERS='  .env
CURRENT_LIST="<paste current PIPELINE_API_KEY value>"
CURRENT_CALLERS="<paste current GATEWAY_CALLERS value, may be empty>"

# 3. Append — PIPELINE_API_KEY is a comma-separated list
#    (pipeline-common middleware.py:120–123; production_secrets.toml:49)
gh secret set PIPELINE_API_KEY --repo "$GH_REPO" --body "${CURRENT_LIST},${KNOLL_KEY}"

# 4. Named caller entry for ai-gateway attribution/rate buckets ("name:token" format)
if [ -n "$CURRENT_CALLERS" ]; then NEW_CALLERS="${CURRENT_CALLERS},knoll:${KNOLL_KEY}"; else NEW_CALLERS="knoll:${KNOLL_KEY}"; fi
gh secret set GATEWAY_CALLERS --repo "$GH_REPO" --body "$NEW_CALLERS"

# 5. Redeploy so the new .env reaches the containers
gh workflow run deploy.yml --repo "$GH_REPO"

Store $KNOLL_KEY in the Knoll secret store as GRAG_API_KEY (plan 2.8). Never hand Knoll the PIPELINE_SERVICE_KEY (it unlocks the X-DPP-* bypass headers).

Expected output: gh secret set prints ✓ Set Actions secret …; the deploy run goes green (smoke + canary steps pass).

Verification probe:

# Positive: new key accepted on a metadata route (any service, all accept the list)
curl -sS -o /dev/null -w '%{http_code}\n' "$GRAG_URL/ai-gateway/api/v1/models" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev"    # expect 200

# Negative: unknown key rejected
curl -sS -o /dev/null -w '%{http_code}\n' "$GRAG_URL/ai-gateway/api/v1/models" \
  -H "Authorization: Bearer bogus" -H "X-Tenant-ID: knoll-dev"            # expect 403

TODO-VERIFY: after the deploy, confirm ai-gateway attributes the key as caller knoll (not default) — check ai-gateway logs / X-RateLimit-* bucket behaviour on a retrieval call; the same token sits in both the PIPELINE_API_KEY list and the GATEWAY_CALLERS map and the middleware precedence between the two pools has not been live-tested.

⚠ Scope note (plan D5): BFF-lane traffic (/next/api/*) never presents this key — the BFF injects the platform key server-side. Caller attribution covers direct service calls only until plan 1.8 adds BFF auth.

Rollback: set PIPELINE_API_KEY back to $CURRENT_LIST and GATEWAY_CALLERS back to $CURRENT_CALLERS, redeploy. This revokes the Knoll key everywhere (it is also the key-rotation procedure, docs/runbooks/deploy.md Scenarios 5–6).

Step 2 — Admin custody split (plan 1.2)

⚠ Use the SINGULAR env names. The operator-facing variables are WORKSPACES_ADMIN_API_KEY and MCP_GATEWAY_ADMIN_API_KEY. Compose maps them onto the plural in-container settings — WORKSPACES_ADMIN_API_KEYS: "${WORKSPACES_ADMIN_API_KEY:-${PIPELINE_API_KEY:-}}" (docker-compose.yml:5169) and MCP_GATEWAY_ADMIN_API_KEYS: "${MCP_GATEWAY_ADMIN_API_KEY:-${PIPELINE_API_KEY:-}}" (:5339). Setting the plural name in .env/secrets does nothing, and admin access silently stays on every PIPELINE_API_KEY entry — including Knoll's runtime key from step 1. That is the pre-split state you are fixing.

Preconditions: step 1 done. A small repo PR is required first: neither variable is currently wired into the deploy (both are absent from production_secrets.toml and deploy.yml; they exist only as commented documentation in .env.example:1165,1200). The PR must add both names to:

1. pipeline-common/contracts/production_secrets.toml ([[external]] table, one rationale line each),
2. the env: block of .github/workflows/deploy.yml (WORKSPACES_ADMIN_API_KEY: ${{ secrets.WORKSPACES_ADMIN_API_KEY }}, same for MCP),
3. the .env assembly heredoc in the same workflow (echo "WORKSPACES_ADMIN_API_KEY=${WORKSPACES_ADMIN_API_KEY}", same for MCP).

TODO-VERIFY: run pytest pipeline-common/tests/test_prod_env_secrets.py -v on the PR branch — the drift guard pins the .env.example ↔ contract ↔ .env.production.example split and its exact expectations for contract-only additions should be confirmed green before merge.

Commands:

GRAG_ADMIN_KEY="admin-$(openssl rand -hex 32)"
gh secret set WORKSPACES_ADMIN_API_KEY  --repo "$GH_REPO" --body "$GRAG_ADMIN_KEY"
gh secret set MCP_GATEWAY_ADMIN_API_KEY --repo "$GH_REPO" --body "$GRAG_ADMIN_KEY"
# merge the wiring PR (that push deploys), or redeploy explicitly:
gh workflow run deploy.yml --repo "$GH_REPO"

(The vars accept comma-lists if you later want distinct workspaces/MCP admin keys.)

Expected output: deploy green; workspaces + mcp-gateway containers restart with the new envs.

Verification probe (the 403 probe — this is the acceptance test of the split):

# Knoll runtime key must now be REJECTED on admin surfaces:
curl -sS -o /dev/null -w '%{http_code}\n' -X POST "$GRAG_URL/workspaces/api/v1/tenants" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll" \
  -H "Content-Type: application/json" -d '{"id":"custody-probe","name":"probe"}'   # expect 403

curl -sS -o /dev/null -w '%{http_code}\n' -X POST "$GRAG_URL/mcp-gateway/api/v1/tokens" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll" \
  -H "Content-Type: application/json" \
  -d '{"scope_level":"tenant","name":"custody-probe"}'                              # expect 403

# Admin key must still work (list tenants is admin-gated, read-only, safe):
curl -sS -o /dev/null -w '%{http_code}\n' "$GRAG_URL/workspaces/api/v1/tenants" \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" -H "X-Tenant-ID: knoll"                # expect 200

If the first probe returns 201, the split is NOT in effect (variable name typo, plural instead of singular, or the deploy did not pick the secret up). Add this probe to the canary set (plan 7.2).

Rollback: gh secret delete WORKSPACES_ADMIN_API_KEY --repo "$GH_REPO" (and the MCP one) + redeploy — compose falls back to PIPELINE_API_KEY, restoring the pre-split behaviour. Note the security consequence: every runtime key is admin again.

Step 3 — Create tenants knoll and knoll-dev (plan 1.3)

Preconditions: step 2 done ($GRAG_ADMIN_KEY live). Tenant ids are DNS-labels (^[a-z0-9][a-z0-9-]{0,62}$); default, system, canary are reserved — never use them.

Commands (operator; the created tenant comes from the JSON body id, the X-Tenant-ID header is still validated by middleware — send the same value):

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/tenants" \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" -H "X-Tenant-ID: knoll" \
  -H "Content-Type: application/json" \
  -d '{"id": "knoll", "name": "Kanzlei Knoll", "metadata": {"env": "prod", "owner": "knoll-analyzer"}}'

curl -sS -X POST "$GRAG_URL/workspaces/api/v1/tenants" \
  -H "Authorization: Bearer $GRAG_ADMIN_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"id": "knoll-dev", "name": "Kanzlei Knoll (Entwicklung)", "metadata": {"env": "dev", "owner": "knoll-analyzer"}}'

Expected output: 201 with the tenant object (schema TenantCreate {id, name, metadata}, extra="forbid"); each create auto-provisions the undeletable general workspace and sets default_workspace_id. A 409 means the tenant already exists — treat as already-provisioned (idempotent; there is no Idempotency-Key header anywhere in the workspaces service, plan 3.1 ⚠).

Verification probe (runtime key — proves Knoll can see its own tenant but nothing admin):

curl -sS "$GRAG_URL/workspaces/api/v1/tenants/knoll" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll"        # 200, own row only

curl -sS "$GRAG_URL/workspaces/api/v1/workspaces" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll"        # {"items":[…]} containing "general"

The kb-methodology KB in general and the client workspaces are not created here — they are Phase-3 runtime provisioning (plan 3.1/3.3; see the final section and 03-id-conventions.md).

Rollback: DELETE /workspaces/api/v1/tenants/{id} (admin key) → 204. This is soft-delete only — no composite purge of voyager collections / Redis keys / vault rows exists (platform limitation; DSGVO offboarding is plan 3.2 / 12-gdpr-compliance.md).

Step 4 — Platform flag set (plan 1.4, informed by plan 0.9)

Preconditions: live flag state dumped (see section 0). Flags are deployment-global (they affect all tenants on the box) — coordinate any true→false flip with the operator of other tenants.

Target state vs the committed .env.production.example:

Rationale for KG_RELATION_EXTRACTION_ENABLED=false (plan D7/1.4; review findings #3/#17): the prod baseline enables relation auto-extraction (quality bar passed on the legal fixture, not German advisory text). It fires when: flag on + entity extraction on + the workspace's active schema declares relationships + no producer-supplied entities. Today it is dormant only because no workspace activates a relationship-declaring schema. The moment Knoll activates knoll-advisory per workspace (plan 3.4) and uploads documents, un-harnessed German relation edges would land in the same AGE graph as the deterministic writer's facts (plan 3.6) — and KG edges are AGE-only, best-effort, non-replayable. Defence in depth: flip the flag off and ship knoll-advisory v1 without a relationships block (07-kg-schema-knoll-advisory.md). Flipping it off is safe for the rest of the box today for the same reason it is dormant. Re-enable only after the German quality harness passes (plan 7.1, schema v2).

Commands: one PR against .env.production.example (flip line 446 to false; append WORKSPACES_DOCUMENT_PURGE_ENABLED=true and LINEAGE_CLEANUP_ENABLED=true near the other workspaces/lineage knobs). Run the repo pre-flight checks locally (python scripts/check_docs_currency.py + the structural pytest set in the repo CLAUDE.md — all these flags already exist, so no test constants change). Merge → auto-deploy.

Expected output: deploy green; .env on the box contains the three changed lines.

Verification probe:

# Tenant settings live (not 503):
curl -sS -o /dev/null -w '%{http_code}\n' "$GRAG_URL/workspaces/api/v1/settings/registry" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev"     # expect 200

# Flag values inside the containers (on the box):
docker compose exec workspaces-api env | grep -E 'WORKSPACES_DOCUMENT_PURGE_ENABLED|TENANT_SETTINGS'
docker compose exec lineage-worker  env | grep LINEAGE_CLEANUP_ENABLED
docker compose exec kg-service-worker env | grep -E 'KG_SERVICE_(INGEST|ENTITY_EXTRACTION|RELATION_EXTRACTION)_ENABLED'
# ⚠ grep the KG_SERVICE_-prefixed names — compose maps the operator vars onto
#   KG_SERVICE_* in-container settings (docker-compose.yml:5091/5106/5114), so
#   grepping the bare KG_* names prints nothing. Matches 05-verification A4–A6.

TODO-VERIFY: exact compose service names for the lineage/kg worker rows (docker compose ps lists them; naming convention is <svc>-worker per ADR 0010). Purge behaviour end-to-end (delete → voyager points gone) is smoke-tested in 05-verification-runbook.md.

Rollback: revert the PR, redeploy. Purge flags back to false returns to soft-delete-only (data already purged stays purged).

Step 5 — TENANT_KEYS_ENABLED cutover: decision note (plan 1.5)

Not an action today — a recorded decision. TENANT_KEYS_ENABLED=false (.env.production.example:166) means Redis keys are un-prefixed and voyager collections are named by bare kb_id (no {tenant}- prefix). Consequences for Knoll:

• KB ids must be globally unique across ALL tenants on the box, including between knoll and knoll-dev. Two tenants creating the same kb_id share one voyager collection — silent cross-tenant vector mixing. Enforce via the id conventions in 03-id-conventions.md: shortid-bearing analysis-KBs are safe by construction; fixed names like kb-methodology must be tenant-distinct until cutover (e.g. kb-methodology in knoll, kb-methodology-dev in knoll-dev).
• Cutover is required before any second real Kanzlei tenant (ADR 0001 Phase C): run scripts/migrate_keys_to_tenant.py, flip the flag, redeploy. TODO-VERIFY: exact invocation, downtime window, and whether existing voyager collections are renamed or re-ingested by the script — read the script + ADR 0001 before scheduling; do a dry run against knoll-dev data first.
• Decision for the pilot: stay false. Single Kanzlei, id-convention discipline covers the risk, and the migration is the riskiest platform operation in this list. Revisit at plan 8.5 (go-live checklist) and mandatorily before Kanzlei #2.

Verification probe (discipline check, run periodically): list all KBs per tenant and diff for id collisions:

for t in knoll knoll-dev; do
  curl -sS "$GRAG_URL/workspaces/api/v1/kbs" \
    -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $t" ; echo
done   # no kb id may appear under both tenants

Step 6 — BYOK keyvault: provider key with budget + RPM (plan 1.6)

Preconditions: GATEWAY_KEYVAULT_ENABLED=true (already committed, line 110) and the GATEWAY_KEYVAULT_MASTER_KEY secret set (it is in production_secrets.toml; the keyvault migrate one-shot runs inline in the deploy when enabled). The Kanzlei's provider key (e.g. OpenAI) at hand. Keyvault CRUD is tenant-scoped under the runtime key — no admin needed; policy: operator performs the initial provisioning, Kanzlei self-service arrives with plan 6.6.

Commands (repeat per tenant; use a separate provider key or at least a separate budget for knoll-dev):

# Optional: list valid providers/services first
curl -sS "$GRAG_URL/ai-gateway/api/v1/keys/catalog" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/keys" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "openai",
    "secret": "sk-<kanzlei-provider-key>",
    "label": "Kanzlei Knoll — OpenAI",
    "scope_level": "tenant",
    "monthly_budget_usd": 200,
    "rate_limit_rpm": 60
  }'

Schema: KeyCreate {provider, secret, label, allowed_services[], scope_level ∈ tenant|workspace|project, workspace_id?, project_id?, enabled, monthly_budget_usd ≥ 0, rate_limit_rpm ≥ 1, expires_at?} (ai-gateway/src/ai_gateway/api/schemas/keys.py:25–36). Budget/RPM values are initial pilot numbers — recalibrate with 14-cost-model.md.

Expected output: 201 with a key summary — id, last4, governance fields, usage block. The secret is write-only and never returned.

Verification probe:

# 1. Key visible with governance:
curl -sS "$GRAG_URL/ai-gateway/api/v1/keys" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"   # {"keys":[{…,"last4":"…","usage":{…}}]}

# 2. Resolution: a completion WITHOUT a per-request api_key must use the stored key
curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/chat/completions" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"Sag nur: ok"}],"max_tokens":5}'
# then re-list keys: last_used_at must have advanced.

# 3. Governance bites (dev tenant only, destructive to the key): PATCH the knoll-dev key
#    to {"enabled": false} and repeat the completion → expect 403; re-enable after.
#    Budget/RPM breaches reject with 402 / 429 respectively — no silent platform-key fallback.

# 4. Spend lands in the ledger (⚠ note the double /ledger in the path — plan 1.6):
curl -sS "$GRAG_URL/ledger/api/v1/ledger/spend?group_by=provider" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
# per-run totals (pairs with X-Pipeline-Id stamping, plan 7.3):
#   GET $GRAG_URL/ledger/api/v1/ledger/totals?pipeline_id=<id>
# group_by accepts ONLY service|provider — per-Analyse spend comes from /totals, never /spend.

Rollback: DELETE /ai-gateway/api/v1/keys/{id} → 204. With no stored key the gateway falls back to the platform env provider key (spend then lands on the platform bill — remove Knoll traffic promptly or re-create the key).

Step 7 — Tenant-settings baseline (plan 1.7)

Preconditions: steps 3 + 4 done (TENANT_SETTINGS_ENABLED=true, tenants exist). All keys below are validated against the SettingSpec registry (pipeline-common/src/pipeline_common/tenant_settings.py); none of them is security-sensitive, so the runtime key suffices. Writes go PUT /workspaces/api/v1/settings/{key} with body {"value": …}; consumers read fail-open with a ~30 s cache, so effects show within a minute.

Baseline (apply to knoll AND knoll-dev; registry-verified key names, types, bounds):

Commands (pattern; loop over the table):

put_setting () {  # $1=key  $2=raw JSON value
  curl -sS -X PUT "$GRAG_URL/workspaces/api/v1/settings/$1" \
    -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
    -H "Content-Type: application/json" -d "{\"value\": $2}"
}
put_setting chat.default_model '"openai/gpt-4o-mini"'
put_setting chat.default_top_k '8'
put_setting chunking.default_chunk_size '1200'
put_setting chunking.default_chunk_overlap '150'
put_setting retrieval.score_threshold '0.15'
put_setting frontend.default_locale '"de"'
# chunking.default_strategy + ingest.dag_template: after resolving the TODO-VERIFYs above

Expected output: 200 per PUT. 400 = value failed registry validation (type/enum/bounds); 403 = you hit a security-sensitive key with the runtime bearer (none in this baseline — if you see it, check the key name); 503 = TENANT_SETTINGS_ENABLED is off (step 4 incomplete).

Verification probe:

curl -sS "$GRAG_URL/workspaces/api/v1/settings" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"    # all overrides present
curl -sS "$GRAG_URL/workspaces/api/v1/settings/audit?key=chat.default_model" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"    # append-only audit row

Rollback: DELETE /workspaces/api/v1/settings/{key} reverts that key to the platform default (absence of a row = env default; never seed defaults you don't mean to own).

Step 8 — Model preloads against cold starts (plan 1.4)

Preconditions: none beyond deploy access. rerank-fusion (cross-encoder) and groundedness (GLiNER2 + ColBERT) lazy-load weights on first call — first retrieval/Gutachten step after a deploy pays 60–120 s. Both preload knobs default false in compose (RERANK_FUSION_PRELOAD, compose:4730; GROUNDEDNESS_PRELOAD, compose:4580; note GROUNDEDNESS_WORKER_PRELOAD already defaults true — the API container is the cold one).

Commands: same PR mechanics as step 4 — append to .env.production.example:

RERANK_FUSION_PRELOAD=true
GROUNDEDNESS_PRELOAD=true

Merge → deploy. Watch the deploy --wait budget: preloading shifts weight-loading into container start; the workflow allows 600 s, but if the deploy starts flapping on --wait timeouts, this is the first suspect.

Expected output: deploy green; after deploy, first search/rerank and first groundedness score respond in normal latency (no minute-long first hit).

Verification probe: immediately after a deploy finishes:

time curl -sS -X POST "$GRAG_URL/groundedness/api/v1/score" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: knoll-dev" \
  -H "Content-Type: application/json" \
  -d '{"response_text":"Berlin ist die Hauptstadt Deutschlands.","chunks":[{"chunk_id":"c1","text":"Berlin ist die Hauptstadt Deutschlands."}]}'
# expect seconds, not minutes

Rollback: revert the two lines (containers start faster, first request slow again).

Step 9 — Backups: backup + PITR profiles and restore smoke (plan 1.13 — blocking for real data)

Preconditions / current state: the deploy workflow runs COMPOSE_PROFILES: retrieval only (deploy.yml:108) — neither the pg_dump backup profile nor pitr/pitr-vault is enabled by the workflow, despite the compose comment "Prod deployments MUST enable this profile" (compose:549–550). The WAL-G secrets are already plumbed (WALG_LIBSODIUM_KEY, WALG_VAULT_LIBSODIUM_KEY, deploy.yml:182–183) but may be empty. TODO-VERIFY first (plan 0.11): whether an operator already brought PITR up manually from the deployed directory — docker compose ps | grep -E 'wal-g|backup' on the box (host-maintenance.yml's schedule comment references live 02:00/04:00 base-backup windows, which suggests they may be running).

Commands:

# 1. Secrets (only if unset — generate SEPARATE keys, never reuse one for both clusters):
gh secret set WALG_LIBSODIUM_KEY       --repo "$GH_REPO" --body "$(openssl rand -base64 32)"
gh secret set WALG_VAULT_LIBSODIUM_KEY --repo "$GH_REPO" --body "$(openssl rand -base64 32)"

# 2. PR against the repo:
#    a) deploy.yml:108  →  COMPOSE_PROFILES: retrieval,backup,pitr,pitr-vault
#       (minimum viable: retrieval,backup — pg_dump sidecars for both clusters)
#    b) .env.production.example: add PG_PLATFORM_ARCHIVE_MODE=on and PG_VAULT_ARCHIVE_MODE=on
#       (required for WAL archiving; NOT present in the file today)
#    c) PITR object-store target: in prod the operator omits the local MinIO and points
#       WAL-G at an external bucket (ADR 0015/0016).

TODO-VERIFY: (a) the exact WAL-G bucket/endpoint env names for an external S3 target (operator-provided per ADR 0015/0016 — read infra/ wal-g service env in docker-compose.yml before the PR); (b) both Postgres clusters must be recreated once after first setting archive_mode=on (repo CLAUDE.md says so for dev; confirm the prod sequence in ADR 0015 before merging — a recreate on the live box is a maintenance window).

Expected output: after deploy, docker compose ps on the box shows dpp-postgres-platform-backup, dpp-postgres-vault-backup (and dpp-wal-g, dpp-walg-vault if PITR enabled) running; dumps appear in the named volumes on the first cycle.

Verification probe + restore smoke (do this ONCE now, not during an incident):

# dumps exist and rotate (volume name carries the compose project prefix — adjust):
docker run --rm -v document-processing-pipelines_dpp-postgres-platform-backups:/backups alpine \
  ls -lh /backups

# restore smoke: the newest workspaces dump is a valid pg_dump custom archive
docker run --rm -v document-processing-pipelines_dpp-postgres-platform-backups:/backups postgres:16-alpine \
  sh -c 'pg_restore --list "$(ls -t /backups/*workspaces* | head -1)" | head -20'
# expect a TOC listing incl. the tenants / knowledge_bases tables

For a full drill follow docs/runbooks/postgres-restore.md (pg_dump scenarios; PITR = Scenario 4a platform / 4b vault). Vault PITR cutover additionally requires the KEK-decrypt smoke test (smoke_decrypt.py) proving restored ciphertext decrypts under the current ANONYMIZATION_VAULT_MASTER_KEY. Ship both backup volumes to encrypted off-box object storage — a backup on the same single-box SPOF does not satisfy plan 0.11.

Rollback: remove the profiles from COMPOSE_PROFILES and redeploy (sidecars stop; existing dumps/WAL remain in the volumes). Losing WALG_*_LIBSODIUM_KEY makes existing PITR archives unrecoverable — escrow both keys with the operator.

Step 10 — Rate-limit review (plan 1.15)

Preconditions: steps 1–4 done. No change is recommended for the pilot — this step is a review + a recorded decision.

Current ceilings a Knoll caller inherits (all verified in repo):

Review outcome for the pilot: defaults suffice. The binding constraint is the retrieval bucket (120 rpm) during Gutachten pipeline step 2 (one search/rerank per checklist document, plan 5.3) and bulk KB backfills — both burst-shaped. grag-client must back off on 429 + Retry-After (plan 2.5/7.4) and alert on sustained 429s (plan 7.6).

If bulk ingest/backfill is scheduled (plan 3.3 Methodenhandbuch, plan 8.1 fixtures): raise the retrieval bucket via a .env.production.example PR — either the global GATEWAY_RATE_LIMIT_RETRIEVAL_RPM or a caller-specific entry in GATEWAY_RATE_LIMIT_OVERRIDES (line 34, empty today). TODO-VERIFY: the GATEWAY_RATE_LIMIT_OVERRIDES value format (per-caller/per-group syntax) in ai-gateway's rate-limit module before using it.

Verification probe: fire >120 retrieval calls in a minute against knoll-dev and confirm 429 + Retry-After + X-RateLimit-* headers appear (and that the Knoll caller lands in its own bucket, not default — ties back to the step 1 TODO-VERIFY).

Rollback: limits are config lines; revert the PR.

Runtime provisioning: Mandant / Analyse (automated by the Knoll backend)

Everything in steps 1–10 is operator, one-time. Day-to-day provisioning is done by the Knoll backend with the runtime key — no operator involvement:

Idempotency contract for all runtime creates: deterministic client-chosen ids + treat 409 as already-provisioned. There is no Idempotency-Key header anywhere in the workspaces service (plan 3.1 ⚠) — do not send one expecting semantics.

Stays operator-only forever (admin custody from step 2): tenant create/patch/delete (new Kanzlei = new tenant, plan D1), security-sensitive tenant settings (gateway.guardrail_*, gateway.firewall_enabled, chunking.anonymization_mode, pageindex.anonymization_mode, groundedness.enforcement, agent_graph.capture_content), MCP token minting (POST /mcp-gateway/api/v1/tokens — plan 5.7 is an operator/provisioning action after the custody split), platform flags + COMPOSE_PROFILES + redeploys, backups/restores, rate-limit changes, TENANT_KEYS_ENABLED cutover. BYOK keyvault CRUD is technically runtime-callable; keep it operator-run until the plan 6.6 Einstellungen tab productizes it.

Cross-references: request/response details for every endpoint above → 02-grag-api-cookbook.md; id rules → 03-id-conventions.md; post-provisioning acceptance suite → 05-verification-runbook.md; error taxonomy and retry/backoff → 11-resilience-and-errors.md; go/no-go gates before real Mandanten data → 12-gdpr-compliance.md and plan 0.11.