10 — Document Pipeline: Checkliste & Uploads (Plan Phase 4)

Purpose. This document specifies how Knoll Analyzer gets documents (Jahresabschlüsse, Verträge, Businesspläne, Fragebogen-Antworten, Gutachten-PDFs) from the UI into a searchable GRAG Knowledge Base per Analyse — and back out again. It covers the end-to-end upload sequence (plan 4.1), progress reporting (4.2), checklist automation (4.3), document lifecycle (4.4), Fragebogen-to-markdown rendering (4.5), and the Gutachten-PDF ingest-back (5.3 step 6). An engineer should be able to implement the Knoll-backend side of Phase 4 from this document plus 02-grag-api-cookbook.md.

Status / verified against: 2026-07-06, repo document-processing-pipelines @ db63a95. Endpoints and payloads verified in source (frontend-next/apps/web/app/api/{upload,ingest,orchestrator,pipeline}/…, docfold/src/docfold/engines/router.py, workspaces/src/workspaces/routers/documents.py, pipeline_common/events.py). Review corrections from the adversarial plan review are applied throughout (⚠ markers).

Sibling docs: 01-architecture.md (big picture), 02-grag-api-cookbook.md (all endpoints + error taxonomy), 03-id-conventions.md (id shapes), 06-knoll-db-schema.md (dateien, grag_refs, checklist_items), 08-gutachten-pipeline-spec.md (consumes indexed docs), 09-chat-integration.md (retrieval over the same KBs), 11-resilience-and-errors.md (retry policy), 12-gdpr-compliance.md (purge/Auskunft), 13-platform-gaps-issues.md (1.8 BFF auth blocker et al.).


1. Preconditions & conventions

  • All examples use $GRAG_URL (= https://app.grag.ai), $GRAG_API_KEY (dedicated Knoll key, plan 1.1), $GRAG_TENANT (= knoll in prod, knoll-dev in dev). External URL pattern: https://app.grag.ai/<traefik-prefix>/<internal-path>.
  • Every request carries Authorization: Bearer $GRAG_API_KEY and X-Tenant-ID: $GRAG_TENANT. ⚠ The /next/api/* BFF endpoints are unauthenticated today and trust any inbound X-Tenant-ID (plan 1.8, hard blocker before real Mandanten data). Send the Authorization header anyway so Knoll's calls keep working unchanged once 1.8 lands; X-Tenant-ID is functionally required on the BFF (missing header silently falls back to tenant default — the grag-client hard-fails on a missing tenant, plan 2.5).
  • The analysis's KB (kb-analysis-<shortid>, provisioned per plan 3.1 / 04-provisioning-runbook.md) and the client workspace (client-<slug>-<shortid>) must exist before the first upload.
  • The orchestrator has no HTTP submit endpoint — ingest submission goes through the BFF lanes below (plan D4/0.4). There is no other externally reachable path.
  • ⚠ No Idempotency-Key header exists anywhere on the platform (review finding [5]); idempotency on this pipeline = client-side state in files/grag_refs + the retry rules in §6.

2. End-to-end upload sequence (plan 4.1)

Swimlane-style flow. Lanes: UIKnoll backendKnoll file storeGRAG BFFdocfoldorchestratorvoyager / kgworkspaces inventory.

UI            Knoll backend        File store      BFF (/next/api)      docfold        orchestrator      voyager/kg     workspaces
│ (1) upload   │                     │                │                    │               │                 │              │
├──────────────▶ validate, create    │                │                    │               │                 │              │
│              │ files row ──────────▶ (2) persist    │                    │               │                 │              │
│              │                     │ binary+sha256  │                    │               │                 │              │
│              │ (3) POST /next/api/upload/start ─────▶ pre-creates row ───────────────────────────────────────────────────▶ documents:
│              │                     │                │ + forwards file ───▶ async convert  │                 │              │ status=queued
│              │ ◀── {documentId, docfoldJobId} ──────┤                    │               │                 │              │
│              │ (4) poll GET /docfold/api/v1/jobs/{id} until completed ───▶               │                 │              │
│              │ (5) POST /next/api/ingest/start ─────▶ fetch markdown ────▶ result        │                 │              │
│              │                     │                │ ensure collection ─────────────────────────────────▶ create if 404 │
│              │                     │                │ sign + LPUSH ──────────────────────▶ chunking node   │              │
│              │ ◀── {jobId} ────────────────────────┤                    │               │ │ voyager_index_ingest ─▶ points│
│ ◀─(6) SSE────┼── GET /next/api/orchestrator/events/{jobId} ─────────────────────────────┤ (+ kg segment   │              │
│  progress    │                     │                │                    │  emission when KG_INGEST on)    │              │
│              │ (7) on `completed`: PATCH documents row → indexed + chunks ──────────────────────────────────────────────▶ status=indexed
│              │ (8) classify → suggest checklist match (§7)               │               │                 │              │
│ ◀─ suggestion│ human confirms → checklist_item status "Received"         │               │                 │              │

Step by step, with the exact calls:

(1) UI → Knoll backend. Upload form on the analysis detail page (6.3), scoped to a checklist_item_id (or "Sonstiges"). Knoll backend authenticates the user (Knoll roles, GRAG authorizes keys not people — plan 7.7).

(2) ⚠ File store FIRST (plan 2.4). Persist the original binary to the Knoll file store before any GRAG call, and create the files row:

files: { id, firm_id, client_id?, analysis_id?, checklist_item_id?, expert_report_id?,
         filename, mime, size_bytes, sha256, source, storage_key,
         grag_document_id?, uploaded_by?, deleted_at? }

(Columns per 06-knoll-db-schema.md §3.14 — normative. Job ids and sync state do NOT live on files: the docfold/ingest job id goes into grag_refs.grag_job_id and the lifecycle into grag_refs.sync_status, see the id-capture points below and 06 §3.15.)

Why file-store-first (this is a hard rule, not an optimization): GRAG retains no binaries. docfold's output is markdown held in Redis job payloads ("fetch promptly"; job archive strips file_base64 > 100 KB after 30 d), and the workspaces documents table is a metadata-only inventory — there is no raw-file GET endpoint anywhere on the platform (review finding [8]). Without the file store, the prototype's per-document DownloadButton (6.3), the expert-report PDF export (6.4), re-upload after a failed ingest (§6), and the GDPR Auskunft/Löschung of originals (7.5, 12-gdpr-compliance.md) are all impossible. Dedupe hint: if sha256 already exists for this analysis, warn the user before re-ingesting.

(3) Convert — BFF upload lane.

curl -sS -X POST "$GRAG_URL/next/api/upload/start" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -F "kb_id=kb-analysis-a3f2" \
  -F "files=@/var/knoll/filestore/ab/cd/jahresabschluss-2025.pdf;filename=Jahresabschluss-2025.pdf"

Response (per file; replaces_document_id is only echoed back — see §8.1):

{
  "kb_id": "kb-analysis-a3f2",
  "jobs": [
    {
      "filename": "Jahresabschluss-2025.pdf",
      "size": 2381244,
      "docfoldJobId": "dfj_9x2k1",
      "documentId": "doc_71b2c9",
      "replacesDocumentId": null
    }
  ]
}

What the BFF did server-side: (a) pre-created the workspaces documents row (status=queued, source_kind=upload) so the doc is visible in the inventory immediately; (b) forwarded the file to POST /docfold/api/v1/convert (async) with kb_id + document_id riding along for lineage self-emission. If convert submission fails, the BFF rolls the pre-created row back and returns docfoldJobId: null + error for that file — treat as terminal, see §6.

IDs captured now: files.grag_document_id = documentId; grag_refs row {knoll_type: "file", knoll_id: files.id, grag_type: "document", grag_tenant_id, grag_workspace_id, grag_kb_id, grag_id: documentId, grag_job_id: docfoldJobId, sync_status: "pending"} (column names + enum per 06 §3.15; pending covers the whole convert+index run until step 7). The checklist_item_id link lives on the files row (one checklist item can accumulate several files — "PartiallyReceived").

(4) Poll docfold until conversion completes (direct service call, properly authenticated):

curl -sS "$GRAG_URL/docfold/api/v1/jobs/dfj_9x2k1" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
# → {"status": "pending" | "processing" | "completed" | "failed", "progress": ..., "error": ...}

Poll every 2–5 s with backoff. failed → §6 row F1. Do not call /jobs/{id}/result yourself for the ingest hand-off — the BFF does that — but you MAY fetch it once for checklist classification input (§7); results are Redis-held, fetch promptly.

(5) Index — BFF ingest lane. Only after docfold completed:

curl -sS -X POST "$GRAG_URL/next/api/ingest/start" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -d '{
    "kb_id": "kb-analysis-a3f2",
    "document_id": "doc_71b2c9",
    "docfold_job_id": "dfj_9x2k1",
    "filename": "Jahresabschluss-2025.pdf",
    "workspace_id": "client-kaesmann-7h2k"
  }'
# → {"jobId": "ingest_ab12cd_1751882400000", "tenantId": "knoll", "status": "queued"}

Always pass workspace_id (the client workspace): it rides the job envelope so the chunking stage's KG segment emission carries it and the kg-worker can resolve the workspace's ACTIVE knoll-advisory schema for entity extraction (plan 3.4, 07-kg-schema-knoll-advisory.md). Server-side the BFF: fetches the markdown from docfold (retries one 409), ensures the per-KB voyager collection ({engine:"dense", dim:1536}, 409 = exists = OK), resolves the tenant's ingest.dag_template / chunking.default_* settings (plan 1.7; defaults: strategy markdown, size 500, overlap 50), stamps parent_artifact_id (docfold CAS id) into the chunking node, signs the envelope, and LPUSHes it to pipeline_execution_queue. A 502 from this route is retriable (§6 row F2).

IDs captured now: grag_refs.grag_job_id = jobId (overwrites the docfold job id from step 3 — the column holds the last job id, diagnostic only, 06 §3.15); sync_status stays pending until step 7. The jobId doubles as the pipeline id for per-job cost lookup: GET $GRAG_URL/ledger/api/v1/ledger/totals?pipeline_id=ingest_ab12cd_1751882400000 ⚠ (correct ledger path per review finding [4]; the orchestrator's own budget poller uses the same route). TODO-VERIFY: that the chunking→voyager lane actually stamps the job id onto its paid gateway calls (embedding upsert) so /totals is non-empty for plain ingest jobs — confirmed for pageindex nodes, not explicitly traced for voyager_index_ingest.

(6) Progress — §5.

(7) Finalize inventory. On terminal completed, PATCH the documents row (belt-and-braces; the workspaces-worker does this automatically only when LINEAGE_ENABLED=true, plan 1.4 — keep the PATCH anyway so dev/dark environments behave identically):

curl -sS -X PATCH "$GRAG_URL/workspaces/api/v1/documents/doc_71b2c9" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -d '{"status": "indexed", "chunks": 42}'

Chunk count comes from the poll response results.<chunkNodeId>.chunks on COMPLETED. Set grag_refs.sync_status = "synced" + last_synced_at (canonical enum, 06 §3.15 — the fine-grained "indexed" state lives GRAG-side on the workspaces documents row, cache-only in Knoll).

(8) Checklist suggestion — §7. The per-analysis document list in the UI (6.3) comes from GET $GRAG_URL/workspaces/api/v1/kbs/kb-analysis-a3f2/documentscache-only in Knoll DB, never duplicated (plan 4.4); downloads come from the file store via files.storage_key.

2.1 ID-capture summary

Where producedFieldPersisted inUsed for
Knoll backendfiles.id, storage_key, sha256filesdownloads, GDPR, dedupe, re-upload
/next/api/upload/startjobs[].documentIdfiles.grag_document_id, grag_refs.grag_idinventory, supersede/delete, chat citations (doc_id)
/next/api/upload/startjobs[].docfoldJobIdgrag_refs.grag_job_id (transient)docfold poll, ingest/start input, classification fetch
/next/api/ingest/startjobIdgrag_refs.grag_job_id (overwrites docfold id)SSE/poll progress, ledger /totals?pipeline_id=, audit
Knoll UI/userchecklist_item_idfiles.checklist_item_idchecklist status automation (§7)

3. Supported input formats

From the docfold engine router (docfold/src/docfold/engines/router.py, extension→engine priority chains; Docling is first-priority for Office + PDF):

FamilyExtensions
PDFpdf
Officedocx, doc, pptx, ppt, xlsx, xls
OpenDocumentodt, odp, ods
Web/markuphtml, htm, xml, md, rst
Plain/tabulartxt, rtf, csv, tsv
Images (OCR)png, jpg, jpeg, tiff, tif, bmp, webp, gif
E-maileml, msg
eBookepub

Everything on the 46-item checklist (Anlage C.I.3) — Jahresabschlüsse (PDF/xlsx), organigrams (pptx/PDF), price lists (xlsx), scanned contracts (images/PDF) — is covered. German OCR/extraction is first-class (Docling, PaddleOCR 80+ languages).

Size limits. docfold defines max_upload_size_mb = 100 (docfold/src/docfold/api/core/config.py:33, overridable via DOCFOLD_MAX_UPLOAD_SIZE_MB), but no enforcement of that setting was found on the /convert route — practical limits are imposed by the layers in front. TODO-VERIFY before accepting large scanned PDFs: (a) whether docfold enforces max_upload_size_mb anywhere, (b) the Next.js BFF formData() buffering limit on /next/api/upload/start, (c) Traefik and the Plesk nginx client_max_body_size on app.grag.ai (nginx defaults to 1 MB if never configured — same nginx layer that needs proxy_buffering off for SSE, plan 0.10b). Recommendation: enforce a Knoll-side limit of 50 MB per file in the upload UI until verified, and reject earlier with a clear German error message.

4. Ingest DAG defaults (what actually runs)

Per plan 1.7, tenant knoll gets TENANT_SETTINGS_ENABLED=true plus a tenant-level ingest.dag_template. Behaviour matrix:

Config stateDAG executed
No tenant template (dev default)chunking(strategy=markdown→structural, size=500, overlap=50)voyager_index_ingest(collection=kb_id)
Knoll template (plan 1.7)adds an enrichment node (the default DAG has none — required if §7 uses gliner_structured in-lane)
Invalid templatelogged + silent fallback to default DAG — verify the template took effect via 05-verification-runbook.md

KG segment emission (KG_INGEST_ENABLED=true, plan 1.4) happens as a side effect of the chunking stage when workspace_id rides the envelope — this is what feeds full-text expansion in chat (09-chat-integration.md) and entity extraction against the knoll-advisory schema. ⚠ Relation auto-extraction stays dormant only because schema v1 declares no relationships block (plan 3.4/D7; KG_RELATION_EXTRACTION_ENABLED=true in the prod env example).

5. Progress: SSE + poll fallback (plan 4.2)

5.1 SSE (primary)

curl -N "$GRAG_URL/next/api/orchestrator/events/ingest_ab12cd_1751882400000" \
  -H "Accept: text/event-stream" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT"

The BFF bridges the orchestrator's Redis pub/sub channel verbatim; each SSE event: name is the publisher's typed event field (pipeline_common/events.py). Contract:

SSE eventPayload (key fields)MeaningTerminal?
subscribed{channel, jobId, tenant}BFF hello, channel is liveno
status{status, progress}generic transition (e.g. STARTED)no
batch{status, progress, node_ids[]}a parallel node batch startsno
node_start{node_id, service_type}node dispatchedno
node_complete{node_id, service_type, latency_ms, cache_hit, streamed}node doneno
node_fail{node_id, service_type, error, retriable}node attempt failed (retries may follow)no
fallback_triggered{from_model, to_model, reason}ai-gateway model fallback (LLM nodes only)no
budget_exceeded{metric, limit, actual, action}job budget blown; action=cancelcancelled followsno
completed{status:"COMPLETED", progress:100}successyes
failed{status:"FAILED", progress, error}pipeline failed (DLQ entry written)yes
cancelled{status:"CANCELLED", progress}cooperative cancel landed (NO DLQ entry)yes
timeout{jobId}BFF's 30-min stream ceiling hit — NOT a job statestream ends
error{message}bridge/Redis error — NOT a job statevaries

Keepalives arrive as SSE comments (: keepalive …) every 20 s. Client rules: reconnect on socket drop (events are pub/sub — missed events are gone, so after any reconnect immediately poll §5.2 to resync); treat timeout/error as "switch to polling", never as job failure. ⚠ SSE must pass unbuffered through the Plesk nginx front (plan 0.10b) — verify before relying on it; polling is the fallback either way.

5.2 Poll fallback

curl -sS "$GRAG_URL/next/api/pipeline/status/ingest_ab12cd_1751882400000" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
{
  "jobId": "ingest_ab12cd_1751882400000",
  "tenant": "knoll",
  "status": "COMPLETED",
  "progress": 100,
  "createdAt": "2026-07-06T09:00:00.000Z",
  "updatedAt": "2026-07-06T09:00:41.000Z",
  "results": { "chunk_node": { "chunks": 42 } },
  "error": null
}

Status values observed in code: queued (seeded by the BFF), STARTED, then terminal COMPLETED | FAILED | CANCELLED — compare case-insensitively and only act on terminal states. results.<chunkNodeId>.chunks is the chunk count for the §2 step-7 PATCH. 404 = unknown job (status hash has no TTL, so 404 shortly after submit means wrong tenant header). The status route is tenant-prefix-aware (t:{tenant}:pipeline_status:{jobId} when TENANT_KEYS_ENABLED=true), so it keeps working across the plan-1.5 cutover.

5.3 UI mapping (checklist tab, 6.3)

Pipeline stategrag_refs.sync_status (06 §3.15)Checklist item (UI)
uploading / converting (steps 2–4)pending (grag_job_id = docfold job id)unchanged (e.g. "Requested") + spinner "Wird verarbeitet…" on the file row
indexing (step 5–6 running)pending (grag_job_id = ingest jobId)unchanged + progress % from batch/status events
completed + PATCH donesyncedsuggestion badge "Received?" (§7) — status changes only on human confirm
any failure (§6)failedunchanged + red error + "Re-upload"

The converting-vs-indexing distinction is derived (which job id is in grag_refs.grag_job_id / which lane is being polled), not a persisted enum — grag_sync_status has no such values. The GRAG-side document status (queued→processing→indexed|failed|superseded) is read live from the workspaces inventory (cache-only, plan 4.4).

The checklist status enum stays the prototype's: Open | Requested | PartiallyReceived | Received | NotAvailable (apps/web/lib/types.ts). Pipeline states never write it directly.

6. Failure matrix (plan 4.2)

Principle: the user's retry path is always re-upload from the Knoll file store (the original is safe there, §2 step 2). The orchestrator DLQ (pipeline_dlq, 30 d TTL, replay via make replay-dlq TENANT=knoll or the BFF DLQ routes) is operator-only — never surface it to users.

#FailureDetectionGRAG-side stateUser-visible stateRecovery
F1docfold convert submission failsupload/start returns docfoldJobId: null + errorBFF rolled back the documents row (no ghost)error toast; checklist unchangedretry upload/start (new row)
F2ingest/start 502 (docfold result unavailable/empty, collection create failed)HTTP 502 with reasondocuments row queued; markdown still in Redis"Verarbeitung verzögert…"retry ingest/start with same ids (safe; result fetch is repeatable) — backoff 3×; then F3 handling
F3docfold job failed (bad/corrupt/encrypted file, OCR failure)poll step 4 status=failed + errordocuments row stuck queuedKnoll must DELETE /workspaces/api/v1/documents/{id} to avoid a ghost inventory rowred error on file row; checklist stays "Requested"user re-uploads (possibly better scan); original downloadable from file store
F4pipeline FAILED (chunking or voyager node exhausted 3 retries; first node exception fails the whole job)terminal failed SSE event / poll FAILED + errorDLQ entry written; documents row flips failed via lineage worker (else PATCH {"status":"failed","error":…} from Knoll)red error; checklist unchangedre-upload → new documentId; delete the failed row (purge removes any stray points via the document_points registry when WORKSPACES_DOCUMENT_PURGE_ENABLED=true, plan 1.4)
F5partial: chunked but not indexed (chunking node completed, voyager_index_ingest failed → same terminal FAILED, but some points may already be upserted)failed + results contains chunking output but no ingest node resultinventory row NOT indexed; possibly stray voyager pointssame as F4same as F4 — the DELETE-with-purge of the failed documentId is what cleans stray points; never mark the checklist item "Received" for a doc that isn't indexed
F6cancelled / budget_exceeded(action=cancel)terminal cancelledno DLQ entry; row stays queued/processing"Cancelled"re-upload or re-run ingest/start; investigate budget (14-cost-model.md)
F7SSE timeout (30-min ceiling) or stream droptimeout/socket closejob may still be runningprogress bar switches to poll modepoll §5.2 until terminal
F8GRAG deploy churn mid-ingest (~30 s container recreate, plan 7.4)502/503 bursts on any calljob survives (Redis queue) or fails per F4brief "Plattform aktualisiert…"grag-client retry/backoff (plan 2.5); resync via poll

Cross-cutting: mark grag_refs.sync_status = "failed" + last_error on every F-row (canonical enum, 06 §3.15) so 05-verification-runbook.md can sweep for stuck refs.

7. Checklist automation (plan 4.3)

Goal: when a document reaches indexed, propose which of the 46 checklist positions (Anlage C.I.3: group A "Fragenkatalog" + groups B.1–B.11, mirroring the 11 success levers — see apps/web/lib/mock-data/checklist.ts) it satisfies.

Hard UX rule (QMS human-in-the-loop): the system only ever suggests. A human confirms the match; the checklist status is never set silently. The suggestion renders as a badge ("Suggestion: B.1 Geschäftsbericht — Received?") with Accept/Reject; manual override is always available (6.3).

7.1 Approach A (default): prompt-JSON LLM classification

⚠ ai-gateway has no structured output (response_format/tools do not exist on CompletionRequest — plan 1.14). Until the gateway passes response_format through, use the plan-1.14 mechanism: prompt-based JSON + Zod validation + bounded retries (≤2), executed as an ai_runs step (agent_id = checklist classifier, nullable analysis_id semantics per plan 2.1). Stamp X-Pipeline-Id: knoll-<analysis-shortid>-classification-<runid> on the call (format per 03-id-conventions.md §8 — charset [a-z0-9-], runid = shortid of the ai_runs row).

Input: filename + first ~4,000 chars of the docfold markdown (fetch GET $GRAG_URL/docfold/api/v1/jobs/{docfoldJobId}/result once, promptly after step 4 — do NOT persist the full text in Knoll DB) + the checklist catalog (id, group, label) from Knoll DB.

curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAG_API_KEY" \
  -H "X-Tenant-ID: $GRAG_TENANT" \
  -H "X-Pipeline-Id: knoll-a3f2-classification-7f3a9c21" \
  -d '{
    "model": "gpt-4o-mini",
    "fallback": ["claude-haiku-4-5"],
    "temperature": 0,
    "max_tokens": 300,
    "messages": [
      {"role": "system", "content": "Du ordnest ein Dokument einer Checklisten-Position (Anlage C.I.3) zu. Antworte NUR mit JSON: {\"checklist_item_id\": string|null, \"confidence\": number, \"begruendung\": string}. Wenn keine Position passt: checklist_item_id = null."},
      {"role": "user", "content": "Checkliste:\n<id|gruppe|bezeichnung … 46 Zeilen>\n\nDateiname: Jahresabschluss-2025.pdf\n\nDokumentanfang:\n<markdown excerpt>"}
    ]
  }'

Validate with Zod; on parse failure retry with the error appended; after bounded retries → no suggestion (item stays as-is, document still lands in the KB — classification is best-effort). Only show suggestions with confidence ≥ 0.7 (calibrate in 8.3). Persist the suggestion on the matched checklist_items row per 06-knoll-db-schema.md: suggested_file_id = the files row, suggested_status (typically Received), suggested_rationale = the LLM rationale — this is the row the checklist tab (6.3) reads. Confidence + model land in the classifier's ai_runs row (audit trail); human confirm writes confirmed_by/confirmed_at and only then the item status — never a silent auto-set.

7.2 Approach B (option): enrichment gliner_structured

Local, no cloud cost, runs either in-lane (add the enrichment node to the tenant DAG, plan 1.7) or ad hoc via POST $GRAG_URL/enrichment/api/v1/enrich with extractors: ["gliner_structured"]. It performs schema-driven extraction (title, document_type, date, summary, …) which Knoll then maps deterministically (document_type → checklist item). TODO-VERIFY: the exact config shape for a custom extraction_schema (46 German document-type labels) and its zero-shot quality on German advisory docs — run the bake-off against Approach A on the 8.1 fixture Mandant before committing. Default to Approach A; B becomes attractive if LLM cost/latency on bulk uploads bites.

7.3 Deterministic rule hook (stays in code, never LLM)

The Knoll-methodology cap: "no business plan → MarketPosition ≤ 1,5". When the checklist item Businessplan (group B.1) is set to NotAvailable — by human action, never by classifier — the Knoll backend records a scoring-cap flag on the analysis. The expert-report pipeline's Scoring step applies it as a deterministic post-LLM cap (plan 5.3 step 4; 08-gutachten-pipeline-spec.md). Symmetric: flipping the item back to Received clears the flag. This rule is code + Knoll DB only; the KG/LLM never enforce it.

8. Document lifecycle (plan 4.4)

8.1 Supersede (replace)

Used for corrected uploads AND Fragebogen re-submits (§9.3). Order matters — the KB is never left without the document:

  1. Upload the new file through §2 steps 2–7 (optionally pass replaces_document_id=<old> to upload/start — it is echoed back only; the BFF does not chain it).
  2. Wait until the NEW document is indexed.
  3. Then:
curl -sS -X POST "$GRAG_URL/workspaces/api/v1/documents/doc_OLD/supersede" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
  -d '{"superseded_by": "doc_NEW"}'

Rules: both rows must be live; 400 on self-supersede. The old row flips to superseded; its voyager points are physically purged when WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4 — without the flag the old chunks are hidden from chat only by the BFF's live-document post-filter). Knoll side: keep the old files row + binary (history/GDPR Auskunft); flip the old grag_refs row to sync_status = "superseded" (06 §3.15 — the partial-unique constraint keeps it as history) and NULL the old row's files.grag_document_id (live-ref mirror invariant, 06 §3.14). The old→new chain lives in the workspaces documents row (superseded_by) and the grag_refs history — no extra files column.

8.2 Delete + purge

curl -sS -X DELETE "$GRAG_URL/workspaces/api/v1/documents/doc_71b2c9" \
  -H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"

Expectations, flag-dependent (verify per 05-verification-runbook.md / plan 7.5 probe):

FlagEffect of DELETE
WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4 target)soft-delete + purge job physically removes the voyager points via the document_points registry
flag off (compose default)soft-delete ONLY — vectors remain; chat hides them via post-filter; not sufficient for DSGVO-Löschung
LINEAGE_ENABLED=true + forward-deleteDELETE $GRAG_URL/lineage/api/v1/artifact/{docfold_artifact_id}?cascade=true marks the artifact subtree (Löschung runbook, 12-gdpr-compliance.md)

Knoll-side completion of a user-facing deletion: file-store binary delete + files hard-delete (or tombstone per retention policy) + grag_refs closed + audit_log entry. Note: KG entities referenced by the document are tenant-wide and have no delete route — tombstoning via properties.status="inactive" per plan 3.5; KG edges are AGE-only/lossy and disappear only via the 3.6 reconcile scope.

8.3 Reingest — v1 is a signal, nothing more

POST $GRAG_URL/workspaces/api/v1/documents/{id}/reingest {"reason": "…"} flips the row to queued and records intent; no replay worker exists (409 if the doc is superseded). Do not build on it: to actually re-process a document, run the supersede flow (§8.1) from the file-store original. Revisit if the platform grows a replay worker (13-platform-gaps-issues.md).

9. Questionnaire → markdown rendering (plan 4.5)

On questionnaire submit (wizard 6.2), the Knoll backend renders the answers into ONE markdown document and ingests it into the analysis-KB so chat (5.1b/5.2) and the expert-report pipeline (5.3 step 1–2) can retrieve and cite the client's own answers. Knoll DB (questionnaire_answers) remains the source of truth; the KB doc is a projection.

9.1 Document structure template

Structure follows the Fragenkatalog (Anlage C.I.1 + C.I.2): 16 chapters = 11 main chapters + 5 IT chapters (apps/web/lib/mock-data/questionnaire.ts): 1 Unternehmen, 2 Von der Vision zur Strategie, 3 Unternehmensmarke, 4 Wettbewerb, 5 Produkte & Dienstleistungen, 6 Preispolitik, 7 Kunden, 8 Markt & Vertrieb, 9 Planung & Controlling, 10 Marketingumsetzung, 11 Personal, IT.0 Aktuelle Probleme & Gesamtsituation, IT.1 Technische Basis, IT.2 Branchenneutrale Anwendungen, IT.3 Branchenspezifische Anwendungen, IT.4 IT-Sicherheit.

---
document_type: questionnaire-answers
analysis_id: a-03
client_id: m-04
questionnaire_version: v1
submitted_at: 2026-07-06
answers_as_of: 2026-07-06T14:32:00Z
---

# Fragebogen-Antworten — Spedition Käsmann GmbH (Analyse a-03)

## Kapitel 1: Unternehmen (Erfolgshebel: Marktposition)

### Frage 1-2: Gründungsjahr
**Antwort:** 1987

### Frage 1-8: Liegt eine aktuelle schriftliche Planung vor … (= Businessplan)
**Antwort:** Nein

## Kapitel 2: Von der Vision zur Strategie (Erfolgshebel: Strategie)

### Frage 2-1: …
**Antwort:** — (nicht beantwortet)

## Kapitel IT.0: IT — Aktuelle Probleme & Gesamtsituation
…

Formatting rules (tuned for the default markdown/structural chunker, so section_hierarchy in citations reads "Kapitel X > Frage Y"):

  • Exactly one H1; one H2 per chapter (## Kapitel {nr}: {title} + (Erfolgshebel: {lever}) where the chapter maps to one — chapters 1–11 do, IT.0–IT.4 do not); one H3 per question (### Frage {id}: {text}).
  • Answer rendering by QuestionType (apps/web/lib/types.ts): yes-no/single-choice → the chosen option verbatim; multiple-choice → bullet list of chosen options; number/scale → the value (+ unit/scale label); free-text → verbatim paragraph; table → a markdown table using the question's columns headers. Unanswered → — (nicht beantwortet) (keeps the heading, so gaps are retrievable facts — the KIU chat and Interview step exploit this).
  • YAML front-matter as above (chunked as ordinary text — harmless, and it makes the doc self-describing in citations).
  • File name: questionnaire-answers-<analysis-id>.md; workspaces document name: "Fragebogen-Antworten <analysis-title>".

9.2 Ingest

Render to a .md file (docfold routes md through unstructured — effectively pass-through), store the rendered file in the file store (cheap, gives Auskunft/audit an exact copy of what the LLM saw), then run §2 steps 3–7 against the analysis-KB with workspace_id set. In parallel, run the deterministic KG upserts (plan 3.6, once the 1.11 write route exists): Client/Metric entities + facts from the answers — see 07-kg-schema-knoll-advisory.md.

9.3 Re-submit semantics ⚠ (plan 4.5 review addition)

Answers get corrected after submission (normal case, not an edge case). On every re-submit:

  1. Re-render the full markdown from Knoll DB (never patch the old doc).
  2. Ingest as a NEW document (§2) → wait indexed.
  3. POST /documents/{old}/supersede {"superseded_by": new} (§8.1) — old answers vanish from retrieval, citations stay consistent.
  4. Re-run the plan-3.6 KG upserts — they are idempotent (AGE MERGE; same canonical entity ids per 03-id-conventions.md), so re-running converges instead of duplicating. The scheduled reconcile job (3.6) catches any missed run.
  5. Update grag_refs (new document id), keep the old rendered file in the file store for history.

10. Expert-report PDF ingest-back (plan 5.3 step 6)

When an expert report is finalized (Partner approval, 6.4): the Report step renders the PDF → file store first (master copy for the PDF export button and Auskunft) → then ingest the PDF into the analysis-KB via §2 so "Frag die Akte" (5.2) and the KIU chat (5.1b) can retrieve and cite the expert report itself.

  • workspaces document name: "ExpertReport <analysis-title> v<N>"; link files.expert_report_id; grag_refs as usual.
  • New expert-report version (re-approval) → supersede the previous expert-report document (§8.1), mirroring the versioned expert_reports rows in Knoll DB (plan 2.1).
  • Do NOT ingest draft/unreleased expert reports — only approved versions reach the KB (the KB is the client-visible corpus).
  • The PDF goes through docfold like any upload; since Knoll also has the source text, a rendered-markdown ingest instead of the PDF is acceptable if PDF conversion quality disappoints — decide during 8.1 dev end-to-end.

11. Verification hooks

  • After every §2 run in dev: GET /workspaces/api/v1/kbs/{kb}/documents shows indexed + chunk count; a search/rerank for a known phrase returns the doc (05-verification-runbook.md).
  • The plan-7.2 knoll-flow canary exercises exactly this pipeline (create KB → ingest fixture → search → chat → purge) plus the BFF-auth probe (unauthenticated POST /next/api/upload/start must 401 after 1.8; today it succeeds — that success is the 1.8 blocker evidence, plan 0.10a).
  • Purge probe (7.5): delete a fixture doc, verify its points are gone from voyager and it no longer surfaces in chat.

Open TODO-VERIFY register (this doc)

  1. Upload size limits (§3): docfold max_upload_size_mb enforcement, Next.js BFF body buffering, Traefik/Plesk nginx client_max_body_size on app.grag.ai.
  2. Per-ingest-job ledger attribution (§2 step 5): does /ledger/api/v1/ledger/totals?pipeline_id=<ingest jobId> return non-zero for the plain chunking→voyager lane (embedding cost attribution)?
  3. gliner_structured custom schema (§7.2): exact config/extraction_schema shape + German zero-shot quality for the 46 checklist labels.