10 — Document Pipeline: Checkliste & Uploads (Plan Phase 4)
Purpose. This document specifies how Knoll Analyzer gets documents (Jahresabschlüsse, Verträge, Businesspläne, Fragebogen-Antworten, Gutachten-PDFs) from the UI into a searchable GRAG Knowledge Base per Analyse — and back out again. It covers the end-to-end upload sequence (plan 4.1), progress reporting (4.2), checklist automation (4.3), document lifecycle (4.4), Fragebogen-to-markdown rendering (4.5), and the Gutachten-PDF ingest-back (5.3 step 6). An engineer should be able to implement the Knoll-backend side of Phase 4 from this document plus 02-grag-api-cookbook.md.
Status / verified against: 2026-07-06, repo document-processing-pipelines @ db63a95. Endpoints and payloads verified in source (frontend-next/apps/web/app/api/{upload,ingest,orchestrator,pipeline}/…, docfold/src/docfold/engines/router.py, workspaces/src/workspaces/routers/documents.py, pipeline_common/events.py). Review corrections from the adversarial plan review are applied throughout (⚠ markers).
Sibling docs: 01-architecture.md (big picture), 02-grag-api-cookbook.md (all endpoints + error taxonomy), 03-id-conventions.md (id shapes), 06-knoll-db-schema.md (dateien, grag_refs, checklist_items), 08-gutachten-pipeline-spec.md (consumes indexed docs), 09-chat-integration.md (retrieval over the same KBs), 11-resilience-and-errors.md (retry policy), 12-gdpr-compliance.md (purge/Auskunft), 13-platform-gaps-issues.md (1.8 BFF auth blocker et al.).
1. Preconditions & conventions
- All examples use
$GRAG_URL(=https://app.grag.ai),$GRAG_API_KEY(dedicated Knoll key, plan 1.1),$GRAG_TENANT(=knollin prod,knoll-devin dev). External URL pattern:https://app.grag.ai/<traefik-prefix>/<internal-path>. - Every request carries
Authorization: Bearer $GRAG_API_KEYandX-Tenant-ID: $GRAG_TENANT. ⚠ The/next/api/*BFF endpoints are unauthenticated today and trust any inboundX-Tenant-ID(plan 1.8, hard blocker before real Mandanten data). Send theAuthorizationheader anyway so Knoll's calls keep working unchanged once 1.8 lands;X-Tenant-IDis functionally required on the BFF (missing header silently falls back to tenantdefault— thegrag-clienthard-fails on a missing tenant, plan 2.5). - The analysis's KB (
kb-analysis-<shortid>, provisioned per plan 3.1 / 04-provisioning-runbook.md) and the client workspace (client-<slug>-<shortid>) must exist before the first upload. - The orchestrator has no HTTP submit endpoint — ingest submission goes through the BFF lanes below (plan D4/0.4). There is no other externally reachable path.
- ⚠ No
Idempotency-Keyheader exists anywhere on the platform (review finding [5]); idempotency on this pipeline = client-side state infiles/grag_refs+ the retry rules in §6.
2. End-to-end upload sequence (plan 4.1)
Swimlane-style flow. Lanes: UI → Knoll backend → Knoll file store → GRAG BFF → docfold → orchestrator → voyager / kg → workspaces inventory.
UI Knoll backend File store BFF (/next/api) docfold orchestrator voyager/kg workspaces
│ (1) upload │ │ │ │ │ │ │
├──────────────▶ validate, create │ │ │ │ │ │
│ │ files row ──────────▶ (2) persist │ │ │ │ │
│ │ │ binary+sha256 │ │ │ │ │
│ │ (3) POST /next/api/upload/start ─────▶ pre-creates row ───────────────────────────────────────────────────▶ documents:
│ │ │ │ + forwards file ───▶ async convert │ │ │ status=queued
│ │ ◀── {documentId, docfoldJobId} ──────┤ │ │ │ │
│ │ (4) poll GET /docfold/api/v1/jobs/{id} until completed ───▶ │ │ │
│ │ (5) POST /next/api/ingest/start ─────▶ fetch markdown ────▶ result │ │ │
│ │ │ │ ensure collection ─────────────────────────────────▶ create if 404 │
│ │ │ │ sign + LPUSH ──────────────────────▶ chunking node │ │
│ │ ◀── {jobId} ────────────────────────┤ │ │ │ voyager_index_ingest ─▶ points│
│ ◀─(6) SSE────┼── GET /next/api/orchestrator/events/{jobId} ─────────────────────────────┤ (+ kg segment │ │
│ progress │ │ │ │ emission when KG_INGEST on) │ │
│ │ (7) on `completed`: PATCH documents row → indexed + chunks ──────────────────────────────────────────────▶ status=indexed
│ │ (8) classify → suggest checklist match (§7) │ │ │ │
│ ◀─ suggestion│ human confirms → checklist_item status "Received" │ │ │ │
Step by step, with the exact calls:
(1) UI → Knoll backend. Upload form on the analysis detail page (6.3), scoped to a checklist_item_id (or "Sonstiges"). Knoll backend authenticates the user (Knoll roles, GRAG authorizes keys not people — plan 7.7).
(2) ⚠ File store FIRST (plan 2.4). Persist the original binary to the Knoll file store before any GRAG call, and create the files row:
files: { id, firm_id, client_id?, analysis_id?, checklist_item_id?, expert_report_id?,
filename, mime, size_bytes, sha256, source, storage_key,
grag_document_id?, uploaded_by?, deleted_at? }
(Columns per 06-knoll-db-schema.md §3.14 — normative. Job ids and sync state do NOT live on files: the docfold/ingest job id goes into grag_refs.grag_job_id and the lifecycle into grag_refs.sync_status, see the id-capture points below and 06 §3.15.)
Why file-store-first (this is a hard rule, not an optimization): GRAG retains no binaries. docfold's output is markdown held in Redis job payloads ("fetch promptly"; job archive strips file_base64 > 100 KB after 30 d), and the workspaces documents table is a metadata-only inventory — there is no raw-file GET endpoint anywhere on the platform (review finding [8]). Without the file store, the prototype's per-document DownloadButton (6.3), the expert-report PDF export (6.4), re-upload after a failed ingest (§6), and the GDPR Auskunft/Löschung of originals (7.5, 12-gdpr-compliance.md) are all impossible. Dedupe hint: if sha256 already exists for this analysis, warn the user before re-ingesting.
(3) Convert — BFF upload lane.
curl -sS -X POST "$GRAG_URL/next/api/upload/start" \
-H "Authorization: Bearer $GRAG_API_KEY" \
-H "X-Tenant-ID: $GRAG_TENANT" \
-F "kb_id=kb-analysis-a3f2" \
-F "files=@/var/knoll/filestore/ab/cd/jahresabschluss-2025.pdf;filename=Jahresabschluss-2025.pdf"
Response (per file; replaces_document_id is only echoed back — see §8.1):
{
"kb_id": "kb-analysis-a3f2",
"jobs": [
{
"filename": "Jahresabschluss-2025.pdf",
"size": 2381244,
"docfoldJobId": "dfj_9x2k1",
"documentId": "doc_71b2c9",
"replacesDocumentId": null
}
]
}
What the BFF did server-side: (a) pre-created the workspaces documents row (status=queued, source_kind=upload) so the doc is visible in the inventory immediately; (b) forwarded the file to POST /docfold/api/v1/convert (async) with kb_id + document_id riding along for lineage self-emission. If convert submission fails, the BFF rolls the pre-created row back and returns docfoldJobId: null + error for that file — treat as terminal, see §6.
IDs captured now: files.grag_document_id = documentId; grag_refs row {knoll_type: "file", knoll_id: files.id, grag_type: "document", grag_tenant_id, grag_workspace_id, grag_kb_id, grag_id: documentId, grag_job_id: docfoldJobId, sync_status: "pending"} (column names + enum per 06 §3.15; pending covers the whole convert+index run until step 7). The checklist_item_id link lives on the files row (one checklist item can accumulate several files — "PartiallyReceived").
(4) Poll docfold until conversion completes (direct service call, properly authenticated):
curl -sS "$GRAG_URL/docfold/api/v1/jobs/dfj_9x2k1" \
-H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
# → {"status": "pending" | "processing" | "completed" | "failed", "progress": ..., "error": ...}
Poll every 2–5 s with backoff. failed → §6 row F1. Do not call /jobs/{id}/result yourself for the ingest hand-off — the BFF does that — but you MAY fetch it once for checklist classification input (§7); results are Redis-held, fetch promptly.
(5) Index — BFF ingest lane. Only after docfold completed:
curl -sS -X POST "$GRAG_URL/next/api/ingest/start" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAG_API_KEY" \
-H "X-Tenant-ID: $GRAG_TENANT" \
-d '{
"kb_id": "kb-analysis-a3f2",
"document_id": "doc_71b2c9",
"docfold_job_id": "dfj_9x2k1",
"filename": "Jahresabschluss-2025.pdf",
"workspace_id": "client-kaesmann-7h2k"
}'
# → {"jobId": "ingest_ab12cd_1751882400000", "tenantId": "knoll", "status": "queued"}
Always pass workspace_id (the client workspace): it rides the job envelope so the chunking stage's KG segment emission carries it and the kg-worker can resolve the workspace's ACTIVE knoll-advisory schema for entity extraction (plan 3.4, 07-kg-schema-knoll-advisory.md). Server-side the BFF: fetches the markdown from docfold (retries one 409), ensures the per-KB voyager collection ({engine:"dense", dim:1536}, 409 = exists = OK), resolves the tenant's ingest.dag_template / chunking.default_* settings (plan 1.7; defaults: strategy markdown, size 500, overlap 50), stamps parent_artifact_id (docfold CAS id) into the chunking node, signs the envelope, and LPUSHes it to pipeline_execution_queue. A 502 from this route is retriable (§6 row F2).
IDs captured now: grag_refs.grag_job_id = jobId (overwrites the docfold job id from step 3 — the column holds the last job id, diagnostic only, 06 §3.15); sync_status stays pending until step 7. The jobId doubles as the pipeline id for per-job cost lookup: GET $GRAG_URL/ledger/api/v1/ledger/totals?pipeline_id=ingest_ab12cd_1751882400000 ⚠ (correct ledger path per review finding [4]; the orchestrator's own budget poller uses the same route). TODO-VERIFY: that the chunking→voyager lane actually stamps the job id onto its paid gateway calls (embedding upsert) so /totals is non-empty for plain ingest jobs — confirmed for pageindex nodes, not explicitly traced for voyager_index_ingest.
(6) Progress — §5.
(7) Finalize inventory. On terminal completed, PATCH the documents row (belt-and-braces; the workspaces-worker does this automatically only when LINEAGE_ENABLED=true, plan 1.4 — keep the PATCH anyway so dev/dark environments behave identically):
curl -sS -X PATCH "$GRAG_URL/workspaces/api/v1/documents/doc_71b2c9" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
-d '{"status": "indexed", "chunks": 42}'
Chunk count comes from the poll response results.<chunkNodeId>.chunks on COMPLETED. Set grag_refs.sync_status = "synced" + last_synced_at (canonical enum, 06 §3.15 — the fine-grained "indexed" state lives GRAG-side on the workspaces documents row, cache-only in Knoll).
(8) Checklist suggestion — §7. The per-analysis document list in the UI (6.3) comes from GET $GRAG_URL/workspaces/api/v1/kbs/kb-analysis-a3f2/documents — cache-only in Knoll DB, never duplicated (plan 4.4); downloads come from the file store via files.storage_key.
2.1 ID-capture summary
| Where produced | Field | Persisted in | Used for |
|---|---|---|---|
| Knoll backend | files.id, storage_key, sha256 | files | downloads, GDPR, dedupe, re-upload |
/next/api/upload/start | jobs[].documentId | files.grag_document_id, grag_refs.grag_id | inventory, supersede/delete, chat citations (doc_id) |
/next/api/upload/start | jobs[].docfoldJobId | grag_refs.grag_job_id (transient) | docfold poll, ingest/start input, classification fetch |
/next/api/ingest/start | jobId | grag_refs.grag_job_id (overwrites docfold id) | SSE/poll progress, ledger /totals?pipeline_id=, audit |
| Knoll UI/user | checklist_item_id | files.checklist_item_id | checklist status automation (§7) |
3. Supported input formats
From the docfold engine router (docfold/src/docfold/engines/router.py, extension→engine priority chains; Docling is first-priority for Office + PDF):
| Family | Extensions |
|---|---|
pdf | |
| Office | docx, doc, pptx, ppt, xlsx, xls |
| OpenDocument | odt, odp, ods |
| Web/markup | html, htm, xml, md, rst |
| Plain/tabular | txt, rtf, csv, tsv |
| Images (OCR) | png, jpg, jpeg, tiff, tif, bmp, webp, gif |
eml, msg | |
| eBook | epub |
Everything on the 46-item checklist (Anlage C.I.3) — Jahresabschlüsse (PDF/xlsx), organigrams (pptx/PDF), price lists (xlsx), scanned contracts (images/PDF) — is covered. German OCR/extraction is first-class (Docling, PaddleOCR 80+ languages).
Size limits. docfold defines max_upload_size_mb = 100 (docfold/src/docfold/api/core/config.py:33, overridable via DOCFOLD_MAX_UPLOAD_SIZE_MB), but no enforcement of that setting was found on the /convert route — practical limits are imposed by the layers in front. TODO-VERIFY before accepting large scanned PDFs: (a) whether docfold enforces max_upload_size_mb anywhere, (b) the Next.js BFF formData() buffering limit on /next/api/upload/start, (c) Traefik and the Plesk nginx client_max_body_size on app.grag.ai (nginx defaults to 1 MB if never configured — same nginx layer that needs proxy_buffering off for SSE, plan 0.10b). Recommendation: enforce a Knoll-side limit of 50 MB per file in the upload UI until verified, and reject earlier with a clear German error message.
4. Ingest DAG defaults (what actually runs)
Per plan 1.7, tenant knoll gets TENANT_SETTINGS_ENABLED=true plus a tenant-level ingest.dag_template. Behaviour matrix:
| Config state | DAG executed |
|---|---|
| No tenant template (dev default) | chunking(strategy=markdown→structural, size=500, overlap=50) → voyager_index_ingest(collection=kb_id) |
| Knoll template (plan 1.7) | adds an enrichment node (the default DAG has none — required if §7 uses gliner_structured in-lane) |
| Invalid template | logged + silent fallback to default DAG — verify the template took effect via 05-verification-runbook.md |
KG segment emission (KG_INGEST_ENABLED=true, plan 1.4) happens as a side effect of the chunking stage when workspace_id rides the envelope — this is what feeds full-text expansion in chat (09-chat-integration.md) and entity extraction against the knoll-advisory schema. ⚠ Relation auto-extraction stays dormant only because schema v1 declares no relationships block (plan 3.4/D7; KG_RELATION_EXTRACTION_ENABLED=true in the prod env example).
5. Progress: SSE + poll fallback (plan 4.2)
5.1 SSE (primary)
curl -N "$GRAG_URL/next/api/orchestrator/events/ingest_ab12cd_1751882400000" \
-H "Accept: text/event-stream" \
-H "Authorization: Bearer $GRAG_API_KEY" \
-H "X-Tenant-ID: $GRAG_TENANT"
The BFF bridges the orchestrator's Redis pub/sub channel verbatim; each SSE event: name is the publisher's typed event field (pipeline_common/events.py). Contract:
| SSE event | Payload (key fields) | Meaning | Terminal? |
|---|---|---|---|
subscribed | {channel, jobId, tenant} | BFF hello, channel is live | no |
status | {status, progress} | generic transition (e.g. STARTED) | no |
batch | {status, progress, node_ids[]} | a parallel node batch starts | no |
node_start | {node_id, service_type} | node dispatched | no |
node_complete | {node_id, service_type, latency_ms, cache_hit, streamed} | node done | no |
node_fail | {node_id, service_type, error, retriable} | node attempt failed (retries may follow) | no |
fallback_triggered | {from_model, to_model, reason} | ai-gateway model fallback (LLM nodes only) | no |
budget_exceeded | {metric, limit, actual, action} | job budget blown; action=cancel → cancelled follows | no |
completed | {status:"COMPLETED", progress:100} | success | yes |
failed | {status:"FAILED", progress, error} | pipeline failed (DLQ entry written) | yes |
cancelled | {status:"CANCELLED", progress} | cooperative cancel landed (NO DLQ entry) | yes |
timeout | {jobId} | BFF's 30-min stream ceiling hit — NOT a job state | stream ends |
error | {message} | bridge/Redis error — NOT a job state | varies |
Keepalives arrive as SSE comments (: keepalive …) every 20 s. Client rules: reconnect on socket drop (events are pub/sub — missed events are gone, so after any reconnect immediately poll §5.2 to resync); treat timeout/error as "switch to polling", never as job failure. ⚠ SSE must pass unbuffered through the Plesk nginx front (plan 0.10b) — verify before relying on it; polling is the fallback either way.
5.2 Poll fallback
curl -sS "$GRAG_URL/next/api/pipeline/status/ingest_ab12cd_1751882400000" \
-H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
{
"jobId": "ingest_ab12cd_1751882400000",
"tenant": "knoll",
"status": "COMPLETED",
"progress": 100,
"createdAt": "2026-07-06T09:00:00.000Z",
"updatedAt": "2026-07-06T09:00:41.000Z",
"results": { "chunk_node": { "chunks": 42 } },
"error": null
}
Status values observed in code: queued (seeded by the BFF), STARTED, then terminal COMPLETED | FAILED | CANCELLED — compare case-insensitively and only act on terminal states. results.<chunkNodeId>.chunks is the chunk count for the §2 step-7 PATCH. 404 = unknown job (status hash has no TTL, so 404 shortly after submit means wrong tenant header). The status route is tenant-prefix-aware (t:{tenant}:pipeline_status:{jobId} when TENANT_KEYS_ENABLED=true), so it keeps working across the plan-1.5 cutover.
5.3 UI mapping (checklist tab, 6.3)
| Pipeline state | grag_refs.sync_status (06 §3.15) | Checklist item (UI) |
|---|---|---|
| uploading / converting (steps 2–4) | pending (grag_job_id = docfold job id) | unchanged (e.g. "Requested") + spinner "Wird verarbeitet…" on the file row |
| indexing (step 5–6 running) | pending (grag_job_id = ingest jobId) | unchanged + progress % from batch/status events |
completed + PATCH done | synced | suggestion badge "Received?" (§7) — status changes only on human confirm |
| any failure (§6) | failed | unchanged + red error + "Re-upload" |
The converting-vs-indexing distinction is derived (which job id is in grag_refs.grag_job_id / which lane is being polled), not a persisted enum — grag_sync_status has no such values. The GRAG-side document status (queued→processing→indexed|failed|superseded) is read live from the workspaces inventory (cache-only, plan 4.4).
The checklist status enum stays the prototype's: Open | Requested | PartiallyReceived | Received | NotAvailable (apps/web/lib/types.ts). Pipeline states never write it directly.
6. Failure matrix (plan 4.2)
Principle: the user's retry path is always re-upload from the Knoll file store (the original is safe there, §2 step 2). The orchestrator DLQ (pipeline_dlq, 30 d TTL, replay via make replay-dlq TENANT=knoll or the BFF DLQ routes) is operator-only — never surface it to users.
| # | Failure | Detection | GRAG-side state | User-visible state | Recovery |
|---|---|---|---|---|---|
| F1 | docfold convert submission fails | upload/start returns docfoldJobId: null + error | BFF rolled back the documents row (no ghost) | error toast; checklist unchanged | retry upload/start (new row) |
| F2 | ingest/start 502 (docfold result unavailable/empty, collection create failed) | HTTP 502 with reason | documents row queued; markdown still in Redis | "Verarbeitung verzögert…" | retry ingest/start with same ids (safe; result fetch is repeatable) — backoff 3×; then F3 handling |
| F3 | docfold job failed (bad/corrupt/encrypted file, OCR failure) | poll step 4 status=failed + error | documents row stuck queued — Knoll must DELETE /workspaces/api/v1/documents/{id} to avoid a ghost inventory row | red error on file row; checklist stays "Requested" | user re-uploads (possibly better scan); original downloadable from file store |
| F4 | pipeline FAILED (chunking or voyager node exhausted 3 retries; first node exception fails the whole job) | terminal failed SSE event / poll FAILED + error | DLQ entry written; documents row flips failed via lineage worker (else PATCH {"status":"failed","error":…} from Knoll) | red error; checklist unchanged | re-upload → new documentId; delete the failed row (purge removes any stray points via the document_points registry when WORKSPACES_DOCUMENT_PURGE_ENABLED=true, plan 1.4) |
| F5 | partial: chunked but not indexed (chunking node completed, voyager_index_ingest failed → same terminal FAILED, but some points may already be upserted) | failed + results contains chunking output but no ingest node result | inventory row NOT indexed; possibly stray voyager points | same as F4 | same as F4 — the DELETE-with-purge of the failed documentId is what cleans stray points; never mark the checklist item "Received" for a doc that isn't indexed |
| F6 | cancelled / budget_exceeded(action=cancel) | terminal cancelled | no DLQ entry; row stays queued/processing | "Cancelled" | re-upload or re-run ingest/start; investigate budget (14-cost-model.md) |
| F7 | SSE timeout (30-min ceiling) or stream drop | timeout/socket close | job may still be running | progress bar switches to poll mode | poll §5.2 until terminal |
| F8 | GRAG deploy churn mid-ingest (~30 s container recreate, plan 7.4) | 502/503 bursts on any call | job survives (Redis queue) or fails per F4 | brief "Plattform aktualisiert…" | grag-client retry/backoff (plan 2.5); resync via poll |
Cross-cutting: mark grag_refs.sync_status = "failed" + last_error on every F-row (canonical enum, 06 §3.15) so 05-verification-runbook.md can sweep for stuck refs.
7. Checklist automation (plan 4.3)
Goal: when a document reaches indexed, propose which of the 46 checklist positions (Anlage C.I.3: group A "Fragenkatalog" + groups B.1–B.11, mirroring the 11 success levers — see apps/web/lib/mock-data/checklist.ts) it satisfies.
Hard UX rule (QMS human-in-the-loop): the system only ever suggests. A human confirms the match; the checklist status is never set silently. The suggestion renders as a badge ("Suggestion: B.1 Geschäftsbericht — Received?") with Accept/Reject; manual override is always available (6.3).
7.1 Approach A (default): prompt-JSON LLM classification
⚠ ai-gateway has no structured output (response_format/tools do not exist on CompletionRequest — plan 1.14). Until the gateway passes response_format through, use the plan-1.14 mechanism: prompt-based JSON + Zod validation + bounded retries (≤2), executed as an ai_runs step (agent_id = checklist classifier, nullable analysis_id semantics per plan 2.1). Stamp X-Pipeline-Id: knoll-<analysis-shortid>-classification-<runid> on the call (format per 03-id-conventions.md §8 — charset [a-z0-9-], runid = shortid of the ai_runs row).
Input: filename + first ~4,000 chars of the docfold markdown (fetch GET $GRAG_URL/docfold/api/v1/jobs/{docfoldJobId}/result once, promptly after step 4 — do NOT persist the full text in Knoll DB) + the checklist catalog (id, group, label) from Knoll DB.
curl -sS -X POST "$GRAG_URL/ai-gateway/api/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAG_API_KEY" \
-H "X-Tenant-ID: $GRAG_TENANT" \
-H "X-Pipeline-Id: knoll-a3f2-classification-7f3a9c21" \
-d '{
"model": "gpt-4o-mini",
"fallback": ["claude-haiku-4-5"],
"temperature": 0,
"max_tokens": 300,
"messages": [
{"role": "system", "content": "Du ordnest ein Dokument einer Checklisten-Position (Anlage C.I.3) zu. Antworte NUR mit JSON: {\"checklist_item_id\": string|null, \"confidence\": number, \"begruendung\": string}. Wenn keine Position passt: checklist_item_id = null."},
{"role": "user", "content": "Checkliste:\n<id|gruppe|bezeichnung … 46 Zeilen>\n\nDateiname: Jahresabschluss-2025.pdf\n\nDokumentanfang:\n<markdown excerpt>"}
]
}'
Validate with Zod; on parse failure retry with the error appended; after bounded retries → no suggestion (item stays as-is, document still lands in the KB — classification is best-effort). Only show suggestions with confidence ≥ 0.7 (calibrate in 8.3). Persist the suggestion on the matched checklist_items row per 06-knoll-db-schema.md: suggested_file_id = the files row, suggested_status (typically Received), suggested_rationale = the LLM rationale — this is the row the checklist tab (6.3) reads. Confidence + model land in the classifier's ai_runs row (audit trail); human confirm writes confirmed_by/confirmed_at and only then the item status — never a silent auto-set.
7.2 Approach B (option): enrichment gliner_structured
Local, no cloud cost, runs either in-lane (add the enrichment node to the tenant DAG, plan 1.7) or ad hoc via POST $GRAG_URL/enrichment/api/v1/enrich with extractors: ["gliner_structured"]. It performs schema-driven extraction (title, document_type, date, summary, …) which Knoll then maps deterministically (document_type → checklist item). TODO-VERIFY: the exact config shape for a custom extraction_schema (46 German document-type labels) and its zero-shot quality on German advisory docs — run the bake-off against Approach A on the 8.1 fixture Mandant before committing. Default to Approach A; B becomes attractive if LLM cost/latency on bulk uploads bites.
7.3 Deterministic rule hook (stays in code, never LLM)
The Knoll-methodology cap: "no business plan → MarketPosition ≤ 1,5". When the checklist item Businessplan (group B.1) is set to NotAvailable — by human action, never by classifier — the Knoll backend records a scoring-cap flag on the analysis. The expert-report pipeline's Scoring step applies it as a deterministic post-LLM cap (plan 5.3 step 4; 08-gutachten-pipeline-spec.md). Symmetric: flipping the item back to Received clears the flag. This rule is code + Knoll DB only; the KG/LLM never enforce it.
8. Document lifecycle (plan 4.4)
8.1 Supersede (replace)
Used for corrected uploads AND Fragebogen re-submits (§9.3). Order matters — the KB is never left without the document:
- Upload the new file through §2 steps 2–7 (optionally pass
replaces_document_id=<old>toupload/start— it is echoed back only; the BFF does not chain it). - Wait until the NEW document is
indexed. - Then:
curl -sS -X POST "$GRAG_URL/workspaces/api/v1/documents/doc_OLD/supersede" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT" \
-d '{"superseded_by": "doc_NEW"}'
Rules: both rows must be live; 400 on self-supersede. The old row flips to superseded; its voyager points are physically purged when WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4 — without the flag the old chunks are hidden from chat only by the BFF's live-document post-filter). Knoll side: keep the old files row + binary (history/GDPR Auskunft); flip the old grag_refs row to sync_status = "superseded" (06 §3.15 — the partial-unique constraint keeps it as history) and NULL the old row's files.grag_document_id (live-ref mirror invariant, 06 §3.14). The old→new chain lives in the workspaces documents row (superseded_by) and the grag_refs history — no extra files column.
8.2 Delete + purge
curl -sS -X DELETE "$GRAG_URL/workspaces/api/v1/documents/doc_71b2c9" \
-H "Authorization: Bearer $GRAG_API_KEY" -H "X-Tenant-ID: $GRAG_TENANT"
Expectations, flag-dependent (verify per 05-verification-runbook.md / plan 7.5 probe):
| Flag | Effect of DELETE |
|---|---|
WORKSPACES_DOCUMENT_PURGE_ENABLED=true (plan 1.4 target) | soft-delete + purge job physically removes the voyager points via the document_points registry |
| flag off (compose default) | soft-delete ONLY — vectors remain; chat hides them via post-filter; not sufficient for DSGVO-Löschung |
LINEAGE_ENABLED=true + forward-delete | DELETE $GRAG_URL/lineage/api/v1/artifact/{docfold_artifact_id}?cascade=true marks the artifact subtree (Löschung runbook, 12-gdpr-compliance.md) |
Knoll-side completion of a user-facing deletion: file-store binary delete + files hard-delete (or tombstone per retention policy) + grag_refs closed + audit_log entry. Note: KG entities referenced by the document are tenant-wide and have no delete route — tombstoning via properties.status="inactive" per plan 3.5; KG edges are AGE-only/lossy and disappear only via the 3.6 reconcile scope.
8.3 Reingest — v1 is a signal, nothing more
POST $GRAG_URL/workspaces/api/v1/documents/{id}/reingest {"reason": "…"} flips the row to queued and records intent; no replay worker exists (409 if the doc is superseded). Do not build on it: to actually re-process a document, run the supersede flow (§8.1) from the file-store original. Revisit if the platform grows a replay worker (13-platform-gaps-issues.md).
9. Questionnaire → markdown rendering (plan 4.5)
On questionnaire submit (wizard 6.2), the Knoll backend renders the answers into ONE markdown document and ingests it into the analysis-KB so chat (5.1b/5.2) and the expert-report pipeline (5.3 step 1–2) can retrieve and cite the client's own answers. Knoll DB (questionnaire_answers) remains the source of truth; the KB doc is a projection.
9.1 Document structure template
Structure follows the Fragenkatalog (Anlage C.I.1 + C.I.2): 16 chapters = 11 main chapters + 5 IT chapters (apps/web/lib/mock-data/questionnaire.ts): 1 Unternehmen, 2 Von der Vision zur Strategie, 3 Unternehmensmarke, 4 Wettbewerb, 5 Produkte & Dienstleistungen, 6 Preispolitik, 7 Kunden, 8 Markt & Vertrieb, 9 Planung & Controlling, 10 Marketingumsetzung, 11 Personal, IT.0 Aktuelle Probleme & Gesamtsituation, IT.1 Technische Basis, IT.2 Branchenneutrale Anwendungen, IT.3 Branchenspezifische Anwendungen, IT.4 IT-Sicherheit.
---
document_type: questionnaire-answers
analysis_id: a-03
client_id: m-04
questionnaire_version: v1
submitted_at: 2026-07-06
answers_as_of: 2026-07-06T14:32:00Z
---
# Fragebogen-Antworten — Spedition Käsmann GmbH (Analyse a-03)
## Kapitel 1: Unternehmen (Erfolgshebel: Marktposition)
### Frage 1-2: Gründungsjahr
**Antwort:** 1987
### Frage 1-8: Liegt eine aktuelle schriftliche Planung vor … (= Businessplan)
**Antwort:** Nein
## Kapitel 2: Von der Vision zur Strategie (Erfolgshebel: Strategie)
### Frage 2-1: …
**Antwort:** — (nicht beantwortet)
## Kapitel IT.0: IT — Aktuelle Probleme & Gesamtsituation
…
Formatting rules (tuned for the default markdown/structural chunker, so section_hierarchy in citations reads "Kapitel X > Frage Y"):
- Exactly one H1; one H2 per chapter (
## Kapitel {nr}: {title}+(Erfolgshebel: {lever})where the chapter maps to one — chapters 1–11 do, IT.0–IT.4 do not); one H3 per question (### Frage {id}: {text}). - Answer rendering by
QuestionType(apps/web/lib/types.ts):yes-no/single-choice→ the chosen option verbatim;multiple-choice→ bullet list of chosen options;number/scale→ the value (+ unit/scale label);free-text→ verbatim paragraph;table→ a markdown table using the question'scolumnsheaders. Unanswered →— (nicht beantwortet)(keeps the heading, so gaps are retrievable facts — the KIU chat and Interview step exploit this). - YAML front-matter as above (chunked as ordinary text — harmless, and it makes the doc self-describing in citations).
- File name:
questionnaire-answers-<analysis-id>.md; workspaces documentname:"Fragebogen-Antworten <analysis-title>".
9.2 Ingest
Render to a .md file (docfold routes md through unstructured — effectively pass-through), store the rendered file in the file store (cheap, gives Auskunft/audit an exact copy of what the LLM saw), then run §2 steps 3–7 against the analysis-KB with workspace_id set. In parallel, run the deterministic KG upserts (plan 3.6, once the 1.11 write route exists): Client/Metric entities + facts from the answers — see 07-kg-schema-knoll-advisory.md.
9.3 Re-submit semantics ⚠ (plan 4.5 review addition)
Answers get corrected after submission (normal case, not an edge case). On every re-submit:
- Re-render the full markdown from Knoll DB (never patch the old doc).
- Ingest as a NEW document (§2) → wait
indexed. POST /documents/{old}/supersede {"superseded_by": new}(§8.1) — old answers vanish from retrieval, citations stay consistent.- Re-run the plan-3.6 KG upserts — they are idempotent (AGE MERGE; same canonical entity ids per 03-id-conventions.md), so re-running converges instead of duplicating. The scheduled reconcile job (3.6) catches any missed run.
- Update
grag_refs(new document id), keep the old rendered file in the file store for history.
10. Expert-report PDF ingest-back (plan 5.3 step 6)
When an expert report is finalized (Partner approval, 6.4): the Report step renders the PDF → file store first (master copy for the PDF export button and Auskunft) → then ingest the PDF into the analysis-KB via §2 so "Frag die Akte" (5.2) and the KIU chat (5.1b) can retrieve and cite the expert report itself.
- workspaces document
name:"ExpertReport <analysis-title> v<N>"; linkfiles.expert_report_id;grag_refsas usual. - New expert-report version (re-approval) → supersede the previous expert-report document (§8.1), mirroring the versioned
expert_reportsrows in Knoll DB (plan 2.1). - Do NOT ingest draft/unreleased expert reports — only approved versions reach the KB (the KB is the client-visible corpus).
- The PDF goes through docfold like any upload; since Knoll also has the source text, a rendered-markdown ingest instead of the PDF is acceptable if PDF conversion quality disappoints — decide during 8.1 dev end-to-end.
11. Verification hooks
- After every §2 run in dev:
GET /workspaces/api/v1/kbs/{kb}/documentsshowsindexed+ chunk count; asearch/rerankfor a known phrase returns the doc (05-verification-runbook.md). - The plan-7.2
knoll-flowcanary exercises exactly this pipeline (create KB → ingest fixture → search → chat → purge) plus the BFF-auth probe (unauthenticatedPOST /next/api/upload/startmust 401 after 1.8; today it succeeds — that success is the 1.8 blocker evidence, plan 0.10a). - Purge probe (7.5): delete a fixture doc, verify its points are gone from voyager and it no longer surfaces in chat.
Open TODO-VERIFY register (this doc)
- Upload size limits (§3): docfold
max_upload_size_mbenforcement, Next.js BFF body buffering, Traefik/Plesk nginxclient_max_body_sizeon app.grag.ai. - Per-ingest-job ledger attribution (§2 step 5): does
/ledger/api/v1/ledger/totals?pipeline_id=<ingest jobId>return non-zero for the plain chunking→voyager lane (embedding cost attribution)? gliner_structuredcustom schema (§7.2): exactconfig/extraction_schemashape + German zero-shot quality for the 46 checklist labels.