Knowledge Module

The knowledge/ package owns the RAG pipeline: document ingestion, text extraction, structural chunking, vector search, and the [KNOWLEDGE] slot of the system prompt. The associated tool surface (search_documents, get_document_text, create_corpus, move_documents, classify_document, extract_document, process_document, index_content, refresh_anchor) lives in tools/knowledge.py and delegates all embedding work to shared/knowledge_service.py.

Two collaborators sit just outside the package boundary but are load-bearing here:

• shared/knowledge_service.py, the single gatekeeper for anything that gets embedded. No code outside that module is allowed to import the raw embedder.
• shared/embeddings.py, the local sentence-transformers backed BGEEmbeddingGenerator.

A third collaborator, shared/content_index.py, owns content-addressable storage (blob identity, dedup, GC reference counts). Knowledge calls into it during upload to anchor the document row to a ContentBlob.

See Engine for the orchestrator and tool dispatch path, Conversations for chat persistence, and Tools for the registry the knowledge tools register against.

Module layout

File	Responsibility
knowledge/router.py	REST endpoints: POST /api/v1/knowledge/corpora, GET /api/v1/knowledge/corpora, POST /api/v1/knowledge/upload (job-based, returns a job handle), GET /api/v1/knowledge/ingest/{job_id} (poll), POST /api/v1/knowledge/query
knowledge/ingest.py	IngestionPipeline.ingest and .ingest_text; backfill_chunk_document_ids
knowledge/extractor.py	PyPDFTextExtractor, unstructured library for DOCX/PDF/PPTX; openpyxl/pyxlsb for Excel; pypdf/python-docx/python-pptx fallbacks
knowledge/chunker.py	StructuralChunker, heading/page-aware splitting with paragraph overlap
knowledge/search.py	SQLiteVectorSearcher (brute-force cosine) and PgVectorSearcher (pgvector HNSW + lexical RRF fusion via .hybrid_search()); active impl wired by DI container
knowledge/context.py	get_prompt_context(tenant_id), the compact [KNOWLEDGE] summary block
tools/knowledge.py	Tool handlers; _exec_* dispatch from the tool registry
shared/knowledge_service.py	KnowledgeService.index, .refresh_anchor, .embed_texts, .embed_query; _split_for_embedding, _mean_pool
shared/embeddings.py	BGEEmbeddingGenerator, device selection (cuda/mps/cpu), max_seq_length=1024, per-device batch sizes, accelerator cache release
shared/content_index.py	hash_bytes, hash_dataframe, ensure_blob, count_references, find_entities_by_blob_hash

The active vector searcher is the wiring decision made by the DI container in container_config.py, Postgres deployments wire PgVectorSearcher; the development fallback path exists for local work, and the hybrid keyword-plus-vector ranking is available only on PostgreSQL.

REST endpoints

Five endpoints live in knowledge/router.py under prefix /api/v1/knowledge and are gated by capabilities resolved through shared/auth.py.

Method	Path	Capability	Notes
POST	/corpora	corpus.create	Creates a tenant-scoped Corpus row
GET	/corpora	corpus.read	Lists non-system corpora for the current tenant with document_count and chunk_count
POST	/upload	document.upload	Sanitizes + durably stores the upload synchronously, then submits ingestion as an async job and returns {job_id, filename, status: "ingesting"} immediately
GET	/ingest/{job_id}	document.upload	Polls an ingestion job: {state, progress, stage, error, result}. The completed result (document_id, chunk_count, message) is delivered exactly once, a later poll after consumption gets state only
POST	/query	search	Embeds the query, runs vector search, calls CatalystLLM.synthesize, returns answer + citations + cost

Upload is job-based, not a blocking request

Before the INGEST_FEEDBACK work (2026-08-13), POST /upload blocked through extraction and a monolithic embed of every chunk, tens of seconds of zero feedback on a real PDF, and a closed tab killed the ingest outright. It now returns a job handle immediately; the frontend polls GET /knowledge/ingest/{job_id} for honest stage progress (extracting → chunking → embedding k/n → storing → searchable). Catalog registration and the “document ready” notification happen inside the submitted job (knowledge/router.py:183-255, the _run_ingest() closure passed to AsyncExecutionService as a JobSpec), so they survive the client navigating away. This is also the reason search_documents and other model-facing paths never see a half-ingested document mid-upload: the document row and its catalog registration land together, inside the job.

The upload endpoint still has the non-obvious safety properties from before the job-ification, they all run synchronously in the request handler, before the job is even submitted (source: knowledge/router.py:96-265):

1. Filename sanitization (router.py:111-113), os.path.basename then re.sub(r"[^\w.\-]", "_", ...) to defang directory traversal and shell metacharacters.
2. Streamed read with hard size cap (router.py:117-119), shared.safe_upload.read_upload_capped enforces settings.max_upload_bytes so an attacker-sized body cannot OOM the worker before the post-read size check fires.
3. Cross-tenant corpus check (router.py:121-133), verifies the corpus belongs to the caller’s tenant OR is a system corpus before writing anything. Returns 404 (not 403) on mismatch to avoid leaking existence to other tenants.
4. Content-addressed durable write (router.py:143-158), bytes are written to uploads/documents/<hash[:2]>/<hash><ext> atomically (write to *.partial.<pid>, then os.replace). Sharding by hash prefix keeps the directory listing manageable. This is the storage that content_blobs.storage_uri references, so the file is never unlinked, it’s the durable path the job’s IngestionPipeline.ingest call reads from.

Only after all four steps succeed is the job submitted; IngestionPipeline.ingest runs inside the job worker, not in the request handler.

Ingestion pipeline

IngestionPipeline.ingest (knowledge/ingest.py:29-235) is wired with three collaborators from the DI container: the TextExtractor, the DocumentChunker, and (named embedder for legacy reasons) the KnowledgeService. The embedder name is preserved; what’s actually injected is the gatekeeper service.

Per-document flow:

1. Hash and ensure blob, SHA-256 of the file bytes via _hash_file; then shared.content_index.ensure_blob does the atomic ON CONFLICT (content_hash) DO NOTHING insert and returns a blob_id. Dedup is anchored to a durable, restart-safe row, not to the document filename.
2. Reuse-or-extract, look for an existing tenant Document with the same blob_id. If it exists AND has chunk_count > 0, return a {duplicate: true, ...} payload that tells the caller to use get_document_text/search_documents on the existing ID. If it exists with zero chunks, the prior ingest failed (scanned PDF, dead blob path, extractor crash); reuse the row id but re-extract.
3. Extract, self._extractor.extract(file_path, mime_type). Empty extraction is not a hard error, the row is still created so the model has a document_id to pivot to extract_from_image for vision OCR.
4. Chunk, self._chunker.chunk(text, {"source": filename, "corpus_id": corpus_id}).
5. Embed, self._embedder.embed_texts(chunk_texts), delegated to KnowledgeService.embed_texts, which routes through the gatekeeper rather than touching BGEEmbeddingGenerator directly.
6. Store, insert or update the Document row (with blob_id, mime_type, chunk_count); insert one ChunkRecord per chunk with document_id stamped into chunk.metadata for search attribution.

The returned dict carries duplicate, re_extracted, extractable, document_id, chunk_count, and a human-readable message whose four variants cover the cross-product of (fresh vs re-extract) × (text found vs no text found). The frontend surfaces that message verbatim and the orchestrator embeds it in file_context so the model sees the actual outcome.

ingest_text (knowledge/ingest.py:236-329) handles the web-page path: same content-hash dedup via ensure_blob, no extractor stage.

After a successful (non-duplicate, no error) upload, router.py:170-187 registers the document in the tenant-level Fabric catalog via catalog.register_document(...) so it shows up in the file picker and on the Fabric page.

backfill_chunk_document_ids (knowledge/ingest.py:331-355) is an idempotent one-shot that walks ChunkRecord rows and stamps document_id into metadata_ for chunks created before that field existed. It runs at startup.

Text extraction

PyPDFTextExtractor (knowledge/extractor.py:13-282), the class name is legacy; the actual extraction logic uses the unstructured library for rich documents.

MIME / extension	Path
application/pdf, .pdf	partition_pdf → fallback pypdf
.docx and OOXML wordprocessing	partition_docx → fallback python-docx
.pptx and OOXML presentation	partition_pptx → fallback python-pptx
.xlsx, .xls	openpyxl (read_only, data_only)
.xlsb	pyxlsb
anything else	UTF-8 text read with errors="replace"

unstructured produces an element stream. The extractor walks elements (extractor.py:74-113) and renders:

• Title → # … ###### based on metadata.category_depth (default H2)
• Header → # …
• Table → markdown via _html_table_to_markdown from metadata.text_as_html; attached to the preceding element if it looks like a table caption (“table”, “pricing”, “following”)
• PageBreak → skipped
• everything else (NarrativeText, ListItem, Text, …) → raw text

This is why the user-visible incident “Word tables dropped during ingestion” is no longer a class of bug here, tables ride into the chunk stream as markdown, not as silently-discarded Table elements. Excel extractors emit one [Sheet: name] block per sheet, with header row + separator + data rows, so spreadsheets remain searchable as text.

Chunking

StructuralChunker (knowledge/chunker.py) reads settings.chunk_size and settings.chunk_overlap and produces Chunk objects with content + metadata (source, section heading, page, chunk_index).

Two passes:

1. _split_sections, line-by-line scan. Matches [Page N] markers to track page; matches ^#{1,6}\s+(.+)$ to track section heading. When a heading fires, the current section is flushed.
2. _split_with_overlap, applied only when a section exceeds chunk_size. Splits on paragraph boundaries (\n\n), accumulating paragraphs until chunk_size would be exceeded, then flushes and primes the next chunk with trailing paragraphs that fit inside chunk_overlap. A paragraph that on its own exceeds chunk_size is hard-split on line boundaries first.

The section metadata propagates into search results, and _paginate_chunks in tools/knowledge.py:199-261 re-emits ## <section> headers (with (continued) markers) at the top of each page of get_document_text.

Embeddings

KnowledgeService (shared/knowledge_service.py)

The single embedding gatekeeper. Callers describe what content is (kind="chunk" for retrievable rows, kind="anchor" for one mean-pooled vector on a target entity) and where it belongs (binding dict carrying some subset of corpus_id, document_id, kit_id, tool_name, primitive_id, tenant_id). The service owns chunking-for-embedding, embedding, mean-pooling, persistence, and cost attribution.

Method	Purpose
index(content, *, kind, binding, return_type="ids", metadata=None)	Persist content as embeddings. kind="chunk" writes one row per chunk into the chunks table; kind="anchor" writes a single mean-pooled vector onto the target entity. return_type="vectors" opts into returning the floats in the response.
refresh_anchor(target, content, *, return_type="ids")	Replace an existing anchor on a Kit / Tool / Primitive. Same machinery as index(kind="anchor"), but the target must already exist. Destructive-update intent is surfaced in the verb.
embed_texts(texts)	Escape hatch for callers that already chunked their content and just need vectors to attach to rows they own. The ingest pipeline uses this.
embed_query(query)	One transient vector for similarity search; truncated to DEFAULT_MAX_CHARS (3500) with a warning if over.

Internal helpers:

• _split_for_embedding(text, max_chars=3500, overlap=200), sentence-ish boundary splitter that exists so a single oversized input doesn’t blow past the embedder’s per-text limit. This is "make it fit" and is distinct from the semantic StructuralChunker in knowledge/chunker.py.
• _mean_pool(vectors), averages a list of unit-norm vectors, re-normalizes. Single input returned unchanged. Used by anchor indexing.
• _index_as_chunks, tries the StructuralChunker; on failure logs and falls back to _split_for_embedding. Persists ChunkRecord rows, either attached to a caller-supplied document_id or to a freshly-created Document keyed off corpus_id. If neither is in the binding, returns vectors without writing.
• _index_as_anchor, splits if oversized, embeds each piece, mean-pools, and writes onto the target row’s typed column (Kit.semantic_anchor, ToolDefinition.description_embedding, Primitive.description_embedding).
• _record_cost, writes a CostRecord for every embed batch. Local BGE-M3 is currently $0, but the row exists so paid embedders fold in cleanly later.

IndexResult carries chunk_ids, chunk_count, truncated_chars, cost_usd, and vectors (None unless return_type="vectors").

BGEEmbeddingGenerator (shared/embeddings.py)

Local sentence-transformers wrapper. Model ID is resolved from model_config via get_model_for_role("embedding"), never hardcoded.

• _select_device picks cuda > mps > cpu. sentence-transformers’ default ladder does not include MPS, so this is the reason Apple Silicon doesn’t pin all cores during dream-cycle batches.
• DEFAULT_MAX_SEQ_LENGTH = 1024 is applied to the loaded model. BGE-M3’s stock cap is 8192, and attention is O(seq_len²). 1024 covers anything we actually embed (tool descriptions, document chunks of a few hundred tokens) and keeps Apple GPU off the OOM cliff.
• _batch_size returns 8 (mps) / 32 (cuda) / 16 (cpu).
• embed drives the batch loop itself instead of letting sentence-transformers handle it, because the library doesn’t release accelerator memory between sub-batches. _release_accelerator_memory calls torch.mps.empty_cache() / torch.cuda.empty_cache() after every sub-batch.

Anchors versus chunks

The same embedding service produces two structurally different kinds of vectors:

• Chunks are retrievable rows stored in the chunks table; they back search_documents and the pre-fetch path.
• Anchors are single mean-pooled vectors stored on entity rows. Kit.semantic_anchor, ToolDefinition.description_embedding, and Primitive.description_embedding all live here. They back search_tools, find_similar_tools, search_kits, and PrimitiveRegistry.find.

The index_content and refresh_anchor tools (tools/knowledge.py:18-92) are the canonical way for the platform (or a tool) to deposit a vector onto an entity, they ensure the same chunking/embedding/persistence path runs for the dream cycle and for ad-hoc indexing.

Vector search

Two implementations of VectorSearcher are available; the DI container picks one. Both now expose two entry points: .search() (pure vector) and .hybrid_search() (the one search_documents actually calls).

SQLiteVectorSearcher (knowledge/search.py:18-155) loads every ChunkRecord for the tenant (with optional document_id / corpus_id filters), computes cosine similarity in Python, sorts, returns top_k. Embeddings stored as JSON are parsed on the fly. Intentionally simple, local dev only. Its hybrid_search() has no lexical channel and just delegates to .search() so callers can invoke hybrid_search() unconditionally regardless of backend.

PgVectorSearcher (knowledge/search.py:158-370) pushes ranking server-side via pgvector’s cosine_distance (<=>) on ChunkRecord.embedding. The Python pass recomputes the dot product to attach a similarity score to each result (the SQL ORDER BY doesn’t return the distance). This is the production path; HNSW index on chunks.embedding is assumed.

Hybrid retrieval (RRF fusion)

PgVectorSearcher.hybrid_search() (knowledge/search.py:278-370) is Reciprocal Rank Fusion (RRF_K = 60) over two channels, each pulled CHANNEL_DEPTH=300 deep:

• Vector channel, the existing .search() call.
• Lexical channel, a heading-weighted Postgres full-text query against chunks.search_tsv, built by lexical_tsquery() (individually-quoted lexemes OR’d together and cast to ::tsquery, deliberately avoiding a second to_tsquery() normalization pass that was found to silently double-stem terms). Heading weight tiers (LEXICAL_WEIGHTS = "{0.05,0.1,0.2,1.0}") bias toward chunks whose section heading matches over ones that only match in body text.

Per-chunk rank in each channel contributes 1/(RRF_K + rank); results are sorted by the summed fused score and truncated to top_k. Fusion determines ordering only, each SearchResult.score returned to callers stays the plain cosine similarity (or 0.0 for a lexical-only hit that never entered the vector channel’s top-N), preserving the [0,1] contract other consumers of .search() rely on. Per-chunk _vector_rank / _lexical_rank / _fused_score ride in chunk.metadata for telemetry and for search_documents’s relevance banner (a lexical-only top hit renders as **Search relevance: LEXICAL MATCH** rather than a misleadingly low cosine score).

Both .search() implementations return list[SearchResult] where SearchResult.chunk.metadata["document_id"] is always populated, if the metadata column is missing it (older rows where backfill_chunk_document_ids hasn’t run), the searcher falls back to record.document_id so attribution is never broken.

Tool surface

The handlers in tools/knowledge.py are thin shims over the services above. Each returns a ToolResult (engine/tool_result.py).

Tool	Handler	Notes
create_corpus	_exec_create_corpus	Tenant-scoped duplicate-name check, then insert.
move_documents	_exec_move_documents	Verifies target corpus belongs to tenant; supports move_all and per-id move; lists available documents when neither is supplied.
get_document_text	_exec_get_document_text	Pagination via _paginate_chunks over ordered ChunkRecord rows. Single-page documents emit the legacy format unchanged. Multi-page responses carry explicit next/previous-page tool calls and an search_documents alternative.
search_documents	_exec_search_documents	Anonymizes the query, embeds via KnowledgeService.embed_query, searches, runs CatalystLLM.synthesize, de-anonymizes, returns synthesis + raw source excerpts.
index_content	_exec_index_content	Single gatekeeper for embedding-backed indexing (chunk or anchor) for tools/dream cycle.
refresh_anchor	_exec_refresh_anchor	Destructive anchor refresh on Kit / Tool / Primitive.
classify_document	_exec_classify_document	Resolves text from document_id if needed; loads taxonomy; calls the LLM via the call_llm tool; parses JSON; emits a classification_result card.
extract_document	_exec_extract_document	Schema-based when a schema_id/schema_name is supplied; otherwise schemaless table extraction via SCHEMALESS_EXTRACTION. Emits an extraction_result card.
process_document	_exec_process_document	Classify → (if low-confidence) suggest new type and extend taxonomy → find or auto-generate schema → extract → persist ProcessingResult. Self-improving in the sense that new document types extend the taxonomy in-place.

Four behaviors in search_documents (tools/knowledge.py:529-680) are load-bearing:

1. Relevance signal in-band. The tool computes top_score and avg_score, then prepends a **Search relevance: LOW|MODERATE|HIGH** line (or a LEXICAL MATCH variant, see hybrid search below). LOW explicitly tells the model not to re-search with different keywords because the corpus doesn’t contain the content. This is the tool’s contract for telling the model whether to keep digging.
2. Raw source excerpts alongside synthesis, never truncated. The handler surfaces all top_k retrieved chunks (currently 10; settings.top_k) with their full, untruncated content and document_id, each prefixed [Source N (document_id=…), relevance: …%, vec#…/lex#…]. This is enforced in code, not just convention, the comment at tools/knowledge.py:650-656 calls it out explicitly as a no-truncation contract: chunk text is bounded only by the model’s context window (managed by compaction), never by the tool. The synthesizer can summarize away the actual values; the excerpts cannot.
3. Synthesis is anonymized. Queries go through container.anonymizer() before the LLM sees them; the synthesis answer is de-anonymized before return. The chunks themselves are NOT anonymized in the excerpt block, the user reading them owns the tenant.
4. Retrieval is hybrid, not pure vector. search_documents calls searcher.hybrid_search(...) unconditionally (not .search()), see “Vector search” below for what that does on each backend.

get_document_text paginates at 25,000 chars per page by default (_DOC_PAGE_SIZE_DEFAULT), which is roughly 6,000 tokens. The threshold matches one page so a document right at the boundary still fits in a single response without pagination metadata. Section markers (## …) are re-emitted as ## … (continued) at the top of every page that splits a section.

Pre-fetch context

This is where the Knowledge-Is-Routing principle materializes. Before the orchestrator builds the system prompt, engine/prompt_builder.py:_pre_fetch_context queries up to three system corpora using the platform’s own vector search and folds the hits into the prompt as [SUGGESTED CONTEXT].

container.knowledge_service().embed_query(user_message)
   → container.vector_searcher().search(..., corpus_id=<system_corpus>)
   → top_k chunk texts (and optionally chunk metadata for back-references)

The corpora consulted, all tagged system=True:

Corpus	Used for	top_k
operational_patterns	Recurring task patterns the dream cycle extracted from telemetry. Rendered as [SUGGESTED CONTEXT] patterns.	3
user_behavior	Per-user behavioural observations. Rendered as [SUGGESTED CONTEXT] preferences.	2
data_catalog	Semantically-matched catalog entries for this user message. Rendered into [CURRENT_STATE] as relevant_entry_ids. Gated off by default.	25

The data_catalog pre-fetch is currently gated by PREFETCH_DATA_CATALOG = False (engine/prompt_builder.py:33). The hypothesis under test is that the chat model reads intent well enough to call search_catalog on its own, so pre-listing semantically-matched datasets just spends tokens. When the gate is off, only this-conversation entries surface; operational_patterns and user_behavior pre-fetch are unaffected.

Two robustness notes that matter when touching this path:

• The success/failure shape of vector search is r.chunk.content (not r.content). The wrong shape was the cause of [SUGGESTED CONTEXT] silently never firing, an AttributeError was caught by a bare except, the keyword fallback ran every time, and developers thought vector search was on. The current code uses r.chunk.content and the keyword fallback exists for genuine outages (no embeddings, no model).
• The keyword fallback (_keyword_search_fallback) pre-filters [PROVISIONAL] chunks before ranking. Provisional chunks are verbatim copies of past user messages and otherwise dominate overlap-ranked top-k results for any task resembling a prior task.

Pre-fetch is non-blocking: any exception returns the default empty dict, so a corpus or embedding failure cannot break the chat request.

Prompt context

knowledge/context.py:get_prompt_context(tenant_id) builds the [KNOWLEDGE] line in the system prompt. It counts non-system corpora and documents (excluding the housekeeping _system_seed_*, _provisional_entries*, _extracted_patterns* filenames) and emits a one-liner like:

- Knowledge: 14 documents across 3 corpora: rcra_guidance (corpus_id=2), tribal_consultation (corpus_id=4), epa_orders (corpus_id=7). Use `search_documents` to find content.

If the tenant has no user corpora, the block is omitted entirely. The model uses search_documents to actually find content; the prompt slot exists so the model knows the surface area, not to dump excerpts.

Content-addressable storage

Document ingestion anchors to ContentBlob via shared/content_index.py. Key entry points:

Function	Purpose
hash_bytes(data, tenant_id=None)	SHA-256 of byte content; HMAC-salted with tenant_id if TenantConfig.isolate_content is set, so isolated tenants never share blobs with anyone else.
hash_dataframe(df, tenant_id=None)	DataFrame hash via pd.util.hash_pandas_object with a column-shape fallback. Same isolation salt rule.
ensure_blob(content_hash, ...)	Atomic insert-or-find via Postgres ON CONFLICT (content_hash) DO NOTHING. The development fallback path uses a get-or-insert. Includes heal-on-write, if an existing blob’s storage_uri points at a path that no longer exists and the caller’s path does, the row is patched to the caller’s path. This was added because the early tempfile-then-unlink flow left storage_uri pointing at /var/folders/.../tmp* paths the OS purges, breaking any downstream feature that needs the original bytes (vision OCR fallback, future re-extract).
count_references(blob_id)	Live FK count across catalog_entries and documents. Used by GC. Counters are intentionally NOT stored on the blob, drifting stored counters were the original bug class this layer eliminates.
find_entities_by_blob_hash(content_hash, tenant_id=None)	Lookup catalog entries by canonical hash. Goes through content_blobs.content_hash → id → catalog_entries.blob_id.

Two ingest paths that matter for knowledge:

• IngestionPipeline.ingest calls ensure_blob(content_hash, mime_type=mime_type, storage_backend="document", storage_uri=file_path). The durable sharded path written by the upload endpoint is what storage_uri records.
• IngestionPipeline.ingest_text does the same with storage_backend="document" and storage_uri=source_url so web-page ingests carry the URL.

Direct reads of entity.content_hash are deprecated, canonical_id, group_by_content, and find_entities_by_blob_hash are the supported entry points.

Tool boundary under the uniform contract

A uniform tool contract is live across the platform: it promotes every tool’s return value to one of three named variants (Ok | Hint | Err) and routes them through a single decorator (engine/tool_registry.py:_apply_contract, wrapping every handler that has a registered ToolSpec at registration time, not yet every tool; see README.md for the current migrated/deferred split). Knowledge handlers are part of the bulk migration that registers a ToolSpec for each tool without changing the handler source, the contract is applied at the dispatch boundary, not the source, so the _exec_* functions in tools/knowledge.py keep returning ToolResult. The orchestrator-facing shape is preserved via a registry-level adapter (envelope_to_tool_result); that adapter and the legacy ToolResult shape stay in place as long as any tool remains spec-only rather than source-decorated (see README.md for the current migration status).

In contract terms:

• search_documents, get_document_text, index_content, refresh_anchor, classify_document, extract_document, process_document, create_corpus, and move_documents are tabular-or-textual results carrying cards, costs, and (for index_content) cost attribution. Once migrated, an Ok envelope carries the rendered message in data, the cards/costs on effects, and any measured schema on schema.
• Recoverable inputs, a missing document_id, a non-existent corpus, a schema_name that resolves to no schema, are the canonical Hint shape (hint_kind="schema" for unknown table/column/entity; hint_kind="fetch" for filter/cap/pagination). The handlers already emit these as content strings; the migration translates them via _translate_tool_result’s "schema hint:" / "fetch hint:" prefix recognition.
• Genuine infra failures (DI container not initialized, OOM, connection refused) become Err.

The contract’s relevance to this module is mainly that:

1. Schema must be measured, not declared. When a knowledge tool that returns tabular data is migrated, the Ok.schema for that tool is produced by engine/contract_schema.measure_schema over the actual result, not by a hand-written field list. The current handlers don’t return tabular results, but extract_document’s schemaless table mode does, and its tables payload is what measure_schema would walk.
2. Side-effects are typed. Cards (classification_result, extraction_result, cited_answer) ride on Ok.effects.cards. Cost (LLM synthesis cost in search_documents, classification/extraction LLM cost) rides on Ok.effects.cost. Hint/Err carry no effects.
3. Catalog/namespace resolution composes with search_catalog. Once resolve_dataset and resolve_catalog_entry migrate, fuzzy ranking will query the data_catalog system corpus via the same vector_searcher path that this module’s _search_corpus uses. The pre-fetch helpers in engine/prompt_builder.py already use that path; the resolvers will too. This is what makes “Knowledge Is Routing” hold across the resolver and discovery surfaces.

See Tool System for the contract runtime.

Configuration

Setting	Source	Notes
Embedding model id	model_config table, role embedding	Resolved by shared/embeddings._get_embedding_model_id(), never hardcoded
max_seq_length	BGEEmbeddingGenerator.DEFAULT_MAX_SEQ_LENGTH (1024)	Caps attention memory
Per-device batch size	_batch_size	8 (mps), 32 (cuda), 16 (cpu)
Per-embed-input cap	DEFAULT_MAX_CHARS (3500), DEFAULT_OVERLAP_CHARS (200)	Used by _split_for_embedding
Chunk size / overlap	settings.chunk_size, settings.chunk_overlap	Used by StructuralChunker
top_k for POST /query	settings.top_k	Defaults supplied per request
Upload size cap	settings.max_upload_bytes	Enforced at stream-read time
Document text pagination	_DOC_PAGE_SIZE_DEFAULT (25_000 chars)	Per-page max in get_document_text
data_catalog pre-fetch toggle	PREFETCH_DATA_CATALOG = False in engine/prompt_builder.py	Off by default; flip in code to A/B

See Glossary for [KNOWLEDGE], [SUGGESTED CONTEXT], anchor, chunk, and Kit definitions; Database Schema for the corpora, documents, chunks, and content_blobs tables.