Insight Module

The Insight module is Eunice’s document-intelligence layer: LLM-driven classification of documents against configurable taxonomies, schema-based or schemaless extraction of structured data, and a self-extending pipeline that grows its own taxonomies and schemas when it encounters document types it has never seen. It operates on documents already ingested by the Knowledge module (see Knowledge) and persists its outputs to the processing_results table for the Insight sidebar view.

The directory insight/ itself is small, it owns the REST surface and a prompt-context snippet. The tool handlers that do the actual work live in tools/knowledge.py (classify/extract/process) and tools/insight.py (taxonomy and schema CRUD). That split is a historical artifact: classification and extraction were originally collocated with the document-text retrieval code in tools/knowledge.py and were never moved. New work should keep the split rather than thrash it.

Module Files

File	Responsibility
insight/__init__.py	Package marker, no exports.
insight/router.py	REST endpoints for the Insight sidebar page and CRUD for taxonomies and schemas.
insight/context.py	One-line [AVAILABLE_DATA] prompt fragment with taxonomy and schema counts.
tools/insight.py	Handlers for list_taxonomies, list_schemas, create_schema, update_schema, add_taxonomy_category, create_taxonomy.
tools/knowledge.py (lines 781-1304 as of 2026-09, check current line numbers, this file grows)	Handlers for classify_document, extract_document, process_document, plus _suggest_document_type and _generate_schema_fields helpers used by the self-extending pipeline.
engine/schemas/insight.py	INSIGHT_TOOLS schema list, the insight-category tools (10 as of 2026-09, including enrich_records below) as registered into SYSTEM_TOOLS.
tools/enrichment.py	Handler for enrich_records, batch record labeling; the corpus-wide sibling of extract_document.

A note on naming: shared/insight_hash.py exists in this tree but belongs to the SandDance chart-snapshot dedup path (tools/chart_builder.py, api/chart_snapshots.py). It has nothing to do with document insight, same word, different domain.

Data Model

Three tables in shared/database.py back the module:

Table	Purpose
taxonomies	Classification schemes. Each row has a categories JSON column, a list of {label, description, examples: [str]} objects, plus is_default, tenant_id, created_by.
extraction_schemas	Field templates. fields JSON column is a list of {name, type, description, required}; applies_to is a list of classification labels that auto-select this schema in the pipeline. Soft-deleted via deleted_at.
processing_results	Pipeline output. One row per process_document run, carrying the classification, extracted fields, per-dimension quality scores, overall quality, status, and cost.

Multi-tenancy follows the standard pattern: queries filter tenant_id IN [current_tenant, "*"] so per-tenant taxonomies and schemas shadow global ones with the same name. Schemas use a soft-delete (deleted_at); taxonomies do not.

Classification

classify_document

Handler: tools/knowledge.py:781 (_exec_classify_document).

Inputs:

Field	Type	Notes
text	string	Raw text. Optional if document_id is given.
document_id	int	If supplied, the handler reads every ChunkRecord for that document and joins them with \n\n. The full reconstructed text is sent to the LLM, no truncation.
taxonomy_id	int	If omitted, falls back to the taxonomy with is_default=True, then the first taxonomy in the tenant scope.

The taxonomy’s categories are rendered into a markdown block (label, description, examples) and substituted into the DOCUMENT_CLASSIFICATION prompt from shared/prompts.py. The call goes through the composable call_llm tool with role="fast", note this differs from the earlier description in some older docs that said strong; the current handler uses fast. The model’s JSON response is parsed with extract_json_from_response(); parse failure returns the raw text as the tool message but still records the cost.

The card emitted is type classification_result with this payload:

{
  "taxonomy_name": str,
  "taxonomy_id": int,
  "primary_label": str,
  "primary_confidence": float,    # 0.0-1.0
  "secondary_labels": list,
  "rationale": str,
  "document_id": int,             # if provided
}

When primary_confidence < 0.6, the tool message appends a directive telling the model to consider calling add_taxonomy_category and re-classifying. The pipeline (process_document) acts on this signal automatically; standalone classify_document calls leave it to the agent.

Extraction

Handler: tools/knowledge.py:904 (_exec_extract_document). The same handler covers both schema-based and schemaless modes, schema_id presence is the switch.

extract_document

Field	Type	Notes
text	string	Raw text, optional if document_id is given.
document_id	int	Same chunk-join reconstruction as classification.
schema_id	int	Selects schema-based mode.
schema_name	string	Alternative to schema_id; resolved via ILIKE %name% against extraction_schemas.name.

Schema-based mode

The schema’s fields list is rendered into the SCHEMA_EXTRACTION prompt (shared/prompts.py). The model returns extracted_fields (a list of per-field {name, value, ...} objects) and a per-dimension quality_scores dict. The handler computes overall quality as a fixed-weight blend:

overall = 0.3 * completeness
        + 0.2 * consistency
        + 0.2 * conformity
        + 0.3 * confidence

The dimensions are scored by the LLM in its response, not measured server-side. Anything outside those four keys is ignored.

Card type: extraction_result. Payload:

{
  "schema_name": str,
  "schema_id": int,
  "extracted_fields": list,
  "quality_scores": dict,         # 4 keys above
  "overall_quality": float,       # 0.0-1.0, rounded to 2 dp
  "document_id": int,             # if provided
}

Schemaless mode

When no schema_id/schema_name is provided, the handler uses the SCHEMALESS_EXTRACTION prompt and expects a tables array of {title, columns, rows} objects plus an optional metadata dict. The resulting card carries schema_name: "Auto-Extract", the tables list, and metadata. Useful for exploratory extraction when the document type isn’t yet modeled.

Both modes route through call_llm with role="fast" and never truncate the input text, the model’s context window is the only constraint.

The process_document Pipeline

Handler: tools/knowledge.py:1076 (_exec_process_document). This is the self-extending path and the most interesting handler in the module, it composes four other tools and grows the platform’s own taxonomy and schema rows when it encounters something new.

Steps:

1. Classify. Calls classify_document with the inputs as-is.
2. Low-confidence recovery. If primary_confidence < 0.6 and we have text, calls _suggest_document_type (tools/knowledge.py:1242). That helper runs the DOCUMENT_TYPE_SUGGESTION prompt and asks the LLM to name the true document type, with description and examples. If the suggested label differs from what classification returned and isn’t already in the default taxonomy, the pipeline invokes add_taxonomy_category to extend the taxonomy, then re-runs classify_document. category_was_created is set so the final message can announce the extension.
3. Schema selection. First tries to find an existing ExtractionSchema whose applies_to contains the (possibly updated) primary label.
4. Schema auto-generation. If no match and we have text, calls _generate_schema_fields (tools/knowledge.py:1273) which runs the SCHEMA_GENERATION prompt to draft a field list, validates each field has name/type/description, then calls create_schema to persist it. schema_was_created is set.
5. Extract. Calls extract_document with the resolved (or newly created) schema_id.
6. Persist. Writes a ProcessingResult row with classification, extracted fields, per-dimension quality scores, overall quality, status "completed", and the rolled-up cost (sum of every internal call_llm invocation across the steps above).

The final message reports the classification, the extraction quality percentage, and, if anything was auto-created, a short notice that future documents of this type will route through the new category and schema. Both classification and extraction cards are returned together.

This is the only place in the module that touches ProcessingResult. Standalone classify_document and extract_document calls don’t persist; they’re meant for ad-hoc agent use.

Batch Enrichment (enrich_records)

Handler: tools/enrichment.py:_exec_enrich_records (is_tabular=True, spec-only migrated). This is the corpus-wide sibling of extract_document: instead of extracting from one document, it reads every row of a cataloged frame (source input), applies an ExtractionSchema’s field definitions as closed-value labels, and catalog.saves the result as a durable label frame (<source>_labels by default) so corpus-wide questions become plain SQL via query_frames instead of repeated per-row LLM calls.

Two things distinguish it from the document-oriented tools in this module:

• Deterministic-first. A named structured parser (_get_parser, currently ticket/jira/itsm via insight/ticket_labels.py:parse_structured_labels) extracts fields that already exist in the source data, priority, category, service area, server, MTTR, with zero model reads. The LLM is reserved for dimensions that genuinely require reading narrative text (e.g. root cause, symptom, resolution). This follows the “reuse, not reinvention” note in the module’s own docstring: the schema still lives in Insight (extraction_schemas), and the LLM call still goes through call_llm.
• No ProcessingResult row. Unlike process_document, this does not write to processing_results (_suppress_result_persist), the label frame in the catalog is the durable output, not a second store.

max_rows (default 500) caps how many rows a single live call labels; a full-corpus pass is expected to run as a scheduled job rather than inline in a chat turn. The mechanism (source → schema → label frame → SQL) is general, though it was designed against a ticket-corpus use case.

Taxonomy and Schema Tools

Defined in tools/insight.py. All operate against taxonomies or extraction_schemas, all scope writes to the current tenant (defaulting to "*"), all return a small text message and optionally a schema_created card.

Tool	File:Line	Behavior
list_taxonomies	tools/insight.py:12	Lists taxonomies in scope (current tenant + "*"), flagging the default.
list_schemas	tools/insight.py:36	Lists schemas in scope, including their applies_to labels. Does not filter soft-deleted rows, see “Structural notes” below.
create_schema	tools/insight.py:60	Refuses duplicate names. Stamps created_by="assistant". Emits a schema_created card with the new id, name, fields, and applies_to.
update_schema	tools/insight.py:108	Supports add_fields, remove_fields, update_applies_to. Bumps version when fields change.
add_taxonomy_category	tools/insight.py:170	Appends to categories JSON; refuses duplicate labels.
create_taxonomy	tools/insight.py:210	Refuses duplicate names. is_default is an explicit input, there’s no atomic swap, so setting two taxonomies as default is possible until somebody enforces it.

REST Endpoints

insight/router.py mounts under /api/v1/insight. Every endpoint is gated by the auth @requires(...) decorator (see Shared Layer).

Method	Path	Capability	Notes
GET	/taxonomies	document.read	All in-scope taxonomies.
GET	/schemas	document.read	Soft-deleted schemas filtered out (deleted_at IS NULL).
GET	/history	document.read	Most recent ProcessingResult rows for the tenant, default limit 50.
GET	/stats	document.read	{documents_processed, avg_quality, taxonomy_count, schema_count, total_cost_usd}. Run through sanitize_for_json for NaN safety.
POST	/schemas	document.write	Refuses duplicate names. Does not set created_by (drift from the tool handler, which stamps "assistant").
PUT	/schemas/{id}	document.write	Same add/remove/update semantics as the update_schema tool. Bumps version when fields change.
DELETE	/schemas/{id}	document.write	Soft delete via deleted_at.
POST	/taxonomies/{id}/categories	document.write	Appends a category.
DELETE	/taxonomies/{id}/categories/{label}	document.write	Removes a category by label.

There is no REST endpoint to create or delete taxonomies wholesale today, taxonomy lifecycle goes through the tools (create_taxonomy) or seed data. Category-level edits are exposed; whole-taxonomy edits are not.

Prompt Context

insight/context.py returns a single line for the [AVAILABLE_DATA] block in the system prompt:

- Insight: 3 taxonomies, 7 extraction schemas available

It returns the empty string when both counts are zero so the prompt stays clean for fresh tenants. The model then uses list_taxonomies and list_schemas to discover specifics on demand.

LLM Usage

Every classification, extraction, and pipeline helper goes through the call_llm composable tool with role="fast". Costs roll up via the ToolResult.cost field returned by call_llm; process_document accumulates them across all child tool calls and writes the total to ProcessingResult.cost_usd. Standalone classify_document and extract_document calls report their own cost in ToolResult.cost and rely on the orchestrator’s usage_log write to record it.

Per the CLAUDE.md “no truncation” rule, document text is sent verbatim to the LLM, the model’s context window is the only constraint. There are commented-out truncation lines in the handler from a prior implementation; they are deliberately disabled.

Uniform Tool Contract

The platform finished a bulk migration onto a uniform Ok | Hint | Err contract; it is live in the running tree (tests/contract_deferred_baseline.json, the migration ratchet’s “still on the legacy path” snapshot, is an empty list as of 2026-09, see README.md). The insight tools participate as follows.

The eight insight tools sitting in tools/insight.py and the three pipeline tools in tools/knowledge.py are all spec-only migrated via migrate_tool(handler, name, *, is_tabular=False), handlers keep returning ToolResult and are wrapped at the dispatch boundary by @uniform_contract (see engine/contract_decorator.py). is_tabular=False is correct for this domain: these tools emit cards and counts, not row-set DataFrames, so the validator legitimately tolerates an empty Ok.schema. None of them is a cost rollup, so is_cost_rollup stays False; process_document’s child-tool costs already roll up through the existing ToolResult.cost summation and are persisted to processing_results.cost_usd independently of the usage_log write the decorator skips for delegate-class tools.

The contract surface for an insight tool call looks like:

• Success, Ok carrying the existing card payload on Ok.effects.cards, the rendered tool message on Ok.data, the same per-call cost on Ok.effects.cost, and a measured (or trivially empty) schema.
• Recoverable correction, handlers like extract_document that today return a ToolResult(content="Schema 123 not found.") migrate to raising HintException(hint_kind="schema", ...) so the model gets a typed Hint with a ranked “did you mean” list rather than a soft-success message it might mis-read. The legacy translator (_translate_dict_result / _translate_tool_result in engine/contract_decorator.py) handles the in-between case for handlers that haven’t been refactored yet, anything starting with "schema hint:" or "fetch hint:" becomes a Hint, other errors become Err(err_kind="legacy"). This means we can land the bulk migration without rewriting every handler’s failure paths in one pass.
• Hard failure, unhandled exceptions become Err(err_kind=type(e).__name__) automatically; the decorator catches HintException first to keep recoverable corrections out of the error path.

ProcessingResult persistence is unchanged by the contract, it happens inside the handler against the database directly, not through the typed Effects channel. The contract’s Effects surface (cards, datasets, exports, cost) is the orchestrator’s read-out; it doesn’t replace handler-side side-effects on domain tables.

Structural Notes

A handful of small inconsistencies are worth flagging for anyone touching this module:

• Where things live. Classification and extraction handlers live in tools/knowledge.py, not tools/insight.py. The split is historical; the schema files (engine/schemas/insight.py) still register them under category: "insight" so the model’s tool surface is consistent.
• Two paths, one schema. POST /api/v1/insight/schemas and _exec_create_schema both create rows but diverge in small ways: the tool stamps created_by="assistant" and emits a card; the REST handler does neither. Reconcile if you touch either.
• list_schemas ignores soft-deletes. The router filters deleted_at IS NULL; the _exec_list_schemas tool handler does not. After a schema is soft-deleted via DELETE /api/v1/insight/schemas/{id}, agents calling list_schemas will still see it.
• is_default is not exclusive. create_taxonomy accepts is_default=True without clearing the flag on whatever taxonomy currently holds it. The classifier falls back to Taxonomy.first() when no default is found; with multiple defaults, behavior depends on insertion order.
• Quality dimensions are LLM-scored. The 4-way weighted blend (completeness/consistency/conformity/confidence) is computed server-side, but the inputs come from the model’s own JSON response. There is no independent validation pass, if the model lies about its confidence, the rolled-up score lies too.
• Schema name resolution is fuzzy. extract_document resolves schema_name via ILIKE %name%. A user asking for "Lab Report" will match "EPA Lab Report v2" first if it exists. Intentional, but worth knowing.

Related Docs

• Knowledge Module, document ingestion, chunking, the source of document_id for the insight tools.
• Engine, agentic loop, call_llm dispatch, tool registry.
• Shared Layer, call_llm, extract_json_from_response, prompts catalog, auth decorator.
• Tools, ToolResult, registry, and the uniform contract surface.
• Glossary, terms: taxonomy, schema, processing result, contract.