Fabric Module

fabric/ is the data layer. It owns the unified catalog of every dataset, document, and export the platform knows about, plus the semantic structure (Kits, temperature, relationship edges, behavioral observations) layered on top. The catalog is the single source of truth that the LLM, the tool layer, and the Fabric UI all read from.

For how Fabric is consumed by individual tools, see Tools. For the orchestrator path that injects catalog context into the prompt, see Engine. For the storage primitives (ContentBlob, CatalogEntryRecord, RelationshipEdge, Kit, etc.), see Database Schema.

Module Files

File	Responsibility
fabric/catalog.py	CatalogEntry/CatalogSummary dataclasses, ConversationCatalog service, get_catalog()/get_tenant_catalog(), idle eviction, dual-write to Parquet + SQL
fabric/router.py	REST endpoints under /api/v1/fabric for datasets, documents, exports, Kits, datasources, dream-cycle trigger
fabric/temperature.py	compute_temperature(), compute_temperatures_bulk(), build_curated_context() for the [AVAILABLE_DATA] block
fabric/kit_service.py	Kit CRUD, append-only KitMembership, semantic-anchor embedding, vector + ILIKE search
fabric/kit_discovery.py	LLM-driven Kit proposal pipeline (dataset profiles, usage summaries, co-occurrence pairs, curator call, apply_proposals)
fabric/dream_cycle.py	Periodic deep analysis: edge build/persist, Kit discovery, pruning, temperature scan, cold cleanup, knowledge extraction
fabric/intelligence.py	Two-pass semantic analysis (generic + domain) of uploaded data; summarize_intelligence() for prompt rendering
fabric/semantic.py	SemanticExtractor: co-occurrence, lineage, schema affinity from telemetry + catalog state
fabric/knowledge_extraction.py	Telemetry-to-corpus pipeline: provisional (no-LLM) and full (LLM) extraction into the operational_patterns / data_catalog corpora
fabric/user_profile.py	Behavioral Observation CRUD, similarity-based reinforcement, [USER_CONTEXT] block builder, observation decay
fabric/snapshots.py	ExecutionSnapshot writer that freezes Kit/dataset resolution at workflow run time

The companion graph and introspection tool handlers live in tools/graph.py, tools/introspection.py, tools/discovery.py, and tools/catalog.py. The dream cycle invokes them by name rather than reimplementing the algorithms.

Design Principles

The Catalog is a Registry, Not a Store

CatalogEntry rows track metadata, provenance, and lifecycle. The actual bytes live elsewhere: tabular data in Parquet under catalog_data/{conversation_id}/{entry_id}.parquet, documents in chunks/documents (owned by the Knowledge module), exports as files on disk. Identity is a UUID; the human-readable name is what the LLM uses; resolution is name-or-id.

One Identity Model for Datasets, Documents, and Exports

CatalogEntry.data_type is "dataset" | "document" | "export". All three share the same row in catalog_entries, the same scope/retention/owner/tombstone rules, and the same eviction pipeline. Datasets carry columns, dtypes, parquet_path, geometry metadata, and an intelligence JSON column. Documents carry document_id, corpus_id, chunk/char counts. Exports carry export_path, export_mime_type, export_source_tool. See fabric/catalog.py:411.

Content-Addressable Storage

Every persistable payload (uploaded file, derived DataFrame, ingested document) routes through shared.content_index.ensure_blob(content_hash, ...) which INSERT … ON CONFLICT into content_blobs. The catalog row holds blob_id; ContentBlob.content_hash is the canonical SHA-256. This is what makes deduplication atomic across restarts and concurrent uploads. register_upload queries the catalog_entries table directly by blob_id and returns the existing entry, not the in-memory _entries list, which is parquet-directory-driven and routinely incomplete (fabric/catalog.py:919-967).

Scoping

Scope	conversation_id value	Lifetime
conversation	conversation UUID	Tied to a single conversation; cleaned when conversation goes cold
tenant	tenant:{tenant_id}	Persists across conversations within a tenant
global	reserved	Platform-wide reference data

Uploads, ingested documents, exports flagged for tenant retention, and PostGIS-backed catalog rows live at tenant:{id}. Derived analyze_data outputs live in the originating conversation. ConversationCatalog._resolve_unlocked walks the hierarchy in order: current conversation → tenant catalog → SQL fallback for tenant-shared rows that never entered any in-memory cache (fabric/catalog.py:1776-1900).

Soft Deletes

CatalogEntry.deleted_at and deleted_reason tombstone rows. Reads filter deleted_at IS NULL. delete() optionally cascades to entries that list the deleted name in parents. Reasons: user_action, conversation_cleanup, cold_cleanup, cascade, ttl_expiry. Hard delete only occurs in the dream cycle’s cold-conversation sweep, which removes non-durable entries plus their Parquet files (fabric/dream_cycle.py:223-229).

Ownership

owner_user_id is required on every save. catalog.save() raises if it’s missing or empty (fabric/catalog.py:642). Background jobs use system_owner_user_id() which returns the oldest user row in the users table (fabric/catalog.py:176-190). This is what lets the dream cycle write _dream_* scratch entries without a request context.

ConversationCatalog

One instance per conversation_id. The module-level registry _conversation_catalogs is keyed by conversation ID; tenant catalogs use "tenant:{id}". Every get_catalog() call refreshes a last-used timestamp and opportunistically evicts conversation-scoped instances that have been idle past conversation_idle_seconds (default 1800). Tenant catalogs are exempt from eviction by prefix check, they hold uploads and shared data that must outlive any single conversation. Eviction also tears down any DuckDB connections the conversation held (fabric/catalog.py:141-173).

__init__ calls _recover_from_disk(), which scans the Parquet storage directory and rebuilds CatalogEntry rows. PostGIS-backed entries (no parquet file) never enter the in-memory cache via this path; they’re hydrated lazily by the _resolve_unlocked SQL fallback.

Save Path

save(name, data, source, parents, retention, creation_context, entry_type, owner_user_id) is the canonical write:

1. Resets the index if it’s named/MultiIndex (groupby/pivot outputs put real keys there).
2. Detects GeoDataFrame, captures crs, geometry_type, total_bounds. Refuses the save if every geometry is null, a defensive guard added after a NC RCRA demo silently registered a shape-correct, content-empty frame (fabric/catalog.py:683-706).
3. Converts to Arrow via _to_arrow_with_object_recovery(), which catches ArrowInvalid("Conversion failed for column …") from mixed-type object columns (HIFLD sourcedate is the canonical case), coerces the offending column to string, and retries up to 32 times (fabric/catalog.py:24-73). Without this the save fails silently and the tool tells the model “saved as X” anyway.
4. For geo data, round-trips through pa.table(arrow_table) to verify the geometry column survived the geopandas→geoarrow encoding.
5. Runs shared.spatial_probe.detect_geometry_columns() and stores the result under intelligence["spatial"].
6. Computes SHA-256 over pd.util.hash_pandas_object(data). Falls back to a shape-and-column-name hash when the data contains unhashable types.
7. ensure_blob(content_hash, mime_type="application/x-parquet", storage_backend="parquet") resolves blob_id.
8. Builds the CatalogEntry, appends to self._entries, then dual-writes: _persist_async() launches a thread that writes Parquet, _persist_to_db() synchronously inserts/updates catalog_entries.
9. Builds a templated searchable_summary via _build_searchable_summary() (no LLM) that leads with single-state-scope when present, then domain, then column-role hints and the top pitfall alert. This same string is what the dream cycle chunks into the data_catalog corpus and embeds, so the keyword and embedding paths see the same vocabulary (fabric/catalog.py:273-359).

Upload Path

register_upload(df, filename, tenant_id, owner_user_id, content_hash, file_path) is the upload-specific wrapper. Dedup is content-addressable and DB-authoritative: it computes content_hash via hash_dataframe(), resolves blob_id, then looks up any existing non-tombstoned CatalogEntryRecord for this tenant referencing the same blob (joining through Conversation for tenant matching, also accepting conversation_id == "tenant:{id}"). On hit it returns the in-memory entry if cached, hydrates from SQL otherwise, returns is_duplicate=True. On miss it runs DataProfiler.profile(), calls save() with EntryType.UPLOAD, then runs the two-pass analyze_dataset intelligence pipeline, caching by blob_id first and identical column schema second (fabric/catalog.py:892-1059).

Resolution

resolve(name) / _resolve_unlocked(name) order:

1. Current conversation’s _entries, reverse order (most-recent wins).
2. Tenant catalog _entries.
3. SQL fallback restricted to conversation_id == "tenant:{current_tenant}". This deliberately excludes other conversations’ scratchpads so that ambient df/results names from prior chats can’t shadow a real tenant upload. Earlier versions joined through tenant_id, which leaked stale conversation-scoped names; the current code is the fix (fabric/catalog.py:1816-1873).
4. Documents get a narrower tenant-global SQL lookup.

get_entry_by_id(entry_id) does the same walk by UUID and finally falls back to resolve(entry_id) so callers can pass either form.

Load Path

load(name) returns a pd.DataFrame or gpd.GeoDataFrame. If arrow_table is None and intelligence.postgis_table is set, it routes to _load_from_postgis(), which counts the table first and refuses to materialize anything over POSTGIS_MATERIALIZE_CAP = 10_000 rows, raising a KeyError that tells the model to use spatial tools or a filtered query() instead. This is the right primitive: large PostGIS tables are queried with predicates pushed down, not yanked into pandas (fabric/catalog.py:1339-1442).

Default Selection

default_dataset(exclude_internal=True) is the last-resort fallback for tools that received no explicit selector. It returns the most-recently-created dataset whose name does not start with _ (the convention for dream-cycle/introspection scratch). A caller that received an explicit selector MUST resolve that first and fail honestly if it doesn’t match, the docstring is explicit about this because export_excel once shipped a 867-row _dream_catalog dump under a user-named alias before this guard existed (fabric/catalog.py:1472-1498).

Context Block

get_context_block() builds a per-conversation [AVAILABLE_DATA] block. The temperature-aware variant is in temperature.build_curated_context() (below) and is what the orchestrator actually injects. The per-catalog version is retained for tools that need a structured snapshot of one conversation’s view.

Namespace Diff (analyze_data integration)

snapshot_namespace(ns) and diff_namespace(pre, ns) are the static helpers that drive analyze_data → catalog promotion. Pre-snapshot is taken before user code runs; diff returns the new/modified DataFrames. _promote_to_geodataframe upgrades plain frames with a geometry column (the pd.merge(tabular, gdf) case where pandas downcasts to DataFrame) so spatial metadata survives. Names starting with _ and the standard df/pd/np slots are filtered. The analyze handler walks the diff and calls catalog.save(name, df, source="analyze_data", parents=[…]) for each promotion (fabric/catalog.py:1722-1760).

Dual-Write Architecture

Every dataset save touches two stores:

Plane	Write	Role
SQL (catalog_entries)	Synchronous via _persist_to_db()	Authoritative index for the Fabric page, cross-conversation queries, dedup, tombstones
Parquet (catalog_data/{conversation_id}/{entry_id}.parquet)	Async via _persist_async() thread	Durable byte storage for recovery across restarts

The SQL row carries parquet_path for recovery. If Parquet files vanish, entries become functionally tombstoned (metadata survives, data does not). If the SQL row vanishes, the parquet file is unindexed and _recover_from_disk() re-creates an entry on next instantiation. The two planes are intentionally separate.

PostGIS-backed datasets (HIFLD archive tables, EJScreen, NHGIS) have parquet_path = None and intelligence["postgis_table"] set; they’re loaded on demand via _load_from_postgis() rather than materialized to Parquet.

Catalog Entry Types

shared.entry_types.EntryType defines six types (FK to catalog_entry_types). DURABLE_TYPES is the subset the dream cycle, Kit discovery, and Fabric overview filter on, they’re what counts as “real” data versus scratch. The default for save() is DERIVED (4); register_upload overrides to UPLOAD; register_export overrides to EXPORT; register_document overrides to DOCUMENT. Intermediate analyze_data outputs stay DERIVED and are excluded from durable surfaces.

Temperature and Curated Context

fabric/temperature.py is what feeds the LLM’s view of “what data exists.” Temperature is f(recency, frequency, breadth) ranging 0-100. The defaults in TemperatureConfig (fabric/temperature.py:21-33):

Component	Weight	Computation
Recency	0.5	100 * exp(-ln(2)/14d * days_since_last_active) (14-day half-life)
Frequency	0.3	min(100, 25 * log2(1 + reference_count))
Breadth	0.2	min(100, 33 * distinct_conversations)

Thresholds: hot ≥ 70, warm ≥ 30, cold ≥ 5, frozen < 5.

Bulk Computation

compute_temperatures_bulk(tenant_id) queries dataset_activity-adjacent telemetry: it loads tenant-scoped tool_call and tool_result events over the trailing 90 days and counts references by structured fields (input.dataset_id, input.dataset, input.left_dataset, input.right_dataset, tool_result.datasets keys), not substring matching, which inflates counts for short names. Entries that share a content_blob via shared.content_index.group_by_content have their stats merged so the same dataset uploaded in three conversations counts as one (fabric/temperature.py:76-221).

Curated [AVAILABLE_DATA]

build_curated_context(conversation_id, tenant_id, user_id, token_budget=4000, active_dataset_ids) is what the orchestrator injects. It pulls all non-tombstoned tenant datasets across every conversation, deduplicates by canonical_id then by name (current-conversation entries shadowing prior-conversation entries with the same name), sorts by temperature, and renders:

Treatment	Trigger
Full detail (name, id, row × col, source tag, column lists by string/numeric split, intelligence summary, creation purpose, latest contextual signal)	is_active (dataset attached to current message) or hot ≥ 70
One-liner (name, rows, source tag, (prior session) tag if cross-conversation, optional purpose suffix)	warm ≥ 30
Omitted	cold/frozen

Caps: max_hot_datasets=15, max_warm_datasets=10, token budget 4000. The renderer adds an Access section that names analyze_data, search_documents, train_model, spatial tools, export, and create_kit. When the current conversation has no uploads but prior-session datasets are surfaced, an extra hint tells the model to load by name via get_dataset('name') rather than assume df is pre-loaded, a fix for the “I don’t have access to that data” failure mode (fabric/temperature.py:439-453).

Kits

Kits are semantic groupings of datasets. They are the only organizational primitive, there are no folders. See Glossary.

KitService (fabric/kit_service.py)

Operation	Notes
create_kit(tenant_id, name, description, creator, pinned, template)	Generates a semantic_anchor embedding from f"{name}: {description}" via the orchestrator’s KnowledgeService.embed_query
add_to_kit(kit_id, dataset_ids, added_by, reasons)	Append-only. Resurrects soft-deleted memberships rather than duplicating. Records per-dataset reason.
remove_from_kit(kit_id, dataset_ids)	Soft delete on KitMembership.deleted_at. The membership row is preserved.
list_kits(tenant_id, user_id, include_system)	Returns Kits with member counts; ordered by last_modified_at
get_kit(kit_id)	Kit + get_kit_members() resolved to CatalogEntryRecord joins with source-type classification (upload/export/derived)
search_kits(tenant_id, query)	pgvector cosine distance on semantic_anchor when available; ILIKE fallback on name/description
auto_assign_to_kits(dataset_id, tenant_id)	Walks RelationshipEdge with weight >= 3.0, finds Kits containing related datasets, adds the new dataset to each
delete_kit(kit_id)	Soft delete on Kit + cascade soft delete on memberships

backfill_kit_embeddings() is the one-shot job that fills semantic_anchor for Kits created before the embedding column was wired.

Membership is append-only by design. Temperature drives display priority, not membership. A cold dataset stays in its Kit; the Fabric UI just deprioritizes it.

LLM-Driven Kit Discovery (fabric/kit_discovery.py)

Replaces connected-component clustering. The pipeline:

1. build_dataset_profiles(tenant_id), durable-type entries with columns, intelligence summary, source.
2. build_usage_summaries(tenant_id, dataset_names), per-dataset conversation count, top co-workers (other datasets seen together), top analytical tools, top meaningful conversation titles. Derived from DatasetActivity + tool_call events.
3. build_co_occurrence_pairs(tenant_id, dataset_names), top 25 pair counts.
4. _format_existing_kits(tenant_id), current system Kits and their members.
5. call_kit_curator(...), fills KIT_DISCOVERY_PROMPT, calls the kit_curator model role (fallback chain: kit_curator → strong → fast), parses JSON, validates every named dataset against the real inventory, records cost via record_llm_cost. Returns KitProposal objects.
6. apply_proposals(proposals, tenant_id), resolves names to entry IDs (most recent per name), looks up existing system Kits, and either merges into an existing Kit if the proposal has >50% overlap with its membership or creates a new Kit. Adds members append-only, recording the proposal description as reason.

The curator picks Kit organizations across 7 facets (regulatory program, geography, entity type, analytical task, data form, source system, output type). The prompt and facet system live in shared/prompts.py::KIT_DISCOVERY_PROMPT.

Dream Cycle (fabric/dream_cycle.py)

Periodic deep analysis. The only mechanism that reshapes, prunes, or creates system Kits. Real-time updates are additive only. Two ways it runs: on demand via POST /api/v1/fabric/dream-cycle (admin-gated; runs in asyncio.to_thread so it doesn’t block the event loop), or on schedule via scripts/background_jobs.py.

Scheduling of that script is now app-owned, not manual-only. app.py’s startup lifespan schedules shared/embedding_health.py:embedding_health_loop() as a background asyncio task (boot heal, then every 6 hours per its docstring). That loop’s job is to heal embedding backlogs directly, but it also calls spawn_background_jobs() (shared/embedding_health.py:243-287), which subprocess.Popens scripts/background_jobs.py as a detached child process (gated by settings.background_jobs_spawn_enabled), background_jobs.py throttles its own dream/probe cadence internally (6h dream, 4h probe), so the spawn call itself can run more often without over-triggering. Manual invocation (--loop, --dream-only) and the Admin > System on-demand trigger remain available; they are no longer the only path to a dream cycle running.

DreamCycle.run(tenant_id) chains:

Step	Action
1	_exec_load_catalog_metadata → _dream_catalog scratch DataFrame
2	_build_co_occurrence_from_activity (queries DatasetActivity joined to durable CatalogEntryRecord) → _dream_co_occurrence; _exec_build_lineage → _dream_lineage
3	_persist_tool_edges resolves names to IDs and upserts RelationshipEdge rows (co_occurrence, lineage)
4	discover_kits(tenant_id), LLM Kit proposals applied via apply_proposals
5	_prune_stale_edges removes edges below stale_edge_weight_threshold
6	compute_temperatures_bulk to count frozen datasets (≤ cold threshold)
7	_cleanup_empty_kits soft-deletes Kits with zero live members
8	_detect_cross_user_affinity records same-blob usage across users
9	fabric.knowledge_extraction.extract_data_catalog(tenant_id) populates the per-tenant data_catalog corpus
10	_cleanup_cold_conversations hard-deletes non-durable entries plus their Parquet files for conversations the temperature scan classified as cold

The graph-building steps invoke tools/graph.py and tools/introspection.py handlers (_exec_build_lineage, _exec_discover_clusters, _exec_load_catalog_metadata) with a tool context populated by system_owner_user_id(). The handlers do the generic work; the dream cycle does the dream-cycle-specific persistence (RelationshipEdge upsert, Kit naming/curation).

A fallback _discover_clusters runs connected-component analysis directly against RelationshipEdge when the tool-produced edge DataFrames aren’t available, historically the LLM curator is the primary path, and the connected-component code is the safety net for cold-start tenants.

Behavioral Observations and User Profile (fabric/user_profile.py)

Atomic, per-user factual statements about how a person works. Stored in the observations table. Seven categories: domain, workflow, data_preference, correction, temporal, friction, policy_violation.

Function	Notes
create_observation(user_id, tenant_id, category, content, failure_mode, outcome_type, source_events, confidence)	Checks for similar existing observations (≥60% word overlap) in the same category; reinforces by evidence_count += 1 and confidence = min(1.0, confidence + 0.1), otherwise inserts
get_user_observations(user_id, tenant_id, active_only)	Ordered by confidence desc
build_user_context_block(user_id, tenant_id)	Renders [USER_CONTEXT] (target 50-100 tokens), top 2 per category, priority order correction > policy_violation > data_preference > domain > workflow > friction > temporal, capped at 12 lines
decay_observations(tenant_id, decay_days=90)	Marks observations active=False if last_reinforced is older than the window

The introspection tool _exec_create_observation is what the LLM uses to write observations explicitly (tools/introspection.py:333). _exec_load_observations materializes the table as a DataFrame so the LLM can analyze its own profile data with the same tools used for environmental data.

Observation extraction at conversation end runs through fabric/knowledge_extraction.py::write_provisional_entry (no LLM) and the queue worker’s full extraction (LLM-powered). Failure modes and outcome types are part of the Observation row; correction/policy_violation observations carry a structured failure_mode. Policy rules and routing rules live in their own tables (policy_rules, routing_rules), observations are the raw signal those tables are derived from.

Semantic Extraction (fabric/semantic.py)

SemanticExtractor builds three edge types into RelationshipEdge:

Type	Source
co_occurrence	Datasets appearing in the same conversation. Uses canonical_id (blob if present, name fallback) so the same upload across conversations counts once. Bridges tenant-scoped uploads into conversations that referenced them by name in parents.
lineage	CatalogEntry.parents → derived child.
schema_affinity	Column-name overlap between datasets.

The dream cycle’s _persist_tool_edges is what actually writes these into the DB. SemanticExtractor is the standalone implementation; the dream cycle invokes the equivalent via the tools/graph.py handlers and persists the result.

Knowledge Extraction (fabric/knowledge_extraction.py)

Telemetry-to-corpus pipeline. Two tiers:

Tier	When	LLM
Provisional	Conversation end	No
Full	Queue worker	Yes

Provisional extraction walks ConversationEvent rows, deduplicates the tool chain (tool_a → tool_b → tool_a becomes tool_a → tool_b → tool_a), detects outcome from correction / friction / message_rating events, and writes a [PROVISIONAL] chunk into operational_patterns. Full extraction (queue-driven) replaces the provisional chunk with a richer LLM-extracted version. extract_data_catalog(tenant_id) is the dream-cycle entry point that re-embeds every datum’s searchable_summary into the data_catalog corpus, the same vector index the catalog resolvers and search_catalog consume.

Execution Snapshots (fabric/snapshots.py)

create_snapshot(workflow_run_id, dataset_ids, kit_id, resolution_context) freezes resolved dataset identities at workflow execution time. For each dataset it captures dataset_id, name, content_hash (via shared.content_index.hash_for_blob(blob_id)), blob_id, row_count, columns. This is the only point where Kit membership becomes a persisted, immutable object, a workflow run can always reference the exact dataset set it executed against, regardless of how the Kit evolved afterward.

REST Endpoints (fabric/router.py)

Prefix /api/v1/fabric. All require the analyze capability except the dream-cycle trigger.

Method	Path	Purpose
GET	/datasets	List tenant uploads grouped by (name, content), with copies and all_ids
GET	/datasets/{id}	Full metadata; resolves through conversation catalog first if conversation_id provided, then tenant catalog
GET	/datasets/{id}/preview?n_rows=10	Head rows, JSON-safe
GET	/datasets/{id}/data?max_rows=500000&columns=...	Full dataset as records for client-side charting
DELETE	/datasets/{id}	Tombstone
GET	/catalogs	List per-conversation catalog summaries
GET	/catalogs/{conversation_id}/datasets/{name}/dependents	Names of derived datasets pointing at this entry
DELETE	/catalogs/{conversation_id}/datasets/{name}	Tombstone with optional cascade
GET	/documents	Tenant documents joined with corpus names
DELETE	/documents/{document_id}	Removes the Document and ChunkRecord rows; tombstones any catalog entries referencing it
GET	/exports	Tenant exports
GET	/entries/{entry_id}	Generic entry lookup by UUID across types
GET	/embedding-status/{item_type}/{item_id}	Embedding presence check
GET	/kits	List Kits
POST	/kits	Create Kit (optionally with initial dataset_ids)
GET	/kits/{id}	Kit + members
POST	/kits/{id}/members	Add datasets
DELETE	/kits/{id}/members/{dataset_id}	Remove (soft)
DELETE	/kits/{id}	Delete Kit (soft)
POST	/kits/search	Vector-or-ILIKE search
GET	/overview	Kits + temperature-sorted datasets (deduped by content + name) for the Fabric page
GET	/datasources	External datasources visible to this tenant (tenant_id IN (current, "*"))
GET	/datasources/{id}	Full datasource detail, auth_config secrets masked
POST	/dream-cycle	Admin-only on-demand trigger; runs in asyncio.to_thread

The document delete endpoint hard-deletes the Document and ChunkRecord rows but tombstones the catalog entry. That diverges from the Soft Deletes rule elsewhere and was a deliberate trade for chunk-storage hygiene. The catalog row remains as a tombstone; only the underlying Knowledge-module rows go.

Tools That Read Fabric

The LLM-facing tool handlers that consume the catalog live in tools/:

File	Handlers
tools/catalog.py	list_catalog, describe_entry (with lenient fuzzy/embedding fallback and datasource-name normalization), create_kit, list_kits, add_to_kit, search_kits, query_telemetry, get_dataset_relationships, get_dataset_usage
tools/discovery.py	search_catalog (three-tier: current conversation → tenant short-term → data_catalog corpus, vector-first with keyword fallback), search_tools, discover_tools
tools/graph.py	build_co_occurrence, build_lineage, build_schema_affinity, discover_clusters, generic graph analytics, no hardcoded tables, no side effects
tools/introspection.py	load_telemetry, load_relationships, load_catalog_metadata, load_observations, create_observation

list_catalog also surfaces in-memory DataFrames and shapely geometries from tools/analysis._conversation_namespaces (the analyze_data namespace) so the LLM sees ephemeral working variables alongside persisted entries (tools/catalog.py:50-100). This is what stops the model from re-fetching data it already has in scope.

search_catalog enforces a data-isolation rule: when _advanced.data_isolation is True (default), prior-conversation matches are hidden from the result list but a hint about their existence is surfaced so the model can advise the user to toggle the setting. When False (research mode), prior matches render inline with [from prior conversation: <title>] annotations (tools/discovery.py:75-160).

Uniform Tool Contract

The catalog and Kit tool handlers are part of the platform-wide Uniform Tool Contract migration. The contract promotes every tool’s return value to one of three named result variants, Ok | Hint | Err, routed through a single @uniform_contract decorator and rendered into one model-facing format at every tool boundary. See Tools and Engine for the full contract.

Concretely for Fabric:

• describe_entry is source-decorated with @uniform_contract and uses the resolver path. When the entry name doesn’t match, the handler raises HintException with hint_kind="schema" and ranked alternatives, which the decorator translates to a Hint(hint_kind="schema", alternatives=[…]) envelope. That’s what makes “did you mean X” a first-class recoverable signal rather than an error string that looks like a platform bug.
• Catalog and Kit handlers (list_catalog, create_kit, list_kits, add_to_kit, search_kits, plus several others) are spec-only migrated via migrate_tool(handler, name, is_tabular=…). The handler source still returns ToolResult; the decorator wraps at registration time. is_tabular=True flags empty-schema Ok envelopes as a smell so synthetic-schema silent failures get caught.
• Resolution honors the Knowledge-Is-Routing rule. Fuzzy ranking inside engine/contract_resolvers.py::resolve_catalog_entry queries the data_catalog system corpus via the same vector_searcher path that search_catalog uses, falling back to local-embedder name cosine for namespace variables. Resolvers and search_catalog cannot disagree about what the catalog contains.
• Effects (catalog registrations, exports, cost) ride on Ok.effects. Hint and Err carry no effects, so a save that fails honestly cannot leave a half-registered row behind.
• The CI ratchet (tests/test_uniform_contract.py + tests/contract_deferred_baseline.json) computes the unmigrated set from the registry by absence of the _uniform_contract marker and can only shrink. As of 2026-09 that baseline file is an empty list, every tool in SYSTEM_TOOLS now registers a ToolSpec one way or another (see README.md).

For Fabric specifically, the migration removes a long-standing failure mode: when save() silently no-op’d (Arrow conversion errors, null geometry, content-hash collision against a tombstoned row), the tool’s ToolResult.content said “saved as X” and the model trusted it. Under the contract, the integrity guards now raise, and once analyze_data migrates off the deferred list, the save-then-claim-success path will be one continuous, validated envelope from data write to model-facing render.

Common Gotchas

• Owner is required. catalog.save(..., owner_user_id="") raises. Tool handlers should pass context["user_id"]; background jobs should pass system_owner_user_id().
• _recover_from_disk() is parquet-directory-driven. PostGIS-backed entries (no parquet file) never enter the in-memory cache through recovery. Resolution falls through to the SQL fallback in _resolve_unlocked. If you bypass resolve() and read _entries directly, you will miss them.
• Membership is append-only. Don’t try to “fix” Kit membership by removing cold datasets. Temperature drives display priority; membership stays.
• Don’t trust arrow_table for geometry detection. Use entry.has_geometry only as a hint; the to_dataframe() method checks for to_pandas capability because the geopandas ArrowTable wrapper lacks it (fabric/catalog.py:524-539).
• default_dataset() is last-resort. If the LLM passed an explicit selector and it doesn’t resolve, fail honestly. Falling back to default_dataset() is how export_excel once shipped a dream-cycle scratch frame under a user’s name.
• Tenant catalogs survive idle eviction. Conversation catalogs don’t. If you hold a reference past the 1800s idle window, you’re holding an evicted shell.
• Search and resolution must share a corpus. New retrieval surfaces should query the data_catalog corpus via vector_searcher, not invent their own ranking. Embeddings already exist for tools, primitives, behavioral observations, and the catalog. If a surface seems to need a new ranking path, the answer is usually that it’s not calling _pre_fetch_context or its equivalent.