External Data

Eunice talks to external data through declarative YAML datasource definitions (EPA, NOAA, FEMA, USGS, Census, HUD, plus a handful of internal PostGIS-backed reference tables), 85 files directly under data/api_definitions/ plus 7 more under data/api_definitions/env_layers/, totaling roughly 293 declared endpoints as of 2026-09 (grep -rc endpoint_id: data/api_definitions/*.yaml is the way to recount; don’t trust a number here). Every datasource is registered in the datasources table and called through a single ConnectorService chokepoint. This page describes how to add a new datasource, how the YAML maps to runtime behaviour, and how the model finds and invokes endpoints.

Related: Tool system, Building tools, Architecture, Glossary.

The shape of a datasource

A datasource is one upstream API (or one PostGIS-backed local table) with one or more endpoints under it. The YAML file is authoritative for structure (base URL, auth method, endpoints, parameters, normalization, freshness, rate limits). The database row tracks runtime state, credentials added through set_datasource_auth, health (status, consecutive_failures, last_error, last_success_at, last_health_check), and the linked documentation corpus.

YAML files live in data/api_definitions/. Sub-folder data/api_definitions/env_layers/ holds environmental layer overlays (wind, ocean currents, NEXRAD) used by spatial tools. The seeder (scripts/seed_api_definitions.py) loads *.yaml in sorted order, applies a static lint, and upserts rows; the health probe (scripts/datasource_probe.py) keeps status honest at runtime.

Two transports are supported:

Transport	Use it for	Runtime path
http (default)	Federal/public APIs reached over HTTPS	ConnectorService._build_url → httpx
postgis	Reference tables loaded into the local PostgreSQL/PostGIS DB (HIFLD archive, EJScreen, TRI/RCRA snapshots, chemicals reference)	ConnectorService._query_postgis

PostGIS-transport YAMLs are conventionally named with a _local suffix (e.g. hifld_local.yaml, echo_facilities_local.yaml). They share the same endpoint/parameter shape as HTTP datasources, only the resolution backend differs.

YAML schema

The keys the seeder and connector recognise. Anything else is ignored.

Top-level

Field	Required	Meaning
id	yes	Stable string ID. By convention datasource-<slug>. Used as the primary key in datasources.
name	yes	Display name. Surfaced to the LLM by list_datasources and describe_entry.
tenant_id	no	Defaults to epa-default. Set * for system-wide.
description	yes	Free text. Embedded into the documentation corpus and shown to the LLM.
base_url	yes	Scheme + host + base path. Per-endpoint base_url can override (see CO-OPS, metadata and data live on different paths).
transport	no	http (default) or postgis.
auth_method	yes	none, api_key_query, or api_key_header.
auth_config	no	Auth fields, see below.
rate_limit	no	{requests_per_period: N, period_seconds: M}. Enforced in-process by ConnectorService._check_rate_limit.
freshness_policy	no	{default_ttl_seconds: N}. Drives freshness_expires_at on cached materialised snapshots.
authority_domains	no	List of {domain, authority} records, canonical/derived/etc. Used by ranking and list_datasources.
endpoints	yes	List of endpoint definitions.
documentation_corpus_id	no	String ID of the linked knowledge corpus. Written by scripts/seed_api_docs.py; do not hand-edit.
status	no	Operator override. Honoured by the seeder only for offline, any other value yields control to the probe’s runtime state.

auth_config fields (varies by auth_method)

Field	Used by	Meaning
key_env	both	Env-var name holding the secret. Always takes precedence over a DB-stored api_key.
key_param / param_name	api_key_query	Query-string parameter name.
key_header / header_name	api_key_header	HTTP header name.
key_prefix	api_key_header	Optional prefix (e.g. "Bearer ").
additional_params	api_key_query	List of {name, value} or {name, value_env}. AQS needs both email and key; this is how the second one gets attached.
api_key	seeded by set_datasource_auth	Persisted runtime secret. The seeder preserves it across re-runs (_merge_auth_config); placeholder strings (DUMMY, PLACEHOLDER, TODO, CHANGEME, etc.) are ignored at request time (ConnectorService._PLACEHOLDER_KEYS).

The api.data.gov-fronted services (CAMPD, Regulations.gov, Federal Register) share one key via CATALYST_REGULATIONS_GOV_API_KEY. One signup at https://api.data.gov/signup/ unlocks all of them.

Endpoint fields

Field	Meaning
endpoint_id	Stable string ID, unique within the datasource.
path	Appended to base_url. {name} placeholders are substituted from runtime params (see “URL construction”).
base_url	Per-endpoint override of the datasource’s base_url.
method	GET (default) or POST. Long URLs auto-upgrade GET→POST around 6.5 KB.
description	Free text. Embedded into the documentation corpus, surfaced by list_datasources.
parameters	List of parameter definitions (see below).
default_params	Endpoint-level dict, applied to every request, overridable by caller.
requires_filter	List of parameter names; at least one must be present at call time. Enforced by tools/external.py:_exec_query_datasource.
response_format	json (default), csv, etc. CSV triggers pd.read_csv on the body. Path-style APIs (Envirofacts) append it to the URL.
response_path	Dot-path to walk into the JSON before normalisation (e.g. value.timeSeries for NWIS).
response_chain	Two-step query config, see below.
response_schema	Documentation-only; rendered in the API doc corpus.
normalization	Strategy block, see “Normalization”.
known_issues	List of strings. Surfaced verbatim back to the LLM with every successful query.
probe_params	Minimal params the health probe should send. Empty {} is valid (parameter-less listing endpoints); the key MUST be present.
test_params	Legacy alias for probe_params. Either is accepted, neither raises the seed lint.
example_queries	List of {name, params} realistic scenarios. Rendered in the doc corpus.
skip_probe	true to opt out of the health probe, used for POST-with-JSON-body endpoints and other shapes the probe can’t replay.
path_style	path (default, Envirofacts-style /key/value/) or operator (Envirofacts views, /key/equals/value/).
on_zero_features_pivot	List of alternative datasource names to suggest when a query returns 0 features. Surfaced inline in the error.
table_name	Required for transport: postgis endpoints. Names the PostgreSQL table.
column	On a postgis parameter, maps the parameter name to a real column. Only declared columns can be filtered.

Parameter fields

Field	Meaning
name	Parameter name as the upstream API expects it.
type	string, number, array, path. path means it appears as a URL segment, not a query string. array with a comma-separated string is split into repeated query params (?k=a&k=b), required by CAMPD streaming and similar.
required	If true, the call is rejected with a precise missing-parameter error before hitting the upstream.
default	Per-parameter default. Applied below default_params (endpoint-level wins).
description	Free text. Rendered in the doc corpus and surfaced when the call is rejected for missing parameters.
column	PostGIS-only. Maps the parameter to a table column for WHERE clauses.

URL construction (HTTP transport)

ConnectorService._build_url (shared/connector.py) merges parameters in priority order: per-parameter default < endpoint default_params < caller params. Then:

1. Path templates. Any {name} in path is substituted from merged params; those names are flagged as consumed.
2. Path-style parameters. Parameters with type: path that weren’t consumed by a template get appended.
   • path_style: path (default), /<key>/<value>/.
   • path_style: operator, /<key>/equals/<value>/. Required for Envirofacts joinable views.
   • The reserved name rows paginates: /rows/N (path style) or /0:N (operator style).
3. Query parameters. Everything else goes into the query string. array-typed parameters with comma-separated string values are split into a list so httpx emits repeated params.
4. Response format suffix. Path-style Envirofacts endpoints get /JSON (or whatever response_format says) appended.
5. ArcGIS defaults. When /arcgis/rest/ appears in the URL, the connector injects f=json, where=1=1, outFields=*, and resultRecordCount=1000 if missing. This protects against ArcGIS’s habit of returning an HTML help page when called with no params.
6. GET → POST upgrade. If len(url) + encoded_qs > 6500 the request is sent as application/x-www-form-urlencoded POST. ArcGIS and Envirofacts accept either; this keeps large geometry payloads alive.

Calling a datasource

The only entry point for tools is ConnectorService.query(datasource: dict, endpoint_id, params, skip_cache=False, max_retries=3) (shared/connector.py:query). It returns a ConnectorResponse(data, status_code, cache_hit, datasource_id, endpoint_id, query_hash, latency_ms, byte_count).

The model reaches it via the query_datasource tool (tools/external.py:_exec_query_datasource), which adds:

• Lenient name resolution. When datasource_name doesn’t match exactly, it strips common suffixes and tries ILIKE before failing. Final fallback is search_catalog (see “Discovery”).
• Endpoint validation. Unknown endpoint_id → error listing the available ones.
• Unfiltered-call guard. If the endpoint has any parameters and none were supplied, the call is rejected with a worked example (built from probe_params/test_params) and the full parameter list. If the LLM put query intent in the reasoning field instead, it’s reflected back.
• requires_filter enforcement. At least one named filter must be present.
• Required-parameter enforcement. Per-parameter required: true is checked before the network call.
• Offline guard. Datasources marked offline short-circuit with a pivot hint (often pointing at the local PostGIS replacement).
• ECHO QID responseset cap. Server-side pagination uses the initial call’s value; the caller can’t raise it past response_chain.follow_params.responseset once the cursor is set.
• ArcGIS bbox tiling. Bboxes larger than 0.1° per side are split into tiles, queried in parallel-ish, and merged with OBJECTID dedup (_query_with_arcgis_tiling).
• ArcGIS truncation detection. When the response sets exceededTransferLimit or returns exactly resultRecordCount rows, a fetch hint is appended telling the LLM to page with resultOffset.
• ArcGIS sibling-layer probing. On 0 features, sibling layers under the same FeatureServer are probed with returnCountOnly=true. Non-zero matches are surfaced inline so the model can pivot endpoint without trial-and-error.
• on_zero_features_pivot. Empty result + YAML pivot list → alternative datasources suggested in the same response.
• Catalog registration. Successful responses are normalised to a DataFrame and saved into the ConversationCatalog as a materialised entry, with source_datasource_id + canonicalised query_parameters recorded for cache lookup. Large GeoDataFrames (>500 features or >1 MB of WKB) are also written to PostGIS as ext_<name> with a GIST index so spatial tools can push down filters.

Response chains (two-step queries)

ECHO and a few other EPA programs return a query ID first, then require a follow-up. Express it on the endpoint:

response_chain:
  id_field: "Results.QueryID"          # dot-path into the first response
  follow_path: "/rcra_rest_services.get_qid"
  id_param: qid
  follow_params:
    output: JSON
    responseset: "5000"
  result_key: "Results.Facilities"     # dot-path into the follow-up

The connector follows the chain inside the same query call, returns the follow-up payload, and times out cleanly with a “narrow your filters” message if step 2 stalls.

Normalization

Every successful response is run through tools/normalizers/normalize_response(data, endpoint_config) (tools/normalizers/__init__.py). Strategy is named in the endpoint’s normalization.strategy; if absent the dispatcher auto-detects:

Strategy	Best for	Key config
flat	List-of-records APIs	expected_columns for shape validation
records	Nested JSON unwound via pd.json_normalize	record_path, meta (supports integer indices that pd.json_normalize rejects)
jmespath	Anything requiring real reshaping	expression (requires jmespath package)
geojson	FeatureCollections / GeoJSON features → GeoDataFrame in EPSG:4326	,
function	Truly unusual shapes	callable: "tools.normalizers.osm.parse_overpass" (dotted path)
header_row	Census ACS, EIA, BLS, [[hdr], [row], [row], ...]	,
scalar	Bare int/float/bool/str (e.g. USGS /count)	column_name (default value)

Auto-detection covers GeoJSON FeatureCollections, ArcGIS {features: [{attributes: {...}}]} (flattened into top-level columns), and common container keys (results, data, items, rows). Empty flat responses raise EmptyResponseError, a distinct exception that’s framed back to the LLM as a data-availability issue, not a parser bug, so it doesn’t pivot away from a working endpoint. The post-strategy column-transform hook (negate, abs, lower, upper, strip) handles upstream quirks (e.g. NWIS depth-below-surface returned as a negative number).

Geometry columns coming back as GeoJSON strings (the PostGIS path emits these) are hydrated into shapely objects by shared.spatial_backend.hydrate_geometry_column so the result is a true GeoDataFrame ready for spatial tools.

Function-strategy normalizers live alongside the dispatcher: tools/normalizers/osm.py (Overpass), tools/normalizers/usgs_nwis.py (NWIS time-series).

PostGIS transport

ConnectorService._query_postgis reads the endpoint’s table_name, builds a parameterised SQL WHERE clause from the parameters whose column field is set, and executes through SessionLocal(). No caller-supplied SQL, only declared filters. The geometry parameter accepts lon_min,lat_min,lon_max,lat_max and runs through _validate_bbox, which catches degenerate extents, out-of-WGS84 ranges, and the common lat/lon swap with a corrected-value suggestion.

When the target table has a geometry column, the query emits ST_AsGeoJSON(geometry)::json AS _geojson alongside SELECT *; the result row carries the GeoJSON as a JSON string (PyArrow can’t unify mixed geometry types as parsed dicts). When the table has no geometry column (e.g. the chemicals reference table), introspection skips the _geojson column entirely, a regression caught after the chemical-hazard rollout crashed every query with UndefinedColumn.

Authentication resolution chain

_resolve_api_key (shared/connector.py:_resolve_api_key) returns the first non-empty match:

1. auth_config.key_env → environment variable.
2. auth_config.api_key (DB-stored), ignoring _PLACEHOLDER_KEYS.

Missing credentials short-circuit the call with an actionable error naming the env var and pointing at set_datasource_auth (tools/external.py:_exec_set_datasource_auth). The connector never lets the request go out with the auth param silently dropped, that’s what produced the unrecoverable AQS “variable is missing or the value is empty: email” before this guard landed.

To attach a runtime credential without redeploying:

# Equivalent to the set_datasource_auth tool
ds.auth_config["api_key"] = "<key>"
ds.auth_config["additional_params"] = [{"name": "email", "value": "..."}]  # AQS-style

scripts/seed_api_definitions.py:_merge_auth_config preserves api_key, oauth_refresh_token, session_cookie, and bearer_token across re-seeds, those are runtime-stored, never overwritten by YAML. The structural fields (key_param, key_env, key_header, additional_params) are YAML-authoritative.

Caching

A response is cached when the catalog save records source_datasource_id and the canonicalised query_parameters on the CatalogEntryRecord. On the next call, _check_cache looks up the most recent live entry with matching (source_datasource_id, query_parameters) whose freshness_expires_at is in the future, and loads the parquet from disk. query_parameters is canonicalised via ConnectorService.canonicalize_params so {"limit": 100} and {"limit": 100.0} collide. The lookup key also folds in _endpoint_id because the writer injects it before persistence, the read key has to match the write key.

GeoParquet metadata (b"geo" key in the file’s Parquet metadata) is detected and routed through geopandas.read_parquet rather than pd.read_parquet. Going through pandas alone strips the geometry metadata and returns WKB bytes, which then get nulled out by the geometry hydrator downstream. Cache HIT is intended to return the same shape as a fresh fetch.

After a save, any older live entry with the same cache key is tombstoned (deleted_at, deleted_reason="cache_refresh"). One live row per (datasource, params).

Circuit breaker and retries

Per-datasource state in ConnectorService._circuits:

• 5 consecutive failures → circuit opens for 300 seconds. Calls during the cooldown return HTTP 503 with the cooldown in the message.
• Any 2xx or 4xx (server responding) resets the failure count.
• _make_client/_make_async_client use a 120 s timeout, long enough for ECHO’s QID round-trip, Envirofacts large-state pulls, and ArcGIS cache warm-ups under load. Tighter timeouts were the source of spurious 503s.
• 5xx is retried with exponential backoff + jitter (max_retries=3 default).
• TLS chain failures trigger one AIA augmentation attempt via shared.tls.maybe_augment_for(host) (_rebuild_clients_for_host); on success the request is replayed without consuming an attempt.
• 4xx HTML bodies are scraped with a regex set (<h1>, <message>, "message") so the LLM sees a usable error rather than a stack of markup.
• 200 OK with Results.Error.ErrorMessage (ECHO) or error.code/message (ArcGIS) is detected post-hoc and surfaced as a tool error so it doesn’t masquerade as success.

Health probe

scripts/datasource_probe.py runs the same ConnectorService.query path the LLM uses, against the endpoint’s probe_params (or test_params). It updates DB columns directly: status, consecutive_failures, last_health_check, last_success_at, last_error.

State machine in _apply_state_machine:

Probe outcome	Effect
ok (2xx)	status = active, consecutive_failures = 0, last_success_at = now
throttled (HTTP 429)	status = active, failures cleared, 429 proves the source is reachable. Without this, ECHO’s 300/hour anonymous cap could offline the source.
fail (5xx or non-2xx non-429)	failure counter +1. First failure on an active source → degraded. Third consecutive → offline.
skipped (no probable endpoint, all skip_probe: true)	No state change.

The probe respects manual operator overrides: status = "offline" set by hand (or via YAML) is never resurrected by a successful probe. Probes are grouped by host with a 1 s delay between calls to the same upstream and a 30 s per-probe wall.

Operationally the probe runs from infrastructure/background_worker.BackgroundWorker; the CLI entry point also exists for one-shot debugging:

python -m scripts.datasource_probe

Knowledge corpora for endpoint discovery

scripts/seed_api_docs.py walks every YAML, builds a long-form markdown reference for the datasource (overview, auth, rate limits, authority domains, every endpoint with parameter tables, default params, response chain, example queries, known issues, response schema), embeds it through the standard IngestionPipeline, and links the resulting Corpus back via documentation_corpus_id. The script is idempotent, it skips datasources whose corpus already has chunks; --force re-generates after YAML edits.

The model reaches these corpora through the same search_documents path it uses for any other knowledge, corpora are flagged system=True and ranked alongside user-uploaded docs by relevance. This is how the LLM discovers endpoints semantically rather than by enumerating list_datasources.

Discovery from the model side

Tool	Use
list_datasources(query=...)	Browse / keyword-filter the registry. Returns name, ID, base URL, auth state, status, and every endpoint with its parameters.
search_catalog	Vector search across catalog entries, datasources show up alongside materialised datasets.
describe_entry	Inspect a single datasource: endpoints, parameter docs, example queries, known issues.
register_datasource	Add a new datasource at runtime. System admins (with system.tenants.manage) can pass scope="system" for tenant_id="*"; everyone else gets tenant scope. Duplicate-URL detection compares normalised URLs and host-only (the April 2026 incident where …/nwis slipped past a host-equality check).
set_datasource_auth	Store credentials in auth_config (refuses placeholder strings).
query_datasource	The actual call.

Adding a new datasource

The fast path:

1. Drop a YAML file into data/api_definitions/. Use a similar existing source as a template, airnow.yaml for header-key auth, nwis.yaml for response-path + records strategy, echo.yaml for response chains, ejscreen.yaml for PostGIS transport.
2. Every endpoint MUST declare probe_params (or legacy test_params), OR set skip_probe: true. The seeder rejects the whole file otherwise (SEED REJECTED, N endpoint(s) missing both probe_params and test_params).
3. Path-style endpoints MUST NOT bake literal filter values into path. Use templates ({STATE_CODE}) or pass values via probe_params. The seeder lints this and refuses (SEED REJECTED, N endpoint(s) have hardcoded filter values baked into the path), verified to catch the regression that broke five EPA YAMLs.
4. Run python scripts/seed_api_definitions.py. The seeder verifies that the DB’s known_endpoints round-trip matches the YAML JSON-for-JSON; mismatches raise immediately at seed time.
5. Set credentials if any. Prefer key_env over auth_config.api_key. For api.data.gov-fronted services, point key_env at CATALYST_REGULATIONS_GOV_API_KEY.
6. Generate the documentation corpus: python scripts/seed_api_docs.py --file <slug>. This links documentation_corpus_id so search_documents can find the endpoints.
7. Verify health: python -m scripts.datasource_probe runs one cycle and writes status/last_success_at/last_error per row.
8. Live-test through the tool surface: query_datasource(datasource_name="...", endpoint_id="...", parameters={...}).

To deactivate a duplicate or dead source, add an entry to data/api_definitions/cleanup_duplicates.yaml:

deactivate:
  - id: datasource-old-broken
    reason: "Superseded by datasource-new, same upstream, fixed path."

The seeder processes that file first and flips status = "offline" before upserting the rest. Offline datasources stay in the registry for traceability and the query_datasource tool returns a pivot hint instead of attempting the call.

Operational notes worth knowing

• Status precedence. Operator-set offline (YAML or SQL) wins over probe state. Any other YAML status defers to whatever the probe last wrote, re-seeding can’t accidentally reset degraded to active.
• Probe presence is a hard requirement. probe_params: {} (empty dict) is valid for parameter-less listings; probe_params absent entirely is rejected.
• Credentials never overwritten by re-seed. api_key, oauth_refresh_token, session_cookie, bearer_token are preserved from the DB across seed_api_definitions.py runs.
• response_chain.follow_params.responseset is a hard cap. Once the ECHO QID cursor is created, the follow-up pagination cap is fixed; the tool layer caps caller-supplied responseset to the chain value.
• Large GeoDataFrames go to PostGIS. >500 features or >1 MB of WKB geometry → ext_<snapshot_name> table with a GIST index, recorded as intelligence.postgis_table on the catalog entry. Spatial tools detect the marker and push down filters.
• The connector singleton is shared across the process. get_connector() (shared/connector.py:get_connector) holds a single httpx sync client and one async client; AIA augmentation rebuilds them in place.