# Shadow Research Agent **Status:** shadow mode only **Scope:** source triage, poll extraction, event classification, and grounded editorial drafts **Out of scope:** model inputs, forecast math, forecast execution, publication, and deployment ## Safety contract The research agent is a staging assistant, not a forecast agent. Its entire data path is: ```text allowlisted source or approved immutable context ↓ sanitized, block-addressed snapshot ↓ strict structured model response ↓ deterministic checks and dedupe hints ↓ needs_human_review shadow record ``` There is no path from this package to `data/polls/polls.csv`, `data/events/events.json`, `data/fundamentals/fundamentals.csv`, `config/model.json`, `forecast/model.py`, the forecast runner, `public/`, the site build, or deployment. The package exposes no function that promotes a source, approves its own work, changes a weight or probability, runs a forecast, publishes content, or deploys the site. Every run reports those capabilities as false and sets `publishable: false`. A valid result is still `needs_human_review`; an unsafe or invalid result is `blocked`. The agent cannot produce an `approved` state. If this work is later integrated with an editorial system, a separately implemented human action must record reviewer identity, timestamp, rationale, and the exact reviewed content hash. ## Responses API boundary The optional live call uses the OpenAI Responses API through a small standard-library client. It follows the official [Structured Outputs guidance](https://developers.openai.com/api/docs/guides/structured-outputs): each task sends a strict JSON Schema through `text.format`, with `strict: true`, all fields required, and nullable types for information that the source does not disclose. Missing scalars must be `null`; missing lists must be empty arrays. The request sets `store: false`, supplies no tools, and sets `tool_choice: none`. Fixed instructions are placed in a developer message. Sanitized source material is placed only in the user message as explicitly untrusted data. This implements the separation recommended in OpenAI's [agent safety guidance](https://developers.openai.com/api/docs/guides/agent-builder-safety), including keeping untrusted content out of developer instructions and constraining data passed between steps. API access is disabled by default. `--use-api` is required, and the call fails before network access unless `OPENAI_API_KEY` is present. The API key is read only by the CLI process, is never placed in a prompt or result, and must not be committed. The endpoint is fixed to `https://api.openai.com/v1/responses`; it is not user-configurable. ## Source discovery and fetch boundary `data/research/allowlist.json` is the sole committed host/content-type policy. Discovery only canonicalizes and filters candidate URLs; it neither fetches pages nor follows links. Fetching requires the separate `--allow-network-fetch` flag and performs one bounded HTTPS GET. The fetcher: - rejects credentials, non-HTTPS URLs, non-default ports, non-allowlisted hosts, and private or special-purpose IP literals; - rechecks every redirect against the same allowlist; - caps time and response bytes; - rejects unapproved MIME types; - follows no links contained in a document; and - keeps the response in memory unless the operator separately requests an immutable shadow record. PDF bytes can be hashed and staged, but this dependency-light MVP does not attempt PDF/OCR extraction. They remain `requires_external_text_extraction: true` until a reviewed parser is added. ## Prompt-injection defenses Before any model call, active HTML elements, hidden direction/control characters, null bytes, and invisible separators are removed. A conservative detector blocks instruction overrides, role impersonation, tool directives, secret-exfiltration requests, and attempts to alter or publish the forecast. A blocked document never reaches the Responses API, even if the operator used `--use-api`. This detector is defense in depth, not a claim that keyword screening solves prompt injection. The stronger controls are architectural: no tools, no credentials in context, no write capability, strict output fields, exact evidence-block IDs, local revalidation, and mandatory human review. ## Provenance and immutability Raw source text is SHA-256 hashed before normalization. Each normalized block has a stable ID and its own hash. A source snapshot records: - source ID, canonical URL, title, publisher, publication and retrieval times; - MIME type; - raw and normalized content hashes; - block IDs, block text, and block hashes; - a document hash over the content identity; and - a snapshot hash over the complete provenance record. Every model-backed or offline-fixture run also records the exact prompt and schema hashes, provider/API mode, model identifier, response ID when available, storage setting, and whether tools were enabled. Structured output must echo the exact document hash. Commentary requires an exact approval hash over the supplied forecast facts and exact approved document hashes for all cited sources. Any later mutation invalidates the approval check. The optional `--record-shadow` action writes only a content-addressed, create-only JSON file under `data/research/shadow_runs/`. It never accepts an output path and refuses conflicting overwrite. ## Deterministic validation and deduplication The Responses API schema is rechecked locally so offline fixtures, mocked transports, and model changes share the same boundary. Poll checks cover source-hash binding, ISO dates, chronology, unique candidates, percentages, totals, evidence-block existence, and disclosure gaps. Event checks prohibit narrative campaign stories from masquerading as defined numerical inputs. Duplicate detection is deliberately conservative: - identical document hash → `exact_document` suggestion; - matching pollster, sponsor, field dates, population, and sample size → `probable_same_sample` suggestion; - every suggestion has `automatic_merge: false`. Question variants and voter-screen variants therefore cannot be silently collapsed. A human decides whether records are duplicates, related releases, or distinct ballot tests. ## Grounded commentary Commentary accepts only a context whose forecast facts and sources carry human approval records and matching immutable hashes. The model never receives a writable forecast object. Forecast values are pre-rendered facts such as `55%`; the model must cite their `forecast_fact_id` and reproduce the rendered value exactly. All prose is returned as individually typed claims: - `model_output` cites only approved forecast fact IDs; - `reported_fact` cites source IDs and block IDs from those sources; - `analysis` cites one or both and should state uncertainty. There is no uncited narrative field. The validator rejects unknown sources or blocks, source/block ownership mismatches, forecast values absent from claim text, and numeric reported facts absent from cited evidence. Even a fully valid draft remains non-publishable and requires human review. ## Running locally Offline fixture run, with no network or API key: ```sh python3 scripts/run_research_agent.py --task poll \ --candidate-records data/research/fixtures/poll_candidate_ledger.json python3 scripts/run_research_agent.py --task event python3 scripts/run_research_agent.py --task commentary python3 scripts/run_research_agent.py --task discover ``` Explicit live structured extraction: ```sh OPENAI_API_KEY=... python3 scripts/run_research_agent.py \ --task poll --use-api --model YOUR_EVALUATED_MODEL_ID ``` The CLI deliberately has no implicit live-model default. Select and pin a model only after the model/prompt/schema combination passes the held-out evaluation; `OPENAI_MODEL` may be used instead of `--model`. The live command still performs no fetch, promotion, forecast run, or publication. To fetch one allowlisted source for inspection: ```sh python3 scripts/run_research_agent.py --task fetch \ --allow-network-fetch \ --url https://www.opb.org/example/ ``` ## Evaluation and launch boundary Run the dependency-free offline suite with: ```sh python3 scripts/evaluate_research_agent.py python3 -m unittest tests.test_research_agent -v ``` The committed fixture gate requires 100% case success, 100% exact critical-field agreement, 100% blocking of its adversarial source, zero unauthorized capabilities, and zero published artifacts. It covers valid and invalid poll structures, duplicate suggestions, event authority, immutable commentary inputs, citation invention, and missing-key/API-key failures. These are structural MVP tests, not a live-model quality claim. Before any production ingestion or public drafting, build a held-out, human-adjudicated Oregon corpus with original poll PDFs, revisions, same-sample variants, OCR failures, irrelevant stories, conflicting sources, and adversarial content. Track discovery recall, exact numeric/date/sample accuracy, citation entailment, unsupported-claim rate, reviewer override rate, and injection success. Prompt, schema, or model changes must pass regression and shadow evaluation. OpenAI similarly recommends task-specific evals, human-calibrated graders, adversarial cases, and continuous evaluation in its [evaluation guidance](https://developers.openai.com/api/docs/guides/evaluation-best-practices). Automatic promotion, automatic approval, forecast integration, and automatic publication are explicitly not authorized by this specification.