# Historical Polling Baseline

**Consequential Races · Oregon Governor Forecast**

Status: **Production-aggregator calibration, not full-model validation**

Published compatibility ID: `weighted-poll-baseline-0.1.0`

Implementation: `production-poll-aggregation-backtest-0.2.0`

## Purpose

This backtest asks how far the production polling estimate was from the final
normalized Democratic–Republican margin at fixed points in historical
gubernatorial campaigns. It now reuses `forecast.polls.aggregate_polls`
directly; there is no second recency or weighting formula in the calibration
script.

It still does **not** validate the 2022 rematch counterfactual, fundamentals
prior, or complete combined forecast. The public probability remains a
research preview until those components pass rolling-origin validation.

## Data and historical cutoffs

`scripts/calibrate_historical.py` reads 2,318 candidate-answer rows from three
overlapping archived FiveThirtyEight gubernatorial polling snapshots and joins
them to the FiveThirtyEight gubernatorial result archive. After duplicate
questions and samples are consolidated, it contains 710 poll observations
across 54 matched races.

For every race and requested horizon, the cutoff is the end of the calendar day
exactly 120, 90, 60, 30, 14, 7, or 1 day before Election Day. A poll is eligible
only if both its fieldwork and archive publication timestamp were available by
that cutoff. The default maximum age is the production configuration's 180
days; `--lookback-days` remains available as an explicit sensitivity override.
The CLI also accepts repeatable `--horizon DAYS` arguments for focused runs.

The archive adapter:

1. deduplicates overlapping snapshot rows and gives each poll sample one
   observation even when it released multiple ballot tests;
2. preserves Democratic, Republican, other, and residual undecided shares;
3. maps the archive's generic `V` population to the conservative production
   `RV` category;
4. maps historical pollster grades proportionally around the production default
   quality value; and
5. marks internal or partisan releases `accepted_with_penalty` and passes their
   sponsorship direction to the production adjustment.

The production aggregator then performs the live model's exact:

- two-candidate normalization,
- 28-day recency half-life,
- sample-size and missing-sample treatment,
- population and mode multipliers,
- pollster-quality and sponsor adjustments,
- within-pollster correlation penalty and organization precision cap, and
- statistical, systematic, and sparse-error calculations.

The result target is transformed to the identical normalized scale:

```text
100 × (Democratic share − Republican share)
    / (Democratic share + Republican share)
```

The complete configuration, source hashes, race-level diagnostics, and summary
statistics are retained in `data/calibration/gubernatorial_poll_error.json`.
For a larger audit artifact containing every cutoff-specific production poll
weight, run the calibration CLI with `--include-poll-details`.

## Sparse and robust strata

Every historical race-cutoff observation is classified explicitly:

- **Sparse:** fewer than three included polls or fewer than two distinct
  pollsters.
- **Robust:** at least three included polls from at least two distinct
  pollsters.

The strata are complements within `all_available`; a race cannot silently
disappear between them. This distinction matters because the current Oregon
forecast is in the sparse stratum, while the existing forecast configuration
still reads the robust series for its generic historical residual.

## Results

| Days before election | Robust races | Robust MAE | Robust error SD | Sparse races | Sparse MAE | Sparse error SD |
|---:|---:|---:|---:|---:|---:|---:|
| 120 | 10 | 4.48 | 6.08 | 12 | 7.28 | 9.16 |
| 90 | 15 | 5.64 | 6.71 | 11 | 6.60 | 8.03 |
| 60 | 21 | 5.53 | 6.34 | 17 | 8.34 | 10.23 |
| 30 | 37 | 5.68 | 6.79 | 9 | 5.09 | 5.91 |
| 14 | 39 | 5.17 | 6.31 | 8 | 5.57 | 6.98 |
| 7 | 40 | 5.11 | 6.19 | 7 | 5.66 | 7.46 |
| 1 | 43 | 4.71 | 5.88 | 5 | 5.39 | 7.13 |

“Error” is poll margin minus result margin. The robust 1-day mean error is D
+0.66 and the 30-day mean error is R +0.24; these directions are not applied as
an Oregon partisan correction. The samples remain too small and cycle-specific
to assume that directional bias will repeat.

The production poll component's own nominal 80% interval covers only 51% of
robust 1-day results and 54% of robust 30-day results in this archive. That is
direct evidence that sampling and within-poll aggregation uncertainty are not
total forecast uncertainty. The forecast therefore still needs a separately
validated election-error layer; this backtest supplies empirical residual
magnitudes but does not validate how the combined model adds them.

## Remaining limitations and validation work

- Although seven matched observations carry a 2022 election date, they are too
  early to qualify at these fixed cutoffs. Effective race-cutoff samples remain
  2018–2020; the archive must be extended with complete 2022 and 2024 polling.
- FiveThirtyEight `created_at` is the best available release-time proxy, not a
  guaranteed original publication timestamp.
- Historical grades are adapted to the current quality-map interface. House
  effects remain empty because estimating them on the same races without
  leave-cycle-out controls would leak outcomes.
- Sparse samples are very small—only five races at the 1-day cutoff—so their
  SDs must not be treated as precise parameters.
- The exact production poll aggregator is now tested, but the fundamentals,
  Johnson redistribution, poll/fundamental combination, and final uncertainty
  composition are not.

Before research-preview status can be removed, the project still needs to:

- extend the polling archive through the 2022 and 2024 cycles;
- validate the fundamentals prior with rolling or leave-one-cycle-out tests;
- compare the 2022 counterfactual against contemporaneous second-choice polling
  and precinct controls;
- evaluate the complete combined model at every fixed historical cutoff;
- report Brier score, log loss, and 50%/80%/95% interval coverage for final
  simulated probabilities; and
- test sensitivity to correlated state/cycle error, third-party support,
  turnout, sparse polling, and the production 50% pollster precision cap.

The baseline establishes two useful facts: late gubernatorial polling errors of
several points are ordinary, and sparse polling has historically been less
stable than robust polling in the available early-cycle cutoffs.