Calibration: Wave 2 Results & Key Decisions

PI Briefing — 6 July 2026 | 30 min | Smoking cessation ABM history matching

Established Where we stand

NROY designs (Design B)

19.5

Best Q_STPM (Design B)

22 / 30

Survive multi-seed (10 seeds)

NROY designs (Design A)

Two bugs fixed (TSQ tunnel-exit clock + initiation rates for ages 31+). Combined effect: −3.2 pp prevalence shift for 99.6% of designs.
Wave 1 replayed with fixed code → best Q_STPM 90→58 (A), 70→55 (B). Still zero NROY under pre-fix V_s.
Wave 2 (10k new designs from conditioned priors) → 30 NROY in Design B, 1 in Design A. Best candidate: design_8136 (Q=19.5, passes Jackson, prevalence falls −0.55 pp).
Multi-seed validation: 22/30 survive 10-seed scoring (73% survival rate). Median multi-seed Q = 31.0, still below cutoff.
Design B (geography frozen) consistently outperforms Design A.

How did Q_STPM improve?

Design A: 90.0 → 32.3
Step	Best Q	Δ	Share
Original W1	90.0	—	—
+ Bug fixes	57.9	−32.2	55.7%
+ V_s increase	35.6	−22.3	38.6%
+ Conditioning	32.3	−3.3	5.7%

Design B: 70.0 → 19.5
Step	Best Q	Δ	Share
Original W1	70.0	—	—
+ Bug fixes	35.6	−34.4	68.1%
+ V_s increase	20.4	−15.1	30.0%
+ Conditioning	19.5	−0.9	1.9%

Bug fixes dominate (56–68%). V_s increase is second (30–39%). Conditioning contributed only 2–6% — this is the key problem driving Decision 2.

Decision What do we do about stochasticity?

What is V_s and why does it matter?

The implausibility score Q_STPM measures how far the model’s outputs are from the calibration targets, scaled by the total uncertainty budget:

Q_STPM = ∑ (model − target)² / (V_obs + V_s + V_md)

V_obs = observational/sampling variance from the STS/STPM targets (fixed)
V_s = stochastic variance — how much the model’s outputs vary between random seeds for the same design
V_md = model discrepancy (currently set to 0)

When V_s is larger, the denominator is larger, so Q_STPM is smaller (more “forgiving”). The question is whether the post-fix V_s is the correct estimate or an overestimate.

What changed

Fact	Detail
V_s increased post-fix	STPM stochastic variance grew 1.77× (A) / 2.12× (B). STS unaffected (1.0×).
Why this is genuine	The TSQ fix makes the quit-success pathway “sticky” — successful quitters no longer relapse immediately, amplifying seed-to-seed variation. The pre-fix V_s was artificially low.
V_s share of total variance	65–85% (up from 37–79% pre-fix)

The consequence

NROY (either design)
with pre-fix V_s

NROY (Design B)
with post-fix V_s

22 / 30

Survive multi-seed
(10 seeds each)

9–11

r* (optimal seed count)
for stable V_s

All 30 NROY designs exist only because V_s increased. Multi-seed validation shows 73% are robust, not just lucky seeds.

Decision needed

Accept post-fix V_s as the correct scoring baseline? The argument: the pre-fix V_s was wrong because the bug suppressed genuine stochasticity. Report both V_s versions as a sensitivity check in publications.
How many seeds per design in wave 3? With 1 seed, single-seed noise is large (8 of 30 NROY flip out under 10-seed scoring). r* suggests 9–11 seeds for stable estimation. Trade-off is runtime (see Runtime Estimates).

Stochasticity report — Design A Stochasticity report — Design B NROY deep dive (1-seed vs 10-seed)

Decision What do we do about conditioning?

The problem

Conditioning contributed only 2–6% of the total Q improvement. The wave-to-wave learning loop is not yet working as intended.
Prior shrinkage was negligible for 35 of 40 parameters (<5% SD reduction). The conditioning was effectively a mean shift, not a width reduction.
maintenance.bias is the only real mover — SD narrowed 30% (A) / 46% (B). This parameter is the structural separator between STPM-quality and prevalence-passing designs.
The structural disjunction persists: STPM-good designs need maintenance.bias near zero; prevalence-passing designs need it strongly positive. The top-1000 selection averaged these two populations.

Why was conditioning weak?

Cause	Detail
Selection too broad	Top-1000 (10%) from 10k — most parameters look the same as the full distribution.
STPM-dominated ranking	Q_STPM gap between good (~100) and constraint-passing (~444) designs is >300, so top-K is almost entirely STPM-quality designs.
Bimodal posterior	`maintenance.bias` ≈ −0.05 (STPM-good) vs +0.66 (prevalence-passing). Gaussian recentring cannot capture both modes.

Wave 2 test evidence (2.5k designs per combo, pre-fix V_s)

Metric	S1×B (STPM-ranked)	S2×B (Constraint-first)
Best Q_STPM	87.0	90.8
Q_STPM p5	145.2	173.5
Prevalence falls	14.0%	42.9%
Jackson pass	29.1%	41.3%
Both pass	5.6%	16.3%

S1 gives better Q_STPM; S2 dramatically improves constraint satisfaction (3× more designs with falling prevalence).

Decision needed

Do we want stricter conditioning for wave 3? The current approach barely moved the priors.
Which strategy?

Strategy	Pros	Cons
Smaller K (top-200 or top-300)	Stronger shrinkage; better concentration	May exclude constraint-passing region
Union pool (top-500 STPM ∪ all constraint-passers)	Captures both regimes	Bimodal; Gaussian recentring can’t capture two modes
Constraint-first (only J+prev passers, ranked by Q)	Forces constraint-passing exploration	Small pool (~300–400); Q floor high (~166)
Mixture-of-Gaussians (2-component GMM)	Properly captures bimodal posterior	More complex; custom Sobol from mixture

Conditioning strategy comparison (wave 1) Wave 3 conditioning options (new) Wave 2 Design B full report

Info Runtime estimates

This machine (measured from wave 2)

Parameter	Value
Machine CPUs	128
Parallel workers	124 (CPU − 4)
ABM run time (per run, 72 ticks)	~10 min
10k × 1 seed × 1 design	16.7h (ABM 13.9h + aggregate 2.8h)
10k × 1 seed × 2 designs	33.5h (sequential A then B)

Wave 3 cost estimates

Scenario	ABM runs	Wall time (est.)	Notes
10k × 1 seed × 2 designs	20,000	~33h	Same as wave 2
10k × 3 seeds × 1 design (B)	30,000	~50h	Multi-seed, Design B only
10k × 5 seeds × 1 design (B)	50,000	~70h (2.9d)	Full stochastic scoring
5k × 10 seeds × 1 design (B)	50,000	~70h (2.9d)	Fewer designs, more seeds
2k × 10 seeds × 1 design (B)	20,000	~28h	Targeted deep exploration

Based on actual wave 2 timings: 10k×1 ABM batch took 13.9h per design on 124 workers. Aggregate adds ~2.8h. Estimates assume linear scaling.

Appendix: All reports

Wave 1 & bug-fix

Original wave 1 (pre-fix) Post-fix wave 1 (10k) Paired comparison (10k) Paired comparison (5k) Replay 5k combined

Stochasticity

Design A Design B

Wave 2

Wave 2 — Design B Wave 2 — Design A Wave 2 — Combined A+B

Conditioning & deep dives

Conditioning comparison Wave 3 options NROY deep dive (W1) NROY deep dive (W1+W2)

Generated 2026-07-06 13:05 | Data: post-fix W1 (20260703_0711), wave 2 (20260704_1141), multi-seed (20260706_0847)

Calibration: Wave 2 Results & Key Decisions

Established Where we stand

How did QSTPM improve?

Decision What do we do about stochasticity?

What is Vs and why does it matter?

What changed

The consequence

Decision needed

Decision What do we do about conditioning?

The problem

Why was conditioning weak?

Wave 2 test evidence (2.5k designs per combo, pre-fix Vs)

Decision needed

Info Runtime estimates

This machine (measured from wave 2)

Wave 3 cost estimates

Appendix: All reports

Wave 1 & bug-fix

Stochasticity

Wave 2

Conditioning & deep dives

How did Q_STPM improve?

What is V_s and why does it matter?

Wave 2 test evidence (2.5k designs per combo, pre-fix V_s)