Wave-1 calibration report — single-seed implausibility analysis

Run: output/calibration_runs/MERGED_10000_complete_R11cov · 10000 sampled parameter sets per design × two design configurations (A: geography free, B: geography frozen) × 1 random seeds. All scores in this report are single-seed.

2. How we score each candidate

Each candidate $\theta$ is scored against two 16-target blocks ($B \in \{\text{STPM}, \text{STS}\}$). For a single seed $s$, the residual vector $d^{(s)}(\theta) = y^{(s)}_{\text{sim}}(\theta) - y_{\text{obs}}$ is scored against the block covariance $\Sigma_B = V_{\text{obs}} + V_{\text{seed}} + V_{\text{disc}}$ (with $V_{\text{disc}}=0$ for this run):

Because this run has one seed per parameter set, the formal Q scores use borrowed ABM stochastic covariance from the previous multi-rep v8 run, rather than estimating V_seed from this trial or setting it to zero.

Two implausibility tests, both at level $\alpha=0.005$:

• Multivariate: reject $\theta$ if $Q_B^{(s)}(\theta) > \chi^2_{16}(0.995) = 34.267$ in either block.
• Univariate (maximin): reject if $\max_j I_j^{(s)}(\theta) > 3$ in either block.

For target-subset sensitivity we use a marginal-Q screen: $\tilde Q_S^{(s)} = \sum_{j \in S} d_j^{2} / \Sigma_{jj}$. This sums each target's squared standardised residual and ignores covariance cross-terms, so it is a screening tool, not the formal multivariate test. The reference cutoff scales as $\tilde Q_S \sim \chi^2_{|S|}(0.995)$ under the null.

3. What if we drop the dominating targets?

Every reactive plot below is driven by the top-K% designs (lowest composite score). The composite is a six-term weighted sum: score(i) = w_Qs·Q_STPM(i) + w_Qt·Q_STS(i) + w_Is·(I^max_STPM−3)₊ + w_It·(I^max_STS−3)₊ + w_J·jackson_fail_count(i) + w_P·(Δprev)₊. The prevalence-direction term penalises designs where overall smoking prevalence increased between 2011 and 2016 (Δprev = prev₂₀₁₆ − prev₂₀₁₁; only the positive part contributes). Use the weight inputs in the floating bar to emphasise or suppress any term (set a weight to 0 to drop it).

3a. Live summary

The $Q/Q^*$ cards show the 5th, 50th, and 95th percentiles across all parameter sets for the selected design and seed. A value of 1.0x is the multivariate cutoff; values above 1.0x fail.

3b. Best-performer envelope per target (top-K%)

For each selected target, the bar shows the [min, max] range of $I_j$ across the top-K% of designs by composite score at the current seed; the diamond marks the median. A target whose envelope sits above the dashed cutoff $I_j = 3$ is unreachable even by the best candidates — those are the targets driving the multivariate-$Q$ ledger.

3c. Subset marginal $\tilde Q_S$ distribution

3d. Per-target fit in natural units

Grouped bar chart of calibration targets (blue, ±1 SE from observed data) and the ABM output for the top-K% designs by composite score (coloured, mean ± 5th–95th percentile range across top-K% designs). STPM y-axis is annual quit rate; STS y-axis is quit-attempt rate over 12 months. The dashed vertical line separates the 10 IMD-period cells (left) from the 6 sex×age cells (right). A good fit has the coloured bar aligned with the blue bar on every cell.

4. Where parameters want to move

For the composite top-K ranking, this section shows the diagnostic reweighting over every free parameter at once: prior (dotted gray) vs. the implausibility-weighted empirical density on the wave-1 candidates (solid colour). Pick the ranking rule that matches the question you want to answer.

5. How many candidates survive each test?

For the current design × seed, this section counts how many scored candidates pass each formal test, individually and in combination. The parallel-coordinates plot below the table renders one line per design across five normalised axes (lower is better); the top-K% by composite score are drawn in colour, the rest are faded grey.

5a. Pass-count ledger

5b. Parallel coordinates — five scores at once

Axes are normalised: $Q_B / \chi^{2*}$, $\max_j I_j / 3$, and survival-failure count (number of checkpoints out of 3 that fall outside the Jackson envelope). A design with every axis $\le 1$ would pass every formal test. Top-K% are drawn solid; the rest are faded.

6. Survival curves vs Jackson envelope

Each thin line is one of the top-K% best designs at the current seed; it traces simulated survival probability at months 3, 6, and 12. The grey ribbon is the selected Jackson et al. (2019) survival band. The dotted line is the placebo reference, the dashed green line is varenicline. Designs are coloured by rank within the top-K (darker = lower composite score).

7. What do the best-ranked candidates imply? — targets and networks

Two views restricted to the top-K% by composite score. The heatmaps show each target's marginal miss size, $I_j^2 = d_j^2 / \Sigma_{jj}$, for the K best-ranked designs; rows are sorted best-to-worst by composite score. A bright column is a target the best-ranked candidates still miss badly. The bar chart below compares the prior network share against the empirical share within the top-K%.

7a. Which targets the top-K% still miss

Each row is one selected parameter set: rank 1 is the lowest composite score in the selected top-K%. Each column is one target. Colour is log-scaled so moderate misses stay visible. The blue-to-orange transition marks $I_j^2 = 9$ ($I_j=3$), the univariate cutoff; orange, red, and purple cells are targets still missed by the selected top-K% designs. This is a marginal per-target view, not the full multivariate Q with covariance cross-terms.

7b. Which networks do the top-K% prefer?

Bars compare the uniform prior (1/4 per candidate) against the top-K% empirical network share. Green outline means the candidate is over-represented vs the prior; red means under-represented.

8. Smoker prevalence 2011–2016 (top-K designs)

Shaded band = p5–p95; solid line = median across the top-K% designs selected by the composite score above. Reactive: updates whenever Top fraction or any weight changes.