Skip to content

fix(sensitivity): batch-means error, inverse-variance pooling, reach rule and gate flags in the curve engine - #220

Closed
MaartenMarsman wants to merge 5 commits into
developfrom
fix/sensitivity-engine
Closed

MaartenMarsman wants to merge 5 commits into
developfrom
fix/sensitivity-engine

Conversation

@MaartenMarsman

Copy link
Copy Markdown
Collaborator

Reworks the reweighted curve of prior_sensitivity_check() and the way the check classifies edges on it. Design record: dev/plans/active/2026-10-22_sensitivity-engine_DESIGN.md (untracked); reach study: dev/audit/checks/2026-10-reweighting-reach/.

Commits

  1. Curve assembly (R/anchor_curve.R). Batch-means Monte Carlo variance of each reweighted inclusion probability, floored at the smoothed-count variance; inverse-variance pooling of the anchors that reach a point; an anchor contributes only if its importance ESS is at least max(ess_floor, ess_fraction * draws); a point where two anchors differ by Cochran's Q at 0.99 and by more than the move tolerance in log BF is disputed and takes the nearest anchor; at an anchor's own grid position the curve is that fit's Rao-Blackwellized probability.
  2. Engine (R/prior_sensitivity.R, R/refit_engine.R). A gate-failing refit keeps its verdict column, flagged in $grid$gate_flag, contributes to neither the curve nor the noise band, and the grid points nearest to it are not assessed; the original fit is treated the same way and the forced column is gone. New argument ess_fraction = 0.1. Per-point MCSE from the batch-means variance. Classification relative to the reported chosen-scale verdict over assessed points and every usable anchor's own verdict; new level not-fully-assessed with coverage, assessed_from, assessed_to, gap_cause. Sufficiency at every usable anchor (insufficient_any, insufficient_scales).
  3. Printed report and man page.
  4. Vignette text and precomputed data.
  5. Weekly certification block rewritten to the reach rule; NEWS entry.

Effect on the Wenchuan analysis (15 variables, 136 edges, 4 x 10000 draws)

Printed report on the tutorial's script (R/16), CRAN build -> this branch: robust 91 -> 55, changed within noise 6 -> 0, changed beyond noise 31 -> 5, not fully assessed 0 -> 4, not certifiable 8 -> 72. $edges$mover on the branch: stable 78, moved-beyond-wobble 48, indistinguishable-from-wobble 4, not-fully-assessed 6; insufficient at the chosen scale 10. The tutorial's own mover count (R/36) goes 31 -> 40.

  • Moves the classification: the mover rule now compares against the reported verdict and includes each anchor's own verdict (R/36 movers 31 -> 40); the dispute flag (29 of 5848 grid points) produces the 6 edges not fully assessed; sufficiency at every anchor produces the 72 not certifiable, which absorb most of the movers in the printed counts.
  • Does not move it here: the gate change (all six fits pass); the exact-anchor override (changes curve values at the anchors only).
  • Not isolated: the batch-means variance, the pooling weights and the reach rule change the curve values between anchors; their separate effect on the classification was not measured.

The classification will move to the anchor fits in a follow-up PR (tutorial side decision of 2026-10-27); this PR ships the curve engine as designed.

Timing

The check on the Wenchuan fit takes 86 s on this branch against 82 s on the CRAN build, with the fit itself at 132 s against 129 s. The engine's own addition is the batch-means pass, about 5 s at this size. The 365 s reported earlier came from a review library built from unoptimized object files, not from the engine.

Verification

  • tests/testthat/test-anchor-curve.R, tests/testthat/test-prior-sensitivity.R pass at tier T0/T1; the weekly block (tier T2) was run once on the branch.
  • R CMD check --as-cran: 0 errors, 0 warnings, 0 notes (2026-10-28).
  • Develop-vs-branch demo: dev/review-2026-08/repro/run-engine.sh (five demos: pooling at a disputed point, curve at 1x against the fit's own probability, reach at 1.6x/2.5x/4x, printed report, gate-failing short fit); comparison in ~/bgms-review/engine-comparison.txt. Develop against branch: a point with one anchor at 0 and one at 0.8 pools to 0.0025 on develop and is disputed on the branch (takes the nearest anchor, 0); the curve at 1x differs from the fit's own Rao-Blackwellized probability by up to 0.0154 on develop and by 1e-6 on the branch; on a bgmCompare anchor the 4x point (ESS 486, 6% of draws) passes the floor of 400 on develop and is masked on the branch, while 1.6x and 2.5x pass on both. Demo 5's short fit passed the gate on both sides, so the gate-failure path is covered by the unit tests only.
  • The reach study (stage 1 and 2) was run with the study scripts, not with the engine as merged here; the engine implements the rule the study selected and was not rerun over the study cases.

…rule and exact anchors in the curve

anchor_reweight() returns the batch-means variance of each reweighted
inclusion probability, floored at the smoothed-count variance, and the
largest normalized weight. assemble_curve() pools the anchors that reach a
point by inverse variance, requires an importance ESS of at least
max(ess_floor, ess_fraction * draws) for an anchor to contribute, flags a
point as disputed when two anchors differ by Cochran's Q at 0.99 and by more
than the move tolerance in log BF (the point then takes the nearest anchor),
and takes each anchor fit's Rao-Blackwellized probability at the anchor's own
grid position.
…dd ess_fraction, classify on assessed points

A refit that fails the convergence gate keeps its verdict column, flagged in
$grid$gate_flag with the diagnostic that failed; it contributes to neither
the curve nor the noise band, and the grid points nearest to it are not
assessed. The original fit is treated the same way; the forced column is
gone. New argument ess_fraction (default 0.1) sets the per-anchor reach
floor together with ess_floor. The per-point Monte Carlo error comes from
the batch-means variance of the pooled estimate. The mover classification
is relative to the reported chosen-scale verdict, over the assessed grid
points and every usable anchor's own verdict, with a not-fully-assessed
level and the coverage, assessed_from, assessed_to and gap_cause columns.
Sufficiency is evaluated at every usable anchor (insufficient_any,
insufficient_scales).
The report lists edges not fully assessed with their assessed range and the
cause, counts an edge as not certifiable when it is insufficient at any
usable anchor and names the anchors, states the gate result for the
supplied fit and for refits with the failing diagnostic, and gives the
reach rule and what the noise band does and does not measure. The roxygen
details describe the batch-means error, the pooling, the reach rule, the
dispute flag, exact anchors, the finite-run offset and the gate.
…ntry

The weekly block fits the bgmCompare anchor three times, refits at 1.6x
twice, and checks that the anchor reaches 1.6x (ESS at least
max(400, 10% of draws)) and does not reach 2.5x, and that the reweighted
probability at 1.6x lies within max(0.02, 4 * noise) of the refits.
@MaartenMarsman

Copy link
Copy Markdown
Collaborator Author

Closed unmerged. Decision of 2026-10-28: the check becomes three full-length refits at 0.5x, 1x and 2x the chosen scale, classified on the fits' own Rao-Blackwellized verdicts, with no reweighted curve, no plot and no convergence gate. A new PR off develop replaces this one; the branch stays for reference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant