Skip to content

fix(sensitivity): refit at fixed multipliers of the slab scale, classify on the fits' own statistics - #221

Open
MaartenMarsman wants to merge 3 commits into
developfrom
fix/anchor-refit-check
Open

MaartenMarsman wants to merge 3 commits into
developfrom
fix/anchor-refit-check

Conversation

@MaartenMarsman

Copy link
Copy Markdown
Collaborator

prior_sensitivity_check() becomes a fixed-scale sensitivity analysis: full-length refits at a few multipliers of the chosen slab scale, classified on the fits' own statistics. The importance-reweighted curve between anchors, its plot method, the replicate refit with its noise band, and the convergence gate are removed. Replaces the closed #220.

What the check does now

  • Refits the model at every multiplier in scale_multipliers (default c(0.5, 2); the argument was anchors) with the original fit's iter and warmup. The original fit is the 1x fit and is not refit. Ordinal refits start from the original fit's final state and NUTS metric; GGM and mixed refits start cold (warm starts for those: backlog Item 48).
  • Every quantity per fit is that fit's own Rao-Blackwellized inclusion probability, log Bayes factor, verdict, and the MCSE of the log Bayes factor from the fit's summary.
  • Per edge, against the original fit's verdict: stable when every refit's verdict equals it; otherwise beyond-monte-carlo-error when, at some refit whose verdict differs, the log-BF change from the original fit exceeds twice the two fits' combined MCSE, and within-monte-carlo-error when it does not at any. moved_at is the differing multiplier nearest 1. stability_lower and stability_upper are the ends of the contiguous run of multipliers around 1 holding the original verdict. insufficient at the chosen scale is the only not-certifiable criterion, computed as before.
  • No gate. Each fit's largest split R-hat, smallest ESS, smallest inclusion ESS, E-BFMI and energy variance ratio are in $grid; the print names fits above R-hat 1.01. All fits enter the classification.
  • Two functions: prior_sensitivity_engine() runs the refits and takes the per-fit tables; sensitivity_classification() classifies from the tables. The refit settings are stored per fit in $grid (backlog Item 49, rerunning one fit longer, builds on this).

Removed: R/anchor_curve.R and its tests, plot.bgms_prior_sensitivity(), the arguments anchors, ess_floor and tolerance, the short refit schedule, the replicate refit and $wobble, refit_convergence_gate(), the weekly reweighting block, the mover levels indistinguishable-from-wobble and moved-beyond-wobble, the curve matrices and $curve. $anchor_verdict is $verdict_by_scale.

Commits

  1. Engine, classification and print; curve code removed; man pages.
  2. Tests: the classification on synthetic per-fit statistics, the print, the check on small fits; the plot-method tests that used the removed draw extractor.
  3. Vignette rewritten and its stored object regenerated; NEWS; intro vignette sentence; data generator.

Wenchuan (tutorial script R/16: 15 variables, 136 edges, 4 x 10000 draws, 5000 warmup)

Fit 137 s, check 245 s (two refits at the fit's length). Printed report:

Prior sensitivity check: are the edge verdicts robust to the slab scale?
Verdicts at an inclusion Bayes factor of 10 from fits at 0.5x, 1x and 2x the
chosen scale; the 1x fit is the original fit. 136 edges.

Verdict counts by scale:
            0.5x 1x 2x
  presence    37 35 32
  undecided   54 39 30
  absence     45 62 74

  verdict held at every scale             102
  changed, within Monte Carlo error         0
  changed, beyond Monte Carlo error        26
  not certifiable at the chosen scale       8

26 edges changed verdict beyond Monte Carlo error:
  edge            0.5x       1x         2x         differs from
  intrusion-numb  undecided  absence    absence    0.5x        
  dreams-upset    presence   undecided  undecided  0.5x        
  flash-upset     undecided  absence    absence    0.5x        
  flash-avoidth   undecided  undecided  absence    2x          
  flash-avoidact  undecided  undecided  absence    2x          
  flash-sleep     undecided  absence    absence    0.5x        
  flash-anger     presence   presence   undecided  2x          
  flash-concen    presence   presence   undecided  2x          
  upset-numb      undecided  undecided  absence    2x          
  upset-future    undecided  undecided  absence    2x          
  and 16 more; see $edges.

8 edges are not certifiable at the chosen scale: intrusion-startle,
dreams-numb, flash-startle, upset-avoidact, upset-concen, avoidth-concen,
avoidth-hyper, avoidact-startle. In the original fit the Bayes factor lies
within Monte Carlo error of a threshold or the chains disagree on the verdict,
so a rerun with another seed could change the verdict without any change of
prior.

Largest split R-hat above 1.01: 2x fit 1.013.

Note: the slab scale is 1, and the estimated edge weights imply a scale of 0.18
(approximate 95% interval 0.148 to 0.218); the slab is 5.6 times that. A slab
wider than the weights it describes moves near-threshold verdicts toward
absence.

Refits:  2 nuts refits (5000 warmup, 10000 sampling iterations each), started from the original fit's final state, 243 s total.
See ?prior_sensitivity_check for the construction.

CRAN build on the same script: robust 91, changed within noise 6, changed beyond noise 31, not certifiable 8, from five short refits and a reweighted curve.

Verification

  • tests/testthat/test-prior-sensitivity.R: 79 pass at tier T0; the slow and certification blocks (bgmCompare, GGM and mixed, warm-versus-cold refit) were not run.
  • R CMD check --as-cran: 0 errors, 0 warnings, 0 notes (2026-10-29).
  • Develop-vs-branch demo dev/review-2026-08/repro/run-anchor.sh on the six-variable Wenchuan fit (2 chains, 2000 draws): develop fits five short refits plus a replicate, reports a 43-point curve and names one edge as scale-dependent whose verdict is undecided at every anchor; the branch fits two refits at the fit's length, has no curve or plot method, and reports that edge as held, with the one not-certifiable edge the only exception. Comparison in ~/bgms-review/anchor-comparison.txt.

…assify on the fits' own statistics

prior_sensitivity_check() refits the model at scale_multipliers (default
c(0.5, 2)) times the chosen scale, each refit with the original fit's iter
and warmup, warm-started for ordinal fits. Each fit contributes its own
Rao-Blackwellized inclusion probability, log Bayes factor, verdict and the
MCSE of the log Bayes factor. An edge is stable when every refit's verdict
equals the original fit's; otherwise it is beyond Monte Carlo error when the
log-BF change to a refit with a different verdict exceeds twice the two
fits' combined MCSE, within it otherwise. insufficient at the chosen scale
is the only not-certifiable criterion. Each fit's diagnostics are in $grid;
no fit is excluded. The reweighted curve, its plot method, the replicate
refit and noise band, the short refit schedule and the convergence gate are
removed; the colour palette the other plots use moves to plot_style.R.
…int, and the check on small fits

The curve tests and the weekly reweighting block go with the curve. The
plot-method tests that read the slab frame through the removed draw
extractor compare the extractor with the raw draws directly.
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.44444% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.14%. Comparing base (0807fba) to head (c3aa239).
⚠️ Report is 13 commits behind head on develop.

Files with missing lines Patch % Lines
R/prior_sensitivity.R 87.67% 26 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #221      +/-   ##
===========================================
+ Coverage    92.84%   93.14%   +0.29%     
===========================================
  Files          115      114       -1     
  Lines        18697    18495     -202     
===========================================
- Hits         17360    17227     -133     
+ Misses        1337     1268      -69     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant