Repository navigation
fix(sensitivity): batch-means error, inverse-variance pooling, reach rule and gate flags in the curve engine - #220
Closed
MaartenMarsman wants to merge 5 commits into
Closed
MaartenMarsman wants to merge 5 commits into
MaartenMarsman wants to merge 5 commits into
Conversation
…rule and exact anchors in the curve anchor_reweight() returns the batch-means variance of each reweighted inclusion probability, floored at the smoothed-count variance, and the largest normalized weight. assemble_curve() pools the anchors that reach a point by inverse variance, requires an importance ESS of at least max(ess_floor, ess_fraction * draws) for an anchor to contribute, flags a point as disputed when two anchors differ by Cochran's Q at 0.99 and by more than the move tolerance in log BF (the point then takes the nearest anchor), and takes each anchor fit's Rao-Blackwellized probability at the anchor's own grid position.
…dd ess_fraction, classify on assessed points A refit that fails the convergence gate keeps its verdict column, flagged in $grid$gate_flag with the diagnostic that failed; it contributes to neither the curve nor the noise band, and the grid points nearest to it are not assessed. The original fit is treated the same way; the forced column is gone. New argument ess_fraction (default 0.1) sets the per-anchor reach floor together with ess_floor. The per-point Monte Carlo error comes from the batch-means variance of the pooled estimate. The mover classification is relative to the reported chosen-scale verdict, over the assessed grid points and every usable anchor's own verdict, with a not-fully-assessed level and the coverage, assessed_from, assessed_to and gap_cause columns. Sufficiency is evaluated at every usable anchor (insufficient_any, insufficient_scales).
The report lists edges not fully assessed with their assessed range and the cause, counts an edge as not certifiable when it is insufficient at any usable anchor and names the anchors, states the gate result for the supplied fit and for refits with the failing diagnostic, and gives the reach rule and what the noise band does and does not measure. The roxygen details describe the batch-means error, the pooling, the reach rule, the dispute flag, exact anchors, the finite-run offset and the gate.
…ntry The weekly block fits the bgmCompare anchor three times, refits at 1.6x twice, and checks that the anchor reaches 1.6x (ESS at least max(400, 10% of draws)) and does not reach 2.5x, and that the reweighted probability at 1.6x lies within max(0.02, 4 * noise) of the refits.
Collaborator
Author
|
Closed unmerged. Decision of 2026-10-28: the check becomes three full-length refits at 0.5x, 1x and 2x the chosen scale, classified on the fits' own Rao-Blackwellized verdicts, with no reweighted curve, no plot and no convergence gate. A new PR off develop replaces this one; the branch stays for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the reweighted curve of
prior_sensitivity_check()and the way the check classifies edges on it. Design record:dev/plans/active/2026-10-22_sensitivity-engine_DESIGN.md(untracked); reach study:dev/audit/checks/2026-10-reweighting-reach/.Commits
R/anchor_curve.R). Batch-means Monte Carlo variance of each reweighted inclusion probability, floored at the smoothed-count variance; inverse-variance pooling of the anchors that reach a point; an anchor contributes only if its importance ESS is at leastmax(ess_floor, ess_fraction * draws); a point where two anchors differ by Cochran's Q at 0.99 and by more than the move tolerance in log BF is disputed and takes the nearest anchor; at an anchor's own grid position the curve is that fit's Rao-Blackwellized probability.R/prior_sensitivity.R,R/refit_engine.R). A gate-failing refit keeps its verdict column, flagged in$grid$gate_flag, contributes to neither the curve nor the noise band, and the grid points nearest to it are not assessed; the original fit is treated the same way and theforcedcolumn is gone. New argumentess_fraction = 0.1. Per-point MCSE from the batch-means variance. Classification relative to the reported chosen-scale verdict over assessed points and every usable anchor's own verdict; new levelnot-fully-assessedwithcoverage,assessed_from,assessed_to,gap_cause. Sufficiency at every usable anchor (insufficient_any,insufficient_scales).Effect on the Wenchuan analysis (15 variables, 136 edges, 4 x 10000 draws)
Printed report on the tutorial's script (R/16), CRAN build -> this branch: robust 91 -> 55, changed within noise 6 -> 0, changed beyond noise 31 -> 5, not fully assessed 0 -> 4, not certifiable 8 -> 72.
$edges$moveron the branch: stable 78, moved-beyond-wobble 48, indistinguishable-from-wobble 4, not-fully-assessed 6;insufficientat the chosen scale 10. The tutorial's own mover count (R/36) goes 31 -> 40.The classification will move to the anchor fits in a follow-up PR (tutorial side decision of 2026-10-27); this PR ships the curve engine as designed.
Timing
The check on the Wenchuan fit takes 86 s on this branch against 82 s on the CRAN build, with the fit itself at 132 s against 129 s. The engine's own addition is the batch-means pass, about 5 s at this size. The 365 s reported earlier came from a review library built from unoptimized object files, not from the engine.
Verification
tests/testthat/test-anchor-curve.R,tests/testthat/test-prior-sensitivity.Rpass at tier T0/T1; the weekly block (tier T2) was run once on the branch.dev/review-2026-08/repro/run-engine.sh(five demos: pooling at a disputed point, curve at 1x against the fit's own probability, reach at 1.6x/2.5x/4x, printed report, gate-failing short fit); comparison in~/bgms-review/engine-comparison.txt. Develop against branch: a point with one anchor at 0 and one at 0.8 pools to 0.0025 on develop and is disputed on the branch (takes the nearest anchor, 0); the curve at 1x differs from the fit's own Rao-Blackwellized probability by up to 0.0154 on develop and by 1e-6 on the branch; on a bgmCompare anchor the 4x point (ESS 486, 6% of draws) passes the floor of 400 on develop and is masked on the branch, while 1.6x and 2.5x pass on both. Demo 5's short fit passed the gate on both sides, so the gate-failure path is covered by the unit tests only.