Repository navigation
ci: one required check, ci, in under ten minutes - #1608
Merged
Merged
Conversation
pytest 9.2 floated into flow-unit through a fresh resolve and failed every pull request. Jobs now install from uv.lock files, test tools included: - scripts/ci/locked_install.py exports each named lock with `uv export --locked` and installs it with --require-hashes --no-deps, then the in-tree packages editable with --no-deps, then runs `uv pip check`. The types, agent, flow-lint, flow-unit, flow-critical, native census and launcher run-ci jobs use it. - .github/ci-tools holds a locked tools project (pytest, packaging, build, pip, ruff, pip-audit, twine) for the Python-support, platform-manifest, quickstart, Desktop audit and release build jobs. - `uv run` in the Desktop workflows passes --locked. - tests/test_ci_locked_installs.py fails on any workflow install that resolves at run time, with the deliberate exceptions listed. - Dependabot: one uv entry for every lock except Desktop's, with a weekly test-tools group and a weekly group for everything else. Lock fixes the locked installs needed: Agent's lock was stale (Flow 2.1.0, Types 0.18.0); the launcher's click moved to 8.5.0, which the CLI's "Did you mean" test relies on; Flow's dev extra adds uiautomation on Windows for the real-app census, which installed it ad hoc before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…issue Pull requests and the merge queue run only the packages a change touches, and the merge queue's push to main runs only the fast lane, so a package nobody touched can break on main unseen. nightly.yml runs daily and on dispatch. It calls Packages (which calls the Desktop test and build workflows) and the launcher CI as a scheduled run, the tier that selects every package and every platform leg. Its last job runs scripts/ci/nightly_report.py, which opens a single "main is red" issue (label main-is-red) listing the failing jobs, updates it if one is open, and closes it when a full run passes. packages.yml and main.yml gain a workflow_call trigger. No required check and nothing on the pull-request lane changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Windows runner's Python ships pipx, whose packaging floor the Flow lock does not meet, and Privacy's lock overrides Presidio's cryptography cap on purpose. uv pip check flagged both. The helper now fails only when a package it installed has an unmet requirement and reports the rest. The CI tools lock takes filelock 4.2.4 for GHSA-v4x8-3wr5-3wq8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… claude/ci-single-required-check
The CI tools lock resolved filelock 4.1.0 for Python 3.10, because the releases that fix GHSA-v4x8-3wr5-3wq8 need 3.11, and dependency review refused it. Only pip-audit pulls filelock (through cachecontrol), and the Desktop audit runs it on 3.12, so it moves to .github/ci-tools/audit with requires-python >=3.11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…claude/ci-single-required-check
ci.yml runs on pull requests and in the merge queue and is the one check a pull request needs, with gitleaks and dependency-review. It calls packages.yml and main.yml on their pull-request tier, which no longer trigger on those events themselves, adds a smoke test of every package, and reports `ci` from all of them. Until branch protection names `ci`, the old required names (`run-ci (ubuntu-latest, 3.12)`, `packages-gate`) report its result. The lane fits by moving slow tests, not dropping them: - Flow's PR-tier unit shards leave out tests/nightly_only.txt: 131 tests over about 5 s on CI (2,127 of 3,993 test-seconds, from a JUnit profile on ubuntu-latest). The realism tests that guard safety stay. tests/lanes.py deselects them under OPENADAPT_TEST_LANE=fast; main and nightly run all. - flow-critical runs its golden, record/compile/replay and critical tests as one xdist step instead of three serial ones (about 13 minutes serially). - The native row census (macOS and Windows) moves to the post-merge tier. - scripts/ci/smoke.py installs every package from its lock, imports it and runs its command with --help, whatever changed. scripts/ci/measure_pr_ci.py measures wall time and job-minutes per merged pull request; docs/monorepo/CI_TIMING.md has the before numbers (median 33 minutes wall, 63 job-minutes over the last 25 PRs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr
enabled auto-merge
October 28, 2026 21:23
…e tray headless - The repository ignores *.txt, so packages/flow/tests/nightly_only.txt never reached CI and every Flow unit shard failed at collection. .gitignore now keeps it. - flow-critical runs tests/e2e and its unit files in two xdist steps. In one run with both, test_structural_action lost its conftest's bundle fixture. - The smoke job sets the tray's dummy input and icon backends, as the tray job does: a headless runner has no X display.
This was referenced Oct 29, 2026
abrichr
added a commit
that referenced
this pull request
Oct 29, 2026
… lands (#1622) * fix(flow): record a held save's result, not the screen before it lands The recorder took a click's after-frame once the screen settled, and its system-of-record read with it. While the agent guard holds a save for its checks the screen holds still too, so on a loaded runner the frame settled and the record was read before the write landed. The save step then showed no new record, compile emitted a placeholder effect, and admission refused the taught program: "no canary ran: a write step lacks an independent read-back: step_005: record_written is placeholder". It failed tests/e2e/test_agent_guard_e2e.py on a 4-core CI runner and passed locally and on main's own run. Holding the save 2 s in the guard's process reproduces it exactly (empty sor_after on the save event, the same admission error). NetworkCapture now counts writes in flight (any method but GET, HEAD or OPTIONS, from the browser's request to its finish or failure), and the recorder waits for them, up to 30 s and pumping Playwright, before a click's after-frame and before each record read. tests/e2e/test_agent_guard_slow_hold_e2e.py holds every guarded save 2 s and requires the recording to show the write and the program to be admitted; it fails without this change. tests/test_recorder_waits_for_writes.py covers the in-flight count and the bounded wait. * ci: lock installable on Python 3.10, check every lock's oldest Python, four PR shards Main went red at 0feec0f: agent shim (py3.10) and Python CI (3.10) could not install onnxruntime 1.24.3, which publishes no cp310 wheel and no sdist, yet the root, Agent and Flow locks pinned it for Python 3.10. The pull-request lane runs 3.12, so only main's 3.10 jobs saw it. - The root, Agent and Flow projects set uv's required-environments to Python 3.10 on Linux, macOS and Windows. Their locks now pin onnxruntime 1.23.2 for 3.10 and keep 1.24.3 and 1.27.0 for 3.11 and up. - scripts/ci/check_lock_wheels.py resolves every lock's pins for its requires-python floor on the runners' platforms (no install). The ci smoke job runs it, so a lock like this fails the pull request. It fails on main's old locks with the onnxruntime error and passes on all ten now. - Four Flow unit shards on pull requests, six benchmark files to nightly whole (this commit missed #1608's merge). - The nightly run's Flow shards take all of tests/e2e; flow-critical also runs the new slow-hold guard test. * fix(flow): wait only for this recording's writes, briefly, and never past the next action An adversarial review of the in-flight write wait found three ways it could stall a recording attached to a person's browser, where the context holds every tab: - Another tab's long-poll POST counted as pending, so each event waited the full 30 s, and a click waited twice. Only writes from the pages the recording follows count now. - A request whose finish Playwright never reports (a beacon from a popup that closes) stayed pending for the rest of the recording. The wait covers only the writes in flight when it starts, each for at most 30 s since it was sent. - An action that arrived during the wait made the recording abort. The wait ends as soon as the next action is queued. The click's after-frame now settles first (its screenshots deliver the save's request event) and settles again once a write it waited for has landed, so the frame and the record read agree. The wait runs on the start tab, as the pump does, because a followed popup may close by itself. None of the review's findings recorded a wrong contract that looks verified: each failed closed or slowed the recording. The new unit tests cover each one; both agent guard e2e tests pass, the slow-hold one included. * fix(flow): rebuild uv.lock on main's (2.4.1, grounding rung) with the 3.10 environments Main changed packages/flow/uv.lock (#1596, #1620) after this branch did, and git merged the two edits without a conflict into a lock uv cannot parse: "Dependency onnxruntime has missing source field but has more than one matching package". This takes main's lock and relocks it with the merged pyproject, so it gains only the Python 3.10 fork (onnxruntime 1.23.2). * test(flow): check the slow-hold run's compiled recording, and say why it is missing In CI the guard's run-1 recording was not at run-ok/recording when the test read it, though the run was clean and the save held 2 s. The test now reads the recording the handoff kept and compiled (<task_dir>/program/recording), which is the evidence that matters, and fails with the guard's notes, error and handoff when it is missing. PYTHONPATH is prepended to, not replaced. * ci: run the slow-hold guard test nightly, not in flow-critical On CI the slow-hold test's run 1 was refused by the recorder ("more than one logical browser action arrived before Flow could retain an exact intermediate frame"), which passes locally and fails closed. It stays in tests/e2e, which the nightly run covers, while that is pinned down separately; flow-critical keeps test_agent_guard_e2e, which passes with the recorder fix.
abrichr
added a commit
that referenced
this pull request
Oct 29, 2026
…guards back on the PR lane (#1632) Main's Packages run fails tests/test_cli_hints.py::test_printed_hints_never_hard_code_the_engine_spelling on registry.py's pull hint (#1623), which printed a literal "openadapt-flow skill ...". Under the launcher that names a program the person never typed; it now goes through cli_hints.command() like every other hint. #1623's own CI never ran the test: #1608 moved it to nightly as a 6 s test, though it is a source scan that guards every printed hint. It returns to the pull-request lane, with five safety guards the same rule moved by mistake: identity (a numeric MRN homonym never verifies), durable resume (never resends), the safety gallery (every dangerous case is safe), halt detail (a typed value never appears) and self-repair (a healthy run never calls the agent). Together they add about 90 test-seconds across the four shards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One required check,
ci, for every pull request. It covers lint, the min-Python syntax check, the Linux tests of the changed packages, and a smoke test of every package. Slow and cross-platform work moves to main and the nightly run.This is built on #1602 (lockfiles) and #1603 (nightly); until they merge, this diff includes them. Branch protection is unchanged here: the switch command is below.
What changes
ci.ymlruns onpull_requestandmerge_group.packages.ymlandmain.ymlon their pull-request tier. Those two no longer trigger on pull requests or the queue themselves, so each job runs once.smoke:scripts/ci/smoke.pyinstalls every package from its lock, imports it and runs its command with--help, whatever changed. Locally that takes 21 s for 8 packages.cijob fails unless Packages, the launcher CI and smoke all passed. Inside Packages,packages-gatestill requires every selected job to run.ci, ci.yml also reportsrun-ci (ubuntu-latest, 3.12)andpackages-gatewithci's result. CodeQL keeps reportingAnalyze (python).OPENADAPT_TEST_LANE=fast.packages/flow/tests/lanes.pythen leaves out the tests intests/nightly_only.txt; main and nightly run everything.tests/test_lanes.pychecks that every entry still exists.-n auto --dist loadfilestep. It used to be three serial steps (about 13 min) and keeps its no-skip check.flow-native-census(macOS and Windows runners) moves to the post-merge tier, with main and nightly. The PR lane is Linux.How the slow tests were chosen
I took a JUnit profile of the whole Flow suite on
ubuntu-latestwith 4 xdist workers (branchclaude/profile-test-durations).test_realism_browser,test_realism_repeated_rows,test_multi_surface_identity, the context and form identity files.test_effect_fault_matrix,test_lending_swer,test_ui_readback_browser,test_readback_*.test_injection_suite,test_injection_paths, ande2e/test_agent_guard_e2ein flow-critical.test_undo_proposal,test_after_stop_mockmed,test_realism_durable_resume.test_agent_guard.The full
tests/e2edirectory, which CI never ran before, now runs on main and nightly. It has two failing tests the nightly run will report:e2e/test_tutorial_break_it_e2e.py::test_the_demo_proof_page_reads_this_pair: 4 choices where 3 are expected.e2e/test_chaos.py::TestLayoutSwapMidRun::test_swapped_type_buttons_still_select_the_right_type.Before and after
Before, over the last 25 merged PRs (
docs/monorepo/ci-timing/before-2026-10-28.json): median PR wall time 33.2 min (p90 70.5), median 63 job-minutes. The long poles were Flow's unit shards (16 to 22 min each) and flow-critical (14 min). After: this PR's own runs, which I'll add here once they finish.Switching branch protection (admin)
After that, remove the
legacy-run-ciandlegacy-packages-gatejobs from ci.yml. The merge-queue ruleset (25486718) is disabled; if it's turned back on, its required checks need the same change.Checked locally
tests/test_ci_required_check.py,test_nightly.py,test_ci_locked_installs.pyandtest_packages_concurrency.py(31 passed). Flow'stest_lanes.pypasses.🤖 Generated with Claude Code