CI performance dashboard #1182

Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.

Headline (run 2026-10-30 08:17 UTC)

Window: slice 04:16 → 08:17 (24h numbers carried where the slice has no samples). Another quiet, healthy slice: 0 merges, 0 merge_group or push runs, 17 full PR runs, all green on the first attempt. Depot healthy: 0 lost hosts, 1 boot over 15 s.

Since the last run (04:16 → 08:17)

• Merge queue: idle since Fix stray files breaking vendored Gradle (#656) #1819 at 22:47 (9.5 h). No enqueues and no merge_group runs.
• PR feedback (created → ci-ok, 17 green full runs): p50 4.80 / p90 4.90 (4.55–5.23). No outliers. The 10-30 04:16 e2e_vendor_pypi_build slow-network leg didn't recur.
• Depot health: queue p50/p90 1 s / 1 s; boot max 26 s, 1 of ~411 Depot jobs over 15 s; 0 lost hosts. CI perf (settings): slow Depot 16-core hosts since 10-13 15:20 add ~1.5–2.5 min to the merge-queue critical path (~0.6k Linux job-min/day) #1512's 10-29 evening episode hasn't come back.
• Nightly (run) red on 2 test shards. This is a correctness failure, not a CI-time one:
  • test (macos-latest, 2) failed on e2e_golang_workspace_build (job), which took 157 s of a 353 s build. The ci-janitor PR Fix Go GOWORK test failing on macOS nightly #1894 (Go GOWORK on macOS) is open for it.
  • test (windows-latest, 1) also failed (job); Fix Maven home tests failing on Windows nightly #1852 (Maven home on Windows) is still open.
  • Cost: 99 L / 56 W / 44 M. macOS queue p50 4 s.
• CI: Let a dispatch run one OS only (#1818) #1820 still delivering: the one macOS-only workflow_dispatch (run, on agent/fix-ci-test-env-guard) cost 1.8 L / 0 W / 77 M, vs 194 L / 100 W / 83 M for a full-matrix dispatch before. It failed on that branch's own change (test (macos-latest, 2) and an e2e_redirect_npm_build macOS leg).
• Job-min in slice: 590 L / 56 W / 121 M. That's 17 full PR runs (33.5–35.4 L each) and 24 one-job stub runs, plus the nightly and the dispatch.
• No new issues; no comments. Nothing in the slice clears the filing bar, and there's nothing new on any open issue.

24h table

Merge-queue settings: build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH (ruleset read this run). Unchanged.

Open ci-perf issues by ROI

ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).

This run (08:17): filed none, commented on none, closed none.

Open ci-perf / ci-janitor PRs: #1852 Maven home tests on the Windows nightly (open), #1854 retry pinned bundler installs in the e2e legs (open), #1890 annotate failing tests in the macOS/Windows test legs (open), #1894 Go GOWORK test on the macOS nightly (open; it fixes this run's macOS nightly failure). Also open: ci-janitor issue #1660 (Depot hosts lost mid-job).

Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect

Method notes

• GraphQL and search/issues are blocked in this environment. Merge-queue state comes from REST timelines of merged PRs. "Enqueue → merge" uses the PR's first added_to_merge_queue event in the window. "Slot wait" = enqueue → the PR's first merge_group CI run created_at.

• Jobs are fetched with filter=all. Queue wait = job started_at − created_at; skipped jobs are excluded. "Slow start" = job started_at → first step started_at over 15 s, Depot jobs only.

• Critical path = merge_group run created_at → ci-ok completed_at of the latest attempt.

• "Discarded" merge_group runs are every CI run except each merged PR's last one.

• Hang cost (~330 entry-min) = for each entry held, its merge time minus when its own group run finished (or, for the Fix Deno global npm tools missed by -g (#444) #1719 hang, minus ~15:30, when the held groups' runs had finished).

• Run 10-26 20:16: 300 of 313 runs listed (3-page cap), and jobs for all 108 CI runs. "PR full runs" leave out 1-job stub runs (ci-ok in under 1 min).

• Runs 10-27 00:16, 04:16, 08:16 and 12:17: the queue was idle, so the 24h merge and merge_group numbers are carried from 20:16. Only the new 4-hour slice was fetched each time. The 12:17 run dropped the 10-24 12:16 history row (18-row cap).

• Run 10-27 16:17: 197 runs in the window (2 pages). Jobs fetched for every CI run since 12:17. The 24h merge numbers combine the 10-26 post-16:19 slice (carried) with today's 9 merges. The oldest history row (10-24 16:16) was dropped (18-row cap).

• Run 10-27 20:16: 157 runs in the window. Jobs were fetched for every CI run since 16:17, plus all merge_group/push CI runs in the window, to get ci-ok start gaps (started_at → first step). The oldest history row (10-24 20:16) was dropped (18-row cap).

• Run 10-28 00:16: 190 runs in the window. Jobs were fetched for every non-stub CI run since 20:16. The 24h enqueue→merge p50 combines the 9 carried 13:07–14:58 values (5.2–8.5, p50 6.1) with Build(deps): Bump zod from 3.25.76 to 4.6.6 in /npm/socket-patch #1702, Build(deps): Bump rustls from 0.23.45 to 0.23.46 #1744 and fix: vendor hosted patches directly from lockfiles #1648 (last enqueue); it's approximate. The oldest history row (10-25 00:16) was dropped (18-row cap).

• Run 10-28 04:16: 227 runs in the window (3 pages). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 11 completed full PR CI runs since 00:16. The oldest history row (10-25 04:16) was dropped (18-row cap).

• Run 10-28 08:17: 283 runs in the window (3 pages). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 24 non-stub CI runs since 04:16. The oldest history row (10-25 08:15) was dropped (18-row cap).

• Run 10-28 12:17: 288 runs in the window (3 pages). No merge_group or push CI runs since 10-27 21:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 17 CI runs since 08:17. The oldest history row (10-25 12:15) was dropped (18-row cap).

• Run 10-28 16:18: 300 of 369 runs listed (3-page cap; the list reaches back to 10-27 22:41, so the 10-27 20:05–21:16 merges are carried). Jobs were fetched for all 39 non-stub CI runs since 12:17 (20 mg, 11 push, 8 PR) and timelines for 15 of the 20 merged PRs (~60 REST requests). Enqueue → merge p50/p90 is over those 15. The oldest history row (10-25 16:15) was dropped (18-row cap).

• Run 10-28 20:16: 300 of 417 runs listed (3-page cap, back to 04:51). Jobs were fetched for all 20 non-stub CI runs since 16:18 and timelines for Keep Go hosted pins after go mod tidy drops the zip line #1748 and Read Go workspaces the way go builds them #1763. The CI perf: release-readiness — 10-second gate waits up to 5 min for a Depot boot in front of coverage and e2e (~1–3 min per run in slow-start periods) #1795 replay keeps each run's measured clippy end and per-job boots and puts release-readiness at its own work time + 6 s; jobs needing clippy too move only as far as clippy allows. The oldest history row (10-25 20:15) was dropped (18-row cap).

• Run 10-29 00:16: 300 of 463 runs listed (3-page cap, back to 10-28 13:44, so all 32 merge_group runs since the 13:56 burst are in it). Jobs were fetched for all 31 non-stub CI runs since 20:16, timelines for the 10 PRs merged since 20:16, and 1 job log. GraphQL is still blocked (403). ~50 REST requests. The oldest history row (10-26 00:16) was dropped (18-row cap).

• Watch item, not filed: one uv/composer e2e leg on a PR (job) spent 18.7 min in Setup Python (3.12.10 isn't in the Depot tool cache, so every leg downloads it in 5–8 s; this one hit getaddrinfo EAI_AGAIN raw.githubusercontent.com at 3.5 min and then hung until 19.3 min). It cost the PR ~20 min and a 24-min re-run. That's 1 of 200 Setup Python steps in 4 h. A step timeout-minutes plus a retry, like the rust-cache cap (Cap hung rust-cache restores so they cannot stall the merge queue #1730), would bound it. It'll be filed if it recurs.

• Not filed: coverage shard balance. coverage (2)'s test step still runs 241–264 s vs 161–179 s for shard 1 on fast hosts, but the Gradle e2e leg ends at ~4.9–5.0 min, so balancing gives at most ~0.3–0.5 min on fast hosts. Balance coverage shards on measured test time #1768's retry with a Linux timing sample stays a low-ROI option.

• Run 10-29 04:16: 300 runs listed (3-page cap, back to 10-28 14:06). No merge_group or push CI runs since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 12 non-stub CI runs since 00:16 (~25 REST requests, no job logs). The oldest history row (10-26 04:16) was dropped (18-row cap).

• Watch item, not filed: one e2e-build took 2.67 min (run), vs 1.12–1.40 in the other 11 runs, and it pushed that run's ci-ok to 6.48. It'll be filed if it recurs.

• Run 10-29 08:17: 300 of 448 runs listed (3-page cap, back to 10-28 14:50). No merge_group or push CI runs since 10-28 22:56, so all 24h merge-queue numbers are carried. Jobs were fetched for the 13 non-stub CI runs since 04:16, plus one 7-day workflow_dispatch listing for CI perf: workflow_dispatch of CI — Windows-only janitor fixes re-run the full Linux + macOS nightly matrix (~100 min/day, ~30 macOS) #1818 (~25 REST requests, no job logs). GraphQL is still blocked (403). The oldest history row (10-26 08:18) was dropped (18-row cap).

• Run 10-29 12:22: 300 of 463 runs listed (3-page cap, back to 10-28 17:35) plus 2 pages of merge_group runs (all 128 in the window). No merge_group or push CI runs since 10-28 22:56, so the 24h merge-queue latency numbers are carried. Jobs were fetched for the 7 full PR CI runs since 08:17 (~20 REST requests, no job logs). GraphQL is still blocked (403). The oldest history row (10-26 12:33) was dropped (18-row cap).

• Run 10-29 16:30: 300 of 500 runs listed (3-page cap, back to 10-29 02:26) plus 1 page of merge_group runs (all 43 CI runs in the window). Jobs were fetched for all 61 CI runs since 12:22 and timelines for the 21 PRs merged since (~95 REST requests, 1 job log). GraphQL is still blocked (403). The 24h job-min estimate sums the last 6 slice totals. "Slot wait" excludes Fix Maven reactor VEX over a base-version module (#584) #1802/Fix vendored gems on CRLF Gemfile.lock (#1808) #1811, which were dropped at 13:40 without a group run. The oldest history row (10-26 16:19) was dropped (18-row cap). Merged-PR rows from 10-22 aged out (7-day window).

• Run 10-29 20:20: 300 of 488 runs listed (3-page cap, back to 10-29 07:09). 25 CI runs since 16:30 (21 PR, 2 mg in flight, 2 dispatch). Jobs fetched for the 4 green full PR runs, the lost-host PR run, the dispatch and the in-flight Retry CI bundler installs on rubygems.org blips #1828 group (~15 REST requests, 1 job log). GraphQL is still blocked (403). No merges in the slice, so merge latency is carried. A PR coverage (2) job spent 1,122 s in Run tests with coverage (vs 255 s); single PR (Read the cargo config chain once (#1832) #1849, cargo config chain), not filed. The oldest history row (10-26 20:16) was dropped (18-row cap).

• Run 10-30 00:17: 300 of 469 runs listed (3-page cap, back to 10-29 13:20). Jobs were fetched for all 23 non-stub CI runs since 20:00 (filter=all), plus timelines for Retry CI bundler installs on rubygems.org blips #1828 and Fix stray files breaking vendored Gradle (#656) #1819 and the ruleset (~30 REST requests, no job logs). GraphQL is still blocked (403). "Lost host" = a job on a depot-* runner with 0 steps. Its minutes are started_at → completed_at. The outage excess compares 15 runs against ~36 L per healthy run. The oldest history row (10-27 00:16) was dropped (18-row cap). Merged-PR rows from before 10-23 00:17 aged out (7-day window).

• Run 10-30 04:16: 72 runs in the window since 00:00 (1 page). No merge_group, push-to-main or schedule runs since 00:17, so all merge-queue numbers are carried. Jobs were fetched for the 12 full PR CI runs (filter=all), plus the ruleset, 2 issue listings and 1 job log (~20 REST requests). The oldest history row (10-27 04:16) was dropped (18-row cap).

• Run 10-30 08:17: 89 runs in the window since 04:10 (1 page). No merge_group, push-to-main or merge runs since 22:47, so all merge-queue numbers are carried. Jobs were fetched for all 43 CI runs (filter=all), plus the ruleset, 3 issue listings and 1 job log (~50 REST requests). The oldest history row (10-27 08:16) was dropped (18-row cap).

Generated by Claude Code