Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-15 16:16 UTC)
Window: 2026-10-14 16:16 → 2026-10-15 16:16 (run list reaches back to 19:49). 24 PRs merged. All 20 merge_group CI runs succeeded: 0 evictions, 0 failures, 0 cancellations, 0 re-runs. 0 non-success jobs in the 1,710 sampled.
- The queue moved again. A 6-PR burst was enqueued at 14:12:59–14:15:26 and merged by 14:36, and 2 more PRs merged at 16:00. The 6th entry, Fix uv project shadowing parent uv settings (#1545) #1566, waited 5.8 min for a build slot (
max_entries_to_build= 5), so it took 15.22 min from enqueue to merge, against 8.13–9.05 for the other five. Numbers posted on #1248. - Slow Depot hosts came back on the first busy afternoon. Slow-start share of Depot Linux jobs: 11h 0% (0/29), 14h 11% (33/306), 15h 81% (48/59), 16h 26% (9/35). The two 15:49 merge_group runs took 10.27 / 10.78 min to
ci-ok(coverage8.17 / 8.28 min, start-up gaps 23–27 s), against 7.43–8.98 for the five 14:13–14:21 runs. #1512 stays open. It already got its comment for 10-15 (00:19), so this relapse is recorded only here; the 20:16 run will add a comment if it continues. - CI: fold last top-level covgap test binaries (#1089) #1568 merged at 14:21. Its effect on
coveragecan't be measured yet: only 2 fast-host runs since, at 5.75 and 6.25 min, against 5.82–6.03 before. - No new issues. Nothing closed. Nothing new cleared the bar (see "Other items watched").
Sample: 300 of 343 runs in 24h (3-page cap reaches back to 10-14 19:49). Job detail (filter=all, first page) for 82 runs: all 20 CI merge_group runs, all 14 push runs, the nightly and the 6 CI dispatches (first 100 of 246 jobs each; their full cost is carried from earlier runs), every third CI pull_request run (33), and 2 runs each of Audit GHA Workflows, Merge queue janitor, Merge queue fail-fast and bughunt ledger. 1,710 non-skipped jobs. Timelines for the 15 most recent merges. No job logs read. About 150 REST requests.
| metric | 24h |
|---|---|
| PRs merged | 24 (bursts at 00:56, 02:57, 05:58, 14:21–14:36, 16:00) |
| Merge queue enqueue → merge | p50 8.82 / p90 11.83 min (n=15, range 8.13–15.22). • #1566: 15.22 (5.8 min waiting for a build slot) • #1575 / #1577 (slow hosts): 11.78 / 11.83 0 evictions; every entry enqueued once. |
merge_group created → ci-ok |
p50 8.17 / p90 10.62 (n=20, range 7.43–11.33). Last job before ci-ok: coverage in 13 runs, Gradle e2e leg in 7. |
PR run created → ci-ok |
p50 7.93 / p90 9.93 (n=8 full runs). 24 of 32 sampled runs were ci-ok-only stubs. |
push to main created → ci-ok |
p50 3.52 / p90 5.32 (n=14, 9.1 job-min per run) |
| Nightly | 10-15 05:46, success. |
| Linux queue wait p50 / p90 / max | Depot 0.02 / 0.02 / 0.18 (n=1,633) |
| macOS queue wait | 0.13 / 0.17 (nightly + dispatch only, n=28). No macOS job on PR, merge_group or push. |
| Windows queue wait p50 / p90 / max | 0.03 / 0.05 / 0.07 (n=49) |
| Job-min/day (Linux / Windows / macOS) | ~3.8k / 0.62k / 0.43k. By event (scaled to 24h from the 20.4h run list): • merge_group: ~0.85k L + 0.07k W (36.8 L / 3.2 W per run, ~23 runs/day) • CI pull_request: ~1.09k (9.5 per run incl. stubs, ~115 runs/day) • push: ~0.15k • nightly: ~0.24k / 0.08k / 0.06k • 6 CI dispatches: ~1.44k / 0.47k / 0.37k Not counted above: Merge queue fail-fast watcher ~0.22k (10.9 min per merge_group run, GitHub-hosted ubuntu-latest; it only waits), Audit GHA Workflows ~0.02k. |
Top jobs by sampled minutes (PR + merge_group + push): coverage 215 min (p50 5.90 / p90 8.17, n=42), Gradle e2e leg 135 (4.70 / 5.10), e2e_vex_build pip 81 (2.73), e2e_redirect_npm_build 77 (2.60), hosted-e2e 76 (1.53 / 2.63), clippy 67 (1.43 / 2.17), windows-compile-check 64 (2.82 / 4.45), e2e-build 61 (1.20), yarn-classic 50, yarn-berry 49, node-addon 44.
Top steps: Run e2e tests 355 min (492 jobs), Run tests with coverage 169 (42, p50 5.47), Type-check the CLI and every CLI test target 48 (20), Checkout 47 (942), Run hosted-mode production e2e 47, Install Rust 40 (321), Compile the CLI and every CLI test target 37 (43), Check every test target before starting expensive jobs 32, Set up job 32.
Critical path (unchanged): clippy → coverage → ci-ok, in parallel with e2e-build → Gradle e2e leg → ci-ok; on fast hosts they finish within about 1 min of each other.
Other items watched, not filed (below the bar): full-matrix CI dispatches (a manual choice); Install Rust (~0.04k/day across 321 sampled jobs); sharding coverage or splitting the Gradle leg (~1 min, and the two paths are already balanced); windows-compile-check p90 4.45 (off the critical path, Windows ~0.06k/day); the hosted fail-fast watcher (waits on free GitHub-hosted Linux, no Depot or macOS cost).
Merge-queue settings (ruleset 24668460, read 16:20): unchanged: build 5 / merge 5, min wait 5 min, ALLGREEN, timeout 90, SQUASH.
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-15 16:16 | 24 PRs merged (20 mg CI successes since 19:49; 0 failures, 0 evictions) | 8.82 / 11.83 (n=15, range 8.13–15.22; #1566 waited 5.8 min for a build slot behind a 6-entry burst) | 8.17 (n=20, p90 10.62; 14:13–14:27 runs 7.43–9.75; 15:49 runs 10.27–10.78 on slow hosts) | 7.93 / 9.93 (n=8 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,633) | ~3.8k / 0.62k / 0.43k (incl. 6 CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 36.8 L / 3.2 W per run; Depot slow-start 11% at 14h, 81% at 15h, 26% at 16h: #1512 relapse on the first busy afternoon) |
| 2026-10-15 12:16 | 19 PRs merged (17 mg CI successes since 13:21; 0 failures, 0 evictions; none since 05:58) | 9.05 / 12.22 (n=15; since 23:48 8.07–9.75; 16:48–20:48 11.37–12.42) | 8.33 (n=17, p90 11.33; the 10 runs since 23:48 7.52–9.03) | 7.93 / 10.18 (n=7 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,470) | ~3.35k / 0.60k / 0.43k (incl. 6 CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 38.9 L / 2.7 W per run; Depot slow-start 1% at 06h, 0% at 11h; quiet morning, no busy-day confirmation for #1512 yet) |
| 2026-10-15 08:16 | 23 PRs merged (19 mg CI successes since 12:24; 0 failures, 0 evictions) | 9.05 / 12.2 (n=15; the 9 entries since 23:48 took 8.07–9.75; 16:48–20:48 entries 11.37–12.42) | 8.55 (n=19, p90 11.38; the 9 runs since 23:48 7.52–9.03) | 9.68 / 12.13 (n=11 full runs) | 0.13 / 0.17 (nightly + 6 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,647) | ~4.45k / 0.61k / 0.43k (incl. 6 full-matrix CI dispatches ≈ 1.44k / 0.47k / 0.37k; mg 38.6 L / 2.8 W per run; Depot slow-start 0% in 06h–08h, 05h hits were the nightly; nightly green → #1525 delivered) |
| 2026-10-15 04:16 | 26 PRs merged (21 mg CI successes since 08:48; 0 failures, 0 evictions) | 9.75 / 12.21 (n=15; the 7 entries since 23:48 took 8.06–9.75; 16:48–20:48 entries 11.36–12.41) | 8.55 (n=21, p90 11.6; the 6 runs since 23:48 7.52–9.03) | 9.65 / 11.37 (n=18 full runs) | 0.13 / 0.17 (4 CI dispatches only) | 0.02 / 0.02 (Depot, n=1,664) | ~4.1k / 0.38k / 0.25k (incl. 4 full-matrix CI dispatches ≈ 0.96k / 0.31k / 0.25k; mg 40.0 L / 2.7 W per run; Depot slow-start 4% / 3% / 17% / 0% in 00h–03h, down from 58% at 23h) |
| 2026-10-15 00:16 | 25 PRs merged (21 mg CI successes since 04:49; 0 failures, 0 evictions) | 10.27 / 12.22 (n=15; fast hosts 7.75–8.88; the 6 entries 16:48–20:48 took 11.4–12.4) | 8.15 (n=21, p90 11.6; fast 7.3–8.15; slow 9.3–12.9; 23:48 back to 7.5–7.6) | 9.65 / 11.37 (n=16 full runs; 7.3–8.2 with no slow host) | 0.12 / 0.17 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=1,553) | ~3.2k / 0.19k / 0.09k (incl. 2 CI dispatches; mg 38.9 L / 2.7 W per run; Depot slow-start 61% 16:16–20:16, 37% 20:16–00:10; coverage 8.3 vs 5.96 min on slow vs fast hosts) |
| 2026-10-14 20:16 | 30 PRs merged (23 mg CI successes since 23:49; 0 failures, 0 evictions) | 8.93 / 12.22 (n=15; 7.77–8.93 on fast hosts; all 5 entries since 16:48 took 11.4–12.4) | 8.15 (n=23, p90 11.6; fast hosts 7.1–8.55; slow 9.3–12.9; all 5 since 16:48 10.6–11.75) | 8.23 / 10.83 (n=13 full runs) | 0.12 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=1,346) | ~2.9k / 0.15k / 0.06k (mg 39.0 L / 2.6 W per run; Depot slow-start 59% of Linux jobs since 16:16 (4-core 59%, 16-core 68%); slow hosts also run coverage 8.2–9.9 min vs p50 5.85) |
| 2026-10-14 16:16 | 31 PRs merged (25 mg CI successes since 22:49; 0 failures, 0 evictions) | 8.73 / 11.4 (n=19; 7.77–8.93 except 6 slow-host entries 10.0–13.4) | 8.55 (n=25, p90 10.73; fast hosts 7.80 vs slow 9.64) | 7.92 / 11.03 (n=17 full runs) | 0.12 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=1,509) | ~3.3k / 0.15k / 0.06k (mg 37.0 L / 2.6 W per run; Depot slow-start back: 16-core 60% at 21–01h, 30% and 4-core 50% at 12–16h → #1512 reopened) |
| 2026-10-14 12:17 | 53 PRs merged (54 mg CI successes 12:17–11:49, 1 failure, 10 cancelled; 4 since 08:16) | 8.58 / 11.4 (n=15; 7.8–8.9 except #1525 13.4 on a slow host) | 8.52 (n=54, p90 11.0; 7.3–8.6 since 09:49, 12.9 at 08:48) | 8.23 / 10.52 (n=15 full runs; 7.65 / 8.63 since 08:16, n=8) | 0.12 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=2,605) | ~4.5k / 0.24k / 0.06k (mg 37.6 L / 2.6 W per run; Depot slow-start blips at 08:xx 91% and 12:xx 61%, 0–2% in 10:xx–11:xx) |
| 2026-10-14 08:16 | 47 PRs merged (50 mg CI successes 13:11–07:49, 1 failure, 10 cancelled) | 8.75 / 11.4 (n=15; 7.8–8.9 since 01:48) | 8.55 (n=50, p90 11.0; 7.3–7.9 since 04:16, n=5) | 9.63 / 12.72 (n=9 full runs, created→ci-ok) | 0.12 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=2,296) | ~4.9k / 0.23k / 0.06k (mg 37.5 L / 2.6 W per run, ~33 L since 04:16; Depot slow-start 3%/1% → #1512 closed; nightly failed on old-uv legs) |
| 2026-10-14 04:16 | 44 PRs merged (45 mg CI successes 13:11–03:49, 1 failure, 10 cancelled) | 10.1 / 11.5 (n=15; 8.2–8.9 since 01:48) | 8.6 (n=45, p90 11.0; 7.1–8.6 since 01:48, n=4) | 9.85 / 12.12 (n=18 full runs, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=2,300) | ~5.5k / 0.22k / 0.064k (mg 38.0 L / 2.6 W per run; 33 at 03:49; Depot slow-start share fell to 13–15% after 01:00) |
| 2026-10-14 00:16 | ~41 (41 mg CI successes 13:11–23:49; 40 PRs merged since 00:16; 0 mg runs 00:16–13:11) | 11.3 / 12.0 (n=15; 9.8–12.7) | 8.7 (n=41, p90 11.0; 9.5 since 20:16, n=8) | 9.92 / 12.08 (n=19 full runs, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=2,188) | ~5.8k / 0.21k / 0.064k (mg 36.7 L / 2.6 W per run; 16-core hosts still slow, 4-core recovered at 22h) |
| 2026-10-13 20:16 | ~39 (39 mg CI successes, 1 failure, 10 cancelled) | 11.4 / 15.6 (n=15; 9.7–12.7 outside the 13:5x eviction) | 8.6 (n=39; 10.8 since 15:49, n=8) | 10.63 / 12.07 (n=13 full runs since 16:20, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=1,980) | ~4.7k / 0.21k / 0.064k (mg per run 36 → 52 L job-min after 15:20: slow Depot hosts) |
| 2026-10-13 16:17 | ~39 (39 mg CI successes; 14-entry burst 13:11–13:35; 1 flake eviction 13:45 → 10 cancelled rebuilds) | 11.6 / 23.7 (n=18; burst entries 6–14 10.4–18.1, eviction victims 23.7–25.3) | 8.38 (n=27, p90 10.83) | 8.02 / 11.67 (n=14 full runs, created→ci-ok) | none ran (no macOS on PR/mg/push) | 0.02 / 0.02 (Depot, n=1,620) | ~4.8k / 0.21k / 0.064k (mg 36.4 / 2.6 / 0 per run + 340 L job-min in cancelled rebuilds) |
| 2026-10-13 12:16 | ~33 (37 mg runs, carried; 0 merges since 22:57) | 8.8 / 10.5 (n=15, carried) | 8.22 (carried) | 7.57 / 7.87 (n=5 fresh full runs, created→ci-ok) | 0.13 / 0.17 (nightly only, carried) | 0.02 / 0.02 (Depot, n=145) | ~3.9k / 0.18k / 0.064k (Release dispatches mostly aged out; mg 35.4 / 2.6 / 0 per run) |
| 2026-10-13 08:16 | ~33 (37 mg runs, carried; 0 merges since 22:57) | 8.8 / 10.5 (n=15, carried) | 8.22 (n=33, carried) | 7.6 / 9.0 (n=6 fresh full runs, created→ci-ok) | 0.13 / 0.17 (nightly only) | 0.02 / 0.02 (Depot) | ~4.1k / 0.25k / 0.10k (incl. Release dispatches; ~3.95k / 0.18k / 0.064k without; nightly green, 244 / 82 / 64) |
| 2026-10-13 04:16 | ~33 (37 mg runs; 0 merges since 22:57; 1 eviction, carried) | 8.8 / 10.5 (n=15, carried) | 8.22 (n=33, p90 9.21) | 8.02 / 10.04 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly + Release only) | 0.02 / 0.02 (Depot, n=1,828) | ~4.1k / 0.22k / 0.07k (incl. 3 Release dispatches; ~3.95k / 0.15k / 0.035k without; mg 35.4 / 2.6 / 0 per run; push ~8.7 per run after #1415) |
| 2026-10-13 00:16 | 35 (40 mg runs; 1 eviction, carried; release 5.0.0) | 8.8 / 10.5 (n=15, max 10.5, 0 evictions) | 8.15 (n=36, p90 9.2) | 8.43 / 9.95 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,309) | ~5.4k / 0.35k / 0.16k (incl. 3 CI + 3 Release dispatches; ~4.9k / 0.16k / 0.03k without; mg 35.1 / 2.7 / 0 per run; push ~8.7 per run) |
| 2026-10-12 20:16 | 36 (42 mg runs; 15:44 burst drained by 16:19; 1 eviction) | 9.7 / 28.8 (n=15; burst entries 27.6–31.9, others 7.9–10.4) | 7.93 (n=38, p90 9.03; 8.43 since 16:16) | 7.67 / 10.22 (n=8 full runs, created→ci-ok) | 0.12 / 0.17 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,534) | ~4.4k / 0.32k / 0.14k (incl. 3 CI + 2 Release dispatches; ~4.05k / 0.17k / 0.03k without; mg 33.9 / 2.9 / 0 per run; push now ~8 per run) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~2.5 | #1512 (settings) slow Depot hosts | ~2–3 min on each merge that hits a slow host (2 of 15 sampled merges in 24h, both at 15:49), plus ~2 min on PR feedback | Relapsed at 15h (81% slow-start, 48/59 jobs) on the first busy afternoon since it reopened. Needs a human to raise it with Depot. |
| ~1.0 | #1248 (settings) merge-queue build concurrency | ~6 min for each entry past the 5th in a burst | Needs a human decision. Hit once today: #1566 waited 5.8 min for a build slot (comment). |
Open ci-perf / ci-janitor PRs: none.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
| #1568 fold last top-level covgap test binaries (#1089) | 10-15 14:21 | coverage on fast hosts (start-up gap ≤ 8 s): 5.82–6.03 min (n=8 mg runs, 10-14 23:50 – 10-15 05:51) → 5.75 / 6.25 (n=2 runs after the merge). No measurable change yet; re-check at 20:16. |
| #1563, #1565 vex / cli suite tests assert codes, not sentences (#1089) | 10-15 14:36 / 14:22 | Test refactors. No CI-time effect expected or measurable. |
| #1560 scan mod tests assert codes, not sentences (#1089) | 10-15 05:57 | Test refactor. No CI-time effect expected or measurable. |
| #1556 scan hosted tests assert codes, not sentences (#1089) | 10-15 02:57 | Test refactor. No CI-time effect expected or measurable. |
| #1546 apply suite asserts codes, not sentences (#1089) | 10-14 21:00 | Test refactor. No CI-time effect expected or measurable. |
| #1544 vendor suite asserts codes, not sentences (#1089) | 10-14 20:00 | Test refactor. No CI-time effect expected or measurable yet. |
| #1525 fix nightly old-uv orphan-unwind failure in uv e2e (ci-janitor) | 10-14 09:01 | The 10-14 nightly failed on old-uv legs → the 10-15 nightly 38688589166 was green (0 failed jobs). Delivered. |
| #1497 fix phantom version bump in release-readiness (ci-janitor) | 10-13 19:59 | Correctness only. No effect on timing. |
#1484 hermetic spawns in vendor build tests; serializes the SOCKET_NPM_REGISTRY test |
10-13 14:59 | The flake from #1466 failed coverage 2× at 13:45–14:23 (1 eviction, 10 cancelled rebuilds, 340 job-min) → 0 coverage failures in the 50 merge_group runs since 14:16 (through 10-14 07:49). Delivered. |
| #1459 ride out crates.io blips in required cargo e2e legs (ci-janitor) | 10-13 13:21 | Reliability only. 0 cargo e2e failures since. |
| #1415 run the full e2e tier nightly, not per push | 10-12 16:11 | Push run: 50 jobs and 38.7–42.7 job-min (n=3, 15:53–16:10) → 9 jobs and 7.2–11.3 job-min (n=14, mean ~8.7, through 22:57). Push created → ci-ok 4.3–5.25 → 3.1–4.85 min. That is ~−0.45k Linux job-min/day at 21 pushes/day, better than the ~−300 expected. Delivered. |
| #1438 annotate failing tests in the coverage job (ci-janitor) | 10-12 17:58 | Diagnostics only. |
| #1418 env race flake in cache_env scrub self-tests (ci-janitor) | 10-12 16:10 | 0 coverage failures in merge_group after 16:02 (n=23 through 22:49). #1416 is closed. |
| #1411 macOS go.work member match (ci-janitor) | 10-12 16:18 | The 10-13 nightly 38353589047 was green, including test (macos-latest, 1), which failed on 10-12. Delivered. |
| #1392 Windows flake in notifier dead-endpoint test (ci-janitor) | 10-11 19:23 | 0 Windows failures in merge_group since. |
| #1386 type-check Windows in the queue, in parallel | 10-11 18:21 | mg created → ci-ok 9.65 → 7.6–7.9 min. e2e-build-windows hasn't been last before ci-ok in this window. Delivered. |
| #1383 skip PR re-test after a clean main merge | 10-11 18:01 | Not yet observable. 0 of ~330 sampled PR runs across seven profiles took the path. |
| #1387 re-download truncated e2e-bin archives (ci-janitor) | 10-11 17:50 | 0 e2e-bin download failures. |
| #1376 dead Depot runner doesn't cancel a merge-queue run | 10-10 13:37 | 0 runner losses. |
| #1355 reuse the merge-queue verdict on main, trim coverage/sbt | 10-10 08:57 | Push-to-main run 22+ → 3.4 min. Delivered. |
| #1375 lean per-package-manager gate | 10-09 21:29 | Job-min per mg run 312 / 69 / 60 → 33.9 / 2.9 / 0. PR → ci-ok 21.8 → 7.7. Delivered. |
| #1362 Depot Linux runners | 10-09 20:53 | Linux queue wait p90 18.3 → 0.02 min. Delivered. |
| #1284, #1275 compat path filters | 10-09 | No compatibility workflow ran in the window. Delivered. |
Method notes
- GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of merged PRs. The timelines use the last enqueue before the merge.
- Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_at. - Daily job-min = per-run mean × 24h run count per event. The PR count is scaled from the 3 listed pages: this run, 106 PR runs over ~22.9 h, about 111 per day. Merges per day come from the closed-PR list.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code