Skip to content

Add ctxrecv analyzer for receives cancellation cannot wake - #9

Draft
ggilder wants to merge 1 commit into
masterfrom
ctxrecv-cancellation-analyzer
Draft

ggilder wants to merge 1 commit into
masterfrom
ctxrecv-cancellation-analyzer

Conversation

@ggilder

@ggilder ggilder commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Draft, opened against my own fork to exercise CI. Not for upstream review yet.

⚠️ CI is expected to be red, with exactly one finding

The new cancellation-safety job will fail with:

go/logic/migrator.go:568:3: blocking receive from ghostTableMigrated cannot be
woken by cancellation: every send to it uses base.SendWithContext, which
abandons the send once the migration context is cancelled; select on
migrationContext.GetContext().Done() alongside it

That is not a regression — it is the check working. Line 568 is the deadlock
#1758 fixes, and github#1758 is not yet
merged. This branch is based on master deliberately, so CI demonstrates the
analyzer catching a real, known bug in real code rather than only in fixtures.
It goes green the moment github#1758 lands. Every other job should pass.

Why

gh-ost has fixed the same deadlock twice: #1677
on rowCopyComplete and #1758 on
ghostTableMigrated. They are one shape.

A channel whose every send goes through base.SendWithContext has a sender that
abandons the send once the migration context is cancelled. A blocking receive
on such a channel therefore has nobody left to wake it: Migrate() never
returns, teardown() never runs, finishedMigrating is never set, and the
status and throttler tickers — which exit on that flag rather than on the
context — keep looping. The process stays alive indefinitely, logging a frozen
status line, until it is killed externally.

Why a custom analyzer

Nothing off the shelf catches it. contextcheck, containedctx and noctx
(all already enabled) are about ctx plumbing, not channel operations. Go's
runtime deadlock detector needs every goroutine asleep, and the tickers above
are still running.

A gocritic/ruleguard rule cannot express it either. ruleguard offers one
level of Parent(), and in go/ast a select-case receive has the same
ExprStmt parent as a bare statement receive — so the correct ctx-guarded fix
pattern is indistinguishable from the bug. go/analysis sees the full ancestor
chain via inspector.WithStack, which makes the distinction mechanical.

How it stays quiet

It keys on the sender contract, not on channel names. Exemptions:

exempt when rationale
any send is bare that sender does not abandon on cancellation
the channel is closed closing wakes every receiver
not a struct field locals are buffered and paired within one function
sends in _test.go a bare send in a test must not exempt a production channel

That last one is load-bearing: migrator_test.go does a bare send to
ghostTableMigrated, which would otherwise exempt the very channel github#1758 is about.

Result on master + this PR: one finding (the github#1758 site), zero
suppressions
. No //nolint anywhere.

Validation

Checked against both historical bugs:

tree reported
before github#1677 rowCopyComplete:237 ✅ and ghostTableMigrated:530
before github#1758 ghostTableMigrated:557 ✅
this branch ghostTableMigrated:568 only
this branch + github#1758 (nothing)

On the pre-github#1677 tree it also reports ghostTableMigrated — it would have
surfaced github#1758's bug as that PR's sibling a year early.

analysistest covers all 7 report cases and all 6 exemptions. I verified the
harness fails in both directions (removing a real expectation → "unexpected
diagnostic"; adding a bogus one → "no diagnostic reported"), so loosening the
classification breaks the test rather than silently reintroducing noise.

Two fixes included

Both are sites the analyzer flags on master, fixed so the check lands at zero:

  • listenOnPanicAbort blocked on a bare PanicAbort receive. Paths that
    abort directly rather than via PanicAbort — consumeRowCopyComplete calls
    abort() itself — cancel the context first, after which no publisher can ever
    send, parking the goroutine for the life of the process. Real leak; benign in
    practice, since the process is exiting anyway.
  • onApplyEventStruct drained its DML batch with a blocking receive. The
    count comes from len() so an event is normally waiting; the receive is now
    non-blocking. Semantics-preserving for the single consumer, and it removes the
    last finding without a suppression comment.

Packaging

tools/ctxrecv is a separate module so golang.org/x/tools never enters the
parent go.mod or vendor/ for a binary the shipped gh-ost never uses. CI
builds it on demand — the same install-at-CI-time pattern already used for
golangci-lint. Verified: parent go.mod/go.sum/vendor/ unchanged, go list ./... excludes it, and go mod vendor remains a no-op.

Scope limits (deliberate)

  • Receives only. A send with no receiver on some path — the shape of
    #1736 — is the absence of code
    and cannot be detected syntactically.
  • Struct fields only. Function-local channels are typically buffered and
    paired in-function.
  • Sends classified per-package. A SendWithContext from outside go/logic
    would be invisible. True today; noted in the README as an assumption the
    analyzer cannot enforce.

Not included

The finishedMigrating → context signal collapse. Those 8 ticker loops are not
broken today — teardown() does run once Migrate() returns — but they are what
turns a future missed receive from a leaked goroutine into a frozen process.
Worth doing separately, on blast-radius grounds.

Local verification

  • script/test-macos test → exit 0, all 6 packages, zero failures (including all
    three testcontainers suites)
  • golangci-lint run ./go/... → 0 issues
  • gofmt clean; go -C tools/ctxrecv test ./... → ok

gh-ost has now fixed the same deadlock twice: github#1677 on rowCopyComplete and
github#1758 on ghostTableMigrated. Both are one shape. A channel whose every send
goes through base.SendWithContext has a sender that abandons the send once the
migration context is cancelled, so a blocking receive on that channel has no
party left to wake it. Migrate() never returns, teardown() never runs,
finishedMigrating is never set, and the status and throttler tickers -- which
exit on that flag rather than on the context -- keep looping. The process stays
alive indefinitely, logging a frozen status line.

Nothing off the shelf catches this. contextcheck, containedctx and noctx are
about ctx plumbing, not channel operations. Go's runtime deadlock detector
requires every goroutine to be asleep, and the tickers above are still running.
A gocritic/ruleguard rule cannot express it either: ruleguard offers one level
of Parent() and a select-case receive has the same ExprStmt parent as a bare
statement receive, so the correct ctx-guarded fix pattern is indistinguishable
from the bug. go/analysis sees the whole ancestor chain via inspector.WithStack,
which makes the distinction mechanical.

The analyzer keys on the sender contract rather than on channel names, so it
stays at zero false positives on the current tree:

  - exempt if any send is bare -- that sender does not abandon on cancellation
  - exempt if the channel is closed -- closing wakes every receiver
  - exempt if not a struct field -- locals are buffered and paired in-function
  - sends in _test.go are ignored when classifying, so a bare send in a test
    cannot exempt a production channel

Verified against both historical bugs: it reports rowCopyComplete on the tree
before github#1677 and ghostTableMigrated on the tree before github#1758. On the pre-github#1677
tree it reports ghostTableMigrated as well, so it would have surfaced github#1758 as
that PR's sibling a year early.

Two sites on master are fixed here, leaving the check at zero findings and zero
suppressions once github#1758 lands:

  - listenOnPanicAbort blocked on a bare PanicAbort receive. Paths that abort
    directly rather than via PanicAbort -- consumeRowCopyComplete calls abort()
    itself -- cancel the context first, after which no publisher can ever send,
    parking the goroutine for the life of the process.
  - onApplyEventStruct drained its DML batch with a blocking receive. The count
    comes from len() so an event is normally waiting; the receive is now
    non-blocking, which is semantics-preserving for the single consumer and
    removes the last finding without a suppression comment.

tools/ctxrecv is a separate module so golang.org/x/tools never enters the parent
go.mod or vendor/ for a binary the shipped gh-ost never uses. CI builds it on
demand, matching the existing install-at-CI-time pattern for golangci-lint.
analysistest covers all seven report cases and all six exemptions, so loosening
the classification fails the test rather than silently reintroducing noise.

Known scope limit: only receives are checked. A send with no receiver on some
path -- the shape of github#1736 -- is the absence of code and cannot be detected
syntactically.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant