Repository navigation
Conversation
gh-ost has now fixed the same deadlock twice: github#1677 on rowCopyComplete and github#1758 on ghostTableMigrated. Both are one shape. A channel whose every send goes through base.SendWithContext has a sender that abandons the send once the migration context is cancelled, so a blocking receive on that channel has no party left to wake it. Migrate() never returns, teardown() never runs, finishedMigrating is never set, and the status and throttler tickers -- which exit on that flag rather than on the context -- keep looping. The process stays alive indefinitely, logging a frozen status line. Nothing off the shelf catches this. contextcheck, containedctx and noctx are about ctx plumbing, not channel operations. Go's runtime deadlock detector requires every goroutine to be asleep, and the tickers above are still running. A gocritic/ruleguard rule cannot express it either: ruleguard offers one level of Parent() and a select-case receive has the same ExprStmt parent as a bare statement receive, so the correct ctx-guarded fix pattern is indistinguishable from the bug. go/analysis sees the whole ancestor chain via inspector.WithStack, which makes the distinction mechanical. The analyzer keys on the sender contract rather than on channel names, so it stays at zero false positives on the current tree: - exempt if any send is bare -- that sender does not abandon on cancellation - exempt if the channel is closed -- closing wakes every receiver - exempt if not a struct field -- locals are buffered and paired in-function - sends in _test.go are ignored when classifying, so a bare send in a test cannot exempt a production channel Verified against both historical bugs: it reports rowCopyComplete on the tree before github#1677 and ghostTableMigrated on the tree before github#1758. On the pre-github#1677 tree it reports ghostTableMigrated as well, so it would have surfaced github#1758 as that PR's sibling a year early. Two sites on master are fixed here, leaving the check at zero findings and zero suppressions once github#1758 lands: - listenOnPanicAbort blocked on a bare PanicAbort receive. Paths that abort directly rather than via PanicAbort -- consumeRowCopyComplete calls abort() itself -- cancel the context first, after which no publisher can ever send, parking the goroutine for the life of the process. - onApplyEventStruct drained its DML batch with a blocking receive. The count comes from len() so an event is normally waiting; the receive is now non-blocking, which is semantics-preserving for the single consumer and removes the last finding without a suppression comment. tools/ctxrecv is a separate module so golang.org/x/tools never enters the parent go.mod or vendor/ for a binary the shipped gh-ost never uses. CI builds it on demand, matching the existing install-at-CI-time pattern for golangci-lint. analysistest covers all seven report cases and all six exemptions, so loosening the classification fails the test rather than silently reintroducing noise. Known scope limit: only receives are checked. A send with no receiver on some path -- the shape of github#1736 -- is the absence of code and cannot be detected syntactically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The new
cancellation-safetyjob will fail with:That is not a regression — it is the check working. Line 568 is the deadlock
#1758 fixes, and github#1758 is not yet
merged. This branch is based on
masterdeliberately, so CI demonstrates theanalyzer catching a real, known bug in real code rather than only in fixtures.
It goes green the moment github#1758 lands. Every other job should pass.
Why
gh-ost has fixed the same deadlock twice: #1677
on
rowCopyCompleteand #1758 onghostTableMigrated. They are one shape.A channel whose every send goes through
base.SendWithContexthas a sender thatabandons the send once the migration context is cancelled. A blocking receive
on such a channel therefore has nobody left to wake it:
Migrate()neverreturns,
teardown()never runs,finishedMigratingis never set, and thestatus and throttler tickers — which exit on that flag rather than on the
context — keep looping. The process stays alive indefinitely, logging a frozen
status line, until it is killed externally.
Why a custom analyzer
Nothing off the shelf catches it.
contextcheck,containedctxandnoctx(all already enabled) are about ctx plumbing, not channel operations. Go's
runtime deadlock detector needs every goroutine asleep, and the tickers above
are still running.
A
gocritic/ruleguardrule cannot express it either. ruleguard offers onelevel of
Parent(), and ingo/asta select-case receive has the sameExprStmtparent as a bare statement receive — so the correct ctx-guarded fixpattern is indistinguishable from the bug.
go/analysissees the full ancestorchain via
inspector.WithStack, which makes the distinction mechanical.How it stays quiet
It keys on the sender contract, not on channel names. Exemptions:
_test.goThat last one is load-bearing:
migrator_test.godoes a bare send toghostTableMigrated, which would otherwise exempt the very channel github#1758 is about.Result on
master+ this PR: one finding (the github#1758 site), zerosuppressions. No
//nolintanywhere.Validation
Checked against both historical bugs:
rowCopyComplete:237✅ andghostTableMigrated:530ghostTableMigrated:557✅ghostTableMigrated:568onlyOn the pre-github#1677 tree it also reports
ghostTableMigrated— it would havesurfaced github#1758's bug as that PR's sibling a year early.
analysistestcovers all 7 report cases and all 6 exemptions. I verified theharness fails in both directions (removing a real expectation → "unexpected
diagnostic"; adding a bogus one → "no diagnostic reported"), so loosening the
classification breaks the test rather than silently reintroducing noise.
Two fixes included
Both are sites the analyzer flags on
master, fixed so the check lands at zero:listenOnPanicAbortblocked on a barePanicAbortreceive. Paths thatabort directly rather than via
PanicAbort—consumeRowCopyCompletecallsabort()itself — cancel the context first, after which no publisher can eversend, parking the goroutine for the life of the process. Real leak; benign in
practice, since the process is exiting anyway.
onApplyEventStructdrained its DML batch with a blocking receive. Thecount comes from
len()so an event is normally waiting; the receive is nownon-blocking. Semantics-preserving for the single consumer, and it removes the
last finding without a suppression comment.
Packaging
tools/ctxrecvis a separate module sogolang.org/x/toolsnever enters theparent
go.modorvendor/for a binary the shippedgh-ostnever uses. CIbuilds it on demand — the same install-at-CI-time pattern already used for
golangci-lint. Verified: parent
go.mod/go.sum/vendor/unchanged,go list ./...excludes it, andgo mod vendorremains a no-op.Scope limits (deliberate)
#1736 — is the absence of code
and cannot be detected syntactically.
paired in-function.
SendWithContextfrom outsidego/logicwould be invisible. True today; noted in the README as an assumption the
analyzer cannot enforce.
Not included
The
finishedMigrating→ context signal collapse. Those 8 ticker loops are notbroken today —
teardown()does run onceMigrate()returns — but they are whatturns a future missed receive from a leaked goroutine into a frozen process.
Worth doing separately, on blast-radius grounds.
Local verification
script/test-macos test→ exit 0, all 6 packages, zero failures (including allthree testcontainers suites)
golangci-lint run ./go/...→ 0 issuesgofmtclean;go -C tools/ctxrecv test ./...→ ok