Repository navigation
fix(sql): stop MySQL deadlocks when workers acquire jobs - #33
Merged
Merged
Conversation
Retention by count deleted the jobs whose id was not in a subquery ordered by finished_at with a LIMIT. MySQL and MariaDB reject LIMIT in an IN subquery, so completeJob() and failJob() threw on MySQL as soon as removeOnComplete or removeOnFail set a count. The Knex adapter now reads the ids to keep first and deletes the others, as the Kysely adapter does. The driver test suite now runs on Knex with MySQL, which had no test group and missed this error.
On MySQL, popFrom() failed with ER_LOCK_DEADLOCK when several workers acquired jobs while others were dispatched, with or without dedup. The Worker then waited 5 s before polling again. Under REPEATABLE READ, the MySQL default, SELECT ... FOR UPDATE SKIP LOCKED skips locked rows but keeps a lock on the gap before every index entry it scans. The claim UPDATE moves the row to another status, so InnoDB inserts its new entry in the status index, often into a gap that a concurrent claim holds. With the Kysely schema, status is a varchar: the new 'active' entry lands in the gap before the first 'pending' one, which every popFrom() locks, and two workers wait on each other. With the Knex schema, an enum, two promotions of delayed jobs deadlock on the execute_at and finished_at indexes the same way. The claims of both SQL adapters (popFrom(), the promotion of delayed jobs, recoverStalledJobs() and claimDueSchedule()) now run under READ COMMITTED on MySQL, which locks the claimed rows only, as PostgreSQL does by default. A retry on ER_LOCK_DEADLOCK would have kept the conflict and only hidden it. Other transactions keep the default level. Related #31
The SQL adapters now acquire jobs and schedules in READ COMMITTED transactions on MySQL. InnoDB rejects writes at that level when binlog_format is STATEMENT, so document which formats work.
Retention by count read the ids to keep, then deleted the other jobs
in a second statement, outside a transaction. Two workers could prune
between these steps: with { count: 1 }, A finished its job and read
[A], B finished a more recent job and pruned A, then the DELETE of A
removed B and no job remained. Kysely had pruned in two steps before;
the Knex adapter did so since eb7520f, which removed a LIMIT subquery
that MySQL rejects.
Both adapters now delete in one statement and select the ids to keep
through a derived table, which MySQL accepts with a LIMIT. A test
pauses the first worker before its DELETE, on two connections, and
checks that the latest job remains, for completeJob() and failJob() on
SQLite and MySQL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On MySQL,
popFrom()of the Knex and Kysely adapters failed withER_LOCK_DEADLOCKwhen several workers acquired jobs while other processes dispatched jobs. Dedup is not necessary. The Worker then waited 5 s (DEFAULT_ERROR_RETRY_DELAY) before it polled again. The deadlock showed while #31 was reviewed.Cause
The deadlock report of InnoDB shows two claim transactions that wait on each other.
MySQL uses REPEATABLE READ by default. At this level,
SELECT ... FOR UPDATE SKIP LOCKEDskips the locked rows, but it keeps a lock on the gap before each index entry that it reads. The claimUPDATEthen changes the status of the row, so InnoDB inserts a new entry in the status indexes. This entry often goes into a gap that a concurrent claim holds.statusis a varchar): the newactiveentry ofstatus_score_idxgoes into the gap before the firstpendingentry. EachpopFrom()locks this gap, so two workers wait on each other.statusis an enum, in a different order): two promotions of delayed jobs (delayedtopending) deadlock on theexecute_atandfinished_atindexes. This occurs less frequently.Fix
On MySQL only, the claims of the two SQL adapters now run under READ COMMITTED:
popFrom(), the promotion of delayed jobs,recoverStalledJobs()andclaimDueSchedule(). At this level, InnoDB locks only the claimed rows, as PostgreSQL does by default. A retry onER_LOCK_DEADLOCKkeeps the conflict and only hides it, so this PR does not add a retry. The dedup transactions keep REPEATABLE READ.The README now tells that
binlog_format=STATEMENTrejects writes under READ COMMITTED.ROWandMIXEDwork.Also in this PR: Knex retention by count on MySQL
This PR adds a "Adapter | Knex (MySQL)" test group, which did not exist. This group found a different bug: retention by count used
NOT IN (SELECT ... LIMIT n), and MySQL and MariaDB reject this query. ThuscompleteJob()andfailJob()failed on MySQL whenremoveOnCompleteorremoveOnFailset a count. The adapter now reads the ids to keep first, as the Kysely adapter does. This fix is in a separate commit (eb7520f).Tests
New test in the shared driver suite: "concurrent popFrom does not fail while jobs are dispatched". It runs 6 dispatchers of 400 jobs each (one job in two is delayed) and 4 workers. The test does not catch errors and does not retry, so a deadlock makes it fail.
ER_LOCK_DEADLOCK)The full suite passes (1575/1575) on MariaDB 10.11 (port 3307), PostgreSQL 15, SQLite and Redis.
Limit: all MySQL tests ran on MariaDB 10.11, not on MySQL 8.4 (the version of
compose.ymland of the CI). Docker was not available. The CI run of this PR is the first run on MySQL 8.4.Related #31