Don't crash retried lazy-load workers when the queue is almost empty

[tekmaven]

Why

When a worker is retried (BUILDKITE_RETRY_COUNT > 0, or a Semaphore rerun) and finds no more tests left in the queue than are running, minitest-queue run prints "Queue almost empty, exiting early..." and skips loading tests and populating the queue (#257, runner.rb). That step is also where the queue gets its entry resolver (queue_population_strategy.rb). Without lazy loading, minitest runs from minitest/autorun's at_exit hook, which loading the test files registers. No test files load, so the worker exits 0.

In lazy-load mode (--lazy-load, --preresolved-tests or CI_QUEUE_LAZY_LOAD), the runner calls Minitest.run itself and only skips it when the queue is exhausted (#375, #386, runner.rb). The queue isn't exhausted while other workers still hold tests, so the worker starts polling. The first test it reserves comes back from resolve_entry as an UnresolvedEntry (worker.rb), and example.run raises (queue.rb):

NoMethodError: undefined method 'run' for an instance of CI::Queue::Redis::UnresolvedEntry

The worker records a worker error and exits 42 (queue.rb). report then prints "Worker N crashed" and exits 42 (build_status_reporter.rb), even when every test passed. The test the worker reserved stays in running until another worker reclaims it as lost.

What

In lazy-load mode, run now exits 0 without running tests when it didn't prepare the queue, as it already did when the queue was exhausted. A retried worker that finds the queue almost empty now exits the same way it does without lazy loading.

Behavior

Scenario: 1 test is left in the queue and another worker holds 2. A --lazy-load worker runs with BUILDKITE_RETRY_COUNT=1.

Before (exit status 42):

1 tests left and 2 workers running.
Queue almost empty, exiting early...

# Running tests with run options --seed 38702:

This worker exited because of an uncaught application error:
NoMethodError: undefined method 'run' for an instance of CI::Queue::Redis::UnresolvedEntry
lib/minitest/queue.rb:233:in 'block (3 levels) in Minitest::Queue.run'
lib/ci/queue/redis/base.rb:83:in 'CI::Queue::Redis::Base#with_heartbeat'
lib/minitest/queue.rb:232:in 'block (2 levels) in Minitest::Queue.run'
lib/ci/queue/redis/worker.rb:118:in 'CI::Queue::Redis::Worker#poll'
...
lib/minitest/queue/runner.rb:140:in 'Minitest::Queue::Runner#run_command'

After (exit status 0):

1 tests left and 2 workers running.
Queue almost empty, exiting early...

Other cases:

• Queue already exhausted, lazy-load: still exits 0, and now prints "All tests were ran already" once instead of twice.
• Not affected:
  • Runs without lazy loading already exit 0 here.
  • Lazy-load workers that prepare the queue run as before, including retried ones that find more tests left than running.
  • rspec-queue and the Python client don't have this early exit.

Tests

test_lazy_load_retry_exits_without_reserving_tests_when_queue_is_almost_empty in ruby/test/integration/minitest_redis_test.rb. A worker streams 3 tests and reserves 2, leaving 1 in the queue. A --lazy-load worker then runs with BUILDKITE_RETRY_COUNT=1 and BUILDKITE_RETRY_TYPE=automatic. It exits 0, prints the two lines above, leaves the last test in the queue, and records no worker error. It fails on main (exit status 42) and passes with this change.

Results locally on Redis 8.6.3 and Ruby 4.0.7:

• minitest 5.27.0: 330 tests, 0 failures.
• minitest 6.0.6: 330 runs, 0 failures.

[kylem-work]

LGTM 👍 We shouldn't be trying to intake tests if the queue setup hasn't even run.