Vitest 5 getter tracking adds overhead to CodSpeed measurements #98

Description

After upgrading to @codspeed/vitest-plugin@6.0.0 and vitest@5.0.4, CodSpeed simulation measurements include substantial overhead from Vitest's diagnostic GetterTracker. Reporting this as a follow-up to #86.

Our migration PR, hckhanh/vn-number#320, changes the test dependencies and be nchmark API without changing the library implementation or benchmark workloads. CodSpeed reports a 29.03% performance drop, with 11 of 22 maintained benchmarks regressing.

In the dense-3000-digit reading benchmark, the new Vitest getter-tracking wrapper contributes approximately 3.4 ms of the 9.2 ms total in self cost. The old baseline is approximately 5.8 ms and does not contain this wrapper.

Steps to Reproduce

On Linux with Node 24.21.0, pnpm 12.11.2, and the CodSpeed CLI available:

1. Check out the public reproduction at the migration commit:

   git clone https://github.com/hckhanh/vn-number.git
   cd vn-number
   git checkout 4a32cdd9f60833f0c396fca24687c4568482a3a8
   pnpm install --frozen-lockfile
   mkdir -p /tmp/codspeed-getter-on /tmp/codspeed-getter-off

2. Measure with Vitest's default getter tracking enabled:

   CODSPEED_SKIP_UPLOAD=true NODE_OPTIONS=--disable-wasm-trap-handler \
     codspeed run --mode simulation \
     --profile-folder /tmp/codspeed-getter-on -- \
     node node_modules/vitest/vitest.mjs bench src/ --run

3. Add this option to the existing test configuration in vitest.config.mts, leaving the CodSpeed plugin and everything else unchanged:

   benchmark: {
     suppressExportGetterWarnings: true,
   },

4. Repeat the same measurement command with --profile-folder /tmp/codspeed-getter-off. Compare the profiles for src/read/index.bench.ts::deterministic read workloads::read/dense 3000 digits (1 conversion) and the other maintained workloads.

The WebAssembly startup option was needed to initialize the old Vitest 4 baseline under the local profiler. It was applied to all three final variants below to keep runtime settings identical.

Expected Behavior

Vitest's diagnostic access counter should not inflate the workload cost reported by CodSpeed, or the integration documentation should explain the interaction and recommended configuration.

Actual Behavior

In the installed Vitest 5.0.4 code, GetterTracker.createTracker wraps export getters with a Map lookup, increment, and write on each access. The tracker is enabled when benchmark.enabled is true and benchmark.suppressExportGetterWarnings is false. That is Vitest's default; the CodSpeed plugin currently leaves it unchanged.

Despite its name, suppressExportGetterWarnings: true disables the diagnostic counter itself, not just the printed warning. It leaves the underlying module export getters in place.

I tested three variants on the same local Linux runner. All three measured the same 22 benchmark URIs successfully. For the dense-3000-digit case:

Disabling only this counter reduces the case's instruction count by 38.75%, returning it to within 0.03% of the original baseline. These are raw Valgrind CPU instruction counts, not wall-clock timings or CodSpeed's combined cycle estimate.

With tracking disabled, 21 of 22 cases use no more instructions than the original baseline. read/quantities (50 conversions) still uses about 10% more; that remaining difference is unexplained, so this report does not claim that the tracker accounts for every difference.

Environment

• Controlled runs: Ubuntu 24.04 ARM64, same Docker image and host.
• Node 24.21.0; NODE_OPTIONS=--disable-wasm-trap-handler in all three final variants.
• CodSpeed CLI 5.4.0, simulation mode; Vite 8.3.4.
• Original baseline commit: 715603476bfc1ba441451e701913cd5862294f59.
• Migration commit: 4a32cdd9f60833f0c396fca24687c4568482a3a8.
• Measurements use the existing source-importing benchmarks; no production code or workloads were changed for the tracking-on/off experiment.

Profiles and Additional Context

• Original CI comparison. This comparison warns about different CPU models and native-core versions, which is why I also ran the controlled experiment above.
• CI comparison against a baseline with the same CPU model still reports approximately a 29% drop. Toolchain differences remain.
• Migration flamegraph for the dense-3000-digit case. The new hotspot is anonymous function at line 127 in Vitest's dist/chunks/init.DrHtV5zd.js, corresponding to GetterTracker.createTracker.
• Vitest's documentation on module-runner overhead.

Could the Vitest 5 integration disable this diagnostic counter during CodSpeed measurements, or document the interaction and recommended workaround? I understand that the underlying export getters are a separate source of overhead and that benchmarking native ESM or a properly externalized built package may be preferable.