Skip to content

Across Rubies: charts and cards that say what happened - #254

Merged
JuanVqz merged 9 commits into
mainfrom
feature/across-rubies-readability
Oct 12, 2026
Merged

Across Rubies: charts and cards that say what happened#254
JuanVqz merged 9 commits into
mainfrom
feature/across-rubies-readability

Conversation

@JuanVqz

@JuanVqz JuanVqz commented Oct 12, 2026

Copy link
Copy Markdown
Member

Summary

Makes the Across Rubies cards say what happened, and fixes charts that could lead to the wrong conclusion.

  • Relative to the claimed idiom. Every report of a benchmark is now relative to the claimed report on the reference Ruby, so on the chart the higher line is the faster idiom. Before, each report was relative to itself, and under YJIT the splat line sat about 130× above the claimed one even though the claimed one was faster. Values keep 4 significant digits, so a report millions of times slower does not round to 0.
  • Log scale. One fast engine or idiom no longer flattens the rest. For example, in "normal vs &method", TruffleRuby near 400M i/s put every MRI line at 0.
  • A verdict on every card, for the build you pick. An icon and a sentence such as "attr_accessor wins or ties on all 16 Rubies (2 ties)", then labelled tiles: how much faster or slower, the claimed idiom's own speed from the oldest to the newest released Ruby, and with a JIT, the JIT against the interpreter. It compares the claimed idiom with the fastest of the others, not just the second report. The tooltip and the table do the same.
  • Easier to scan. The report labels are the card title, kept on one line when they fit at 14px or more. Section headings are larger, with a count and a "Jump to" list. The x labels name the engine once per group. A legend entry explains the suspect triangles. CPU, shard and variation move into "Numbers per build". A "How to read these charts" link points to the explanation, which now covers the icons, the log scale and the baseline.

How it was checked

All checks ran on run 37571671870's artifacts. Built with main's builder, they give the same across-rubies.json as the live site.

  • Builder unit tests pass, including new ones for the shared baseline and for a report 27M times slower.
  • A separate script recomputes wins, ties and losses from the JSON: 234 of 234 headlines match (interpreter, YJIT, ZJIT), and it reports mismatches when wrong counts are planted.
  • A headless Chrome check over 64 combinations at 1200 and 390 px runs 3269 checks with 0 problems. It covers icons and headlines, no overlapping x labels, titles that fit, "Jump to" landing under the picker, and no NaN in any chart.
  • Screenshots at 1300, 760 and 390 px.

Not checked: a run of the Results site workflow from this branch. The builder change only touches the relative value, and the page is static.

JuanVqz added 9 commits October 9, 2026 01:09
@JuanVqz
Step 1: Make every report relative to the claimed one
b89e500 
The relative view divided each report by its own value on the reference
Ruby, so a slower report could draw far above the claimed one (splat at
about 130x under YJIT while the single Array argument still won).
Dividing every report by the claimed report's median on the reference
keeps one baseline per card, so the higher line is the faster report.
@JuanVqz
Step 2: Draw the charts on a log scale
a2729fa 
A linear axis from 0 let one fast engine or report flatten the rest:
with TruffleRuby near 400M i/s every MRI line of "normal vs &method" sat
on 0, and a 35x to 173x gap left the splat line flat. On a log axis a
100x gap and a 10% change between Rubies are both readable. The axis is
fitted to the drawn values with at least a 4x span, so close lines do
not fill the chart with noise.
@JuanVqz
Step 3: Lead each card with what happened on the selected build
18631af 
The card title was the report labels ("normal vs &method"), which says
what was compared but not what happened, and the sentence under it
counted interpreter builds even with YJIT or ZJIT picked. Each card now
opens with a verdict for the selected build: the claim wins everywhere,
wins or ties, wins on some Rubies (and who wins on the others), or
loses. Under it: the win and loss ratios, the claim's own speed across
released Rubies (about the same when within the reference's variation),
the JIT against no JIT, and which Rubies are not counted.

Wins, ties and losses compare the claim with the fastest other report,
not the second one listed. The tooltip and the table use the same rule.
@JuanVqz
Step 4: Make sections easy to find, and add a find box
44cdcdb 
The section headings were small gray capitals in a page of 78 charts,
with no way to get to one. They are now real headings with how many
benchmarks they hold, and the picker has a "Jump to" menu that lands
each heading under the pinned picker. A find box narrows the cards to
those whose title, report names, file or section contain every word
typed; it is kept in the URL as ?find=.
@JuanVqz
Step 5: Tidy the chart labels, legend and card footer
2423d6c 
The TruffleRuby labels ran into each other ("truffle" twice) and every
tick under a JIT build repeated "+yjit". Columns now show the version,
with the engine once under each group, shortened or thinned when the
chart is narrow. A card with suspect results says what the triangles
are in its legend. The CPU, shard and reference variation, the same on
most cards, moved into "Numbers per build"; a noisy benchmark keeps a
visible warning. The intro links to how to read the charts, which now
explains the verdict icons, the log scale and the relative baseline.
@JuanVqz
Step 6: Split the card's numbers into labelled tiles
6ba442b 
One line under the verdict held every number ("Wins by 1.05x (Ruby 3.0)
to 1.2x (Ruby head) · Own speed from Ruby 2.1 to 4.0: +40.4% · Not
counted: ..."), and it read as if Ruby 3.0 were being compared with Ruby
head. Each number now has its own tile: a label, the value, and a
caption saying who it is about and where ("attr_accessor over
getter_and_setter, least on Ruby 3.0, most on Ruby head"). The report
labels are the card title again, above the verdict, and the Rubies left
out of the count move under the chart they refer to.
@JuanVqz
Step 7: Keep long card titles on one line
90ecc22 
A title with many report names, like "value.between? vs plain compare vs
range#cover? vs range#include? vs range#member?", broke over two lines
at 20px. A long title now shrinks to fit one line, down to 14px, and
only wraps (at 14px) when it is too long even then: on desktop that is
the two titles over 100 characters.
@JuanVqz
Step 8: Drop the find box, keep Jump to
60badc3 
A free-text box leaves the reader guessing what to type. Sections are the way in.
@JuanVqz
Step 9: Fix what the code review found
ebfaed3 
- Keep 4 significant digits in the relative value, so a report millions of times slower than the claimed one no longer rounds to 0 and breaks the log axis (26 values on run 37571671870); the axis also skips anything at or below 0, and labels tiny values as 1/50M×.
- A claim that ties on some Rubies and loses on the rest says it never wins (✗), not "wins on 0 of N" (~).
- "the one Ruby" instead of "all 1 Rubies".
- A Ruby where the claim ran but could not be compared says "Not compared", not "Did not run".
- "How to read a card" lists every icon.
- Jump to keeps the chosen section, so arrow keys can move through the list.
- Card titles are measured in one pass and again once the web font loads.
- Drop an unused helper and a leftover from the find box.
@JuanVqz
JuanVqz merged commit c54a474 into main Oct 12, 2026
4 checks passed
@JuanVqz
JuanVqz deleted the feature/across-rubies-readability branch October 12, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JuanVqz