Repository navigation
Conversation
Goal is to make it faster to triage individual test failures while keeping the full existing regrtest fidelity. Fairly minimal integration to hopefully save some human eyeball time. Setup so only the test summary process should output the summary, subprocesses have the env var which will cause output unset.
Member
|
Can you temporarily trigger a test failure so we can see it in action? |
Contributor
Author
|
Yea, planning to pass + fail + iterate on failure to get it pretty then get back to passing and mark as ready for review :) |
edvilme
reviewed
Oct 6, 2026
cmaloney
changed the title
Contributor
|
I think having a screenshot in the description could help out a ton (I know that is the reason for the intentional failures, but may be easier to glance and get more exposure) |
Locate annotations at the last traceback frame in a test file instead of mapping each test to its file. Pass annotations to workers in RunTests, like use_junit, so tests which run regrtest don't inherit GITHUB_STEP_SUMMARY. Shrink the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fall back to the first traceback frame (the test) instead of passing the test module file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update libregrtest to output GitHub Job Summary and Error Annotations for each failing job, test when run in Github Actions. Screenshots of the changes below.
This creates a number of UI adjustments which should make it a little easier to get around. These only happen if run in Github. Goal is to make triage a little bit simpler/faster but keep the full fidelity of the existing libregrtest output.
note: iOS and Android are excluded for now, as are other kinds of jobs like
docsandmypy. Can expand support to those but would rather do as separate well scoped PRs.Job Summaries
If you click on "checks" tab on a PR then the "tests" you get taken to the summary page (ex. https://github.com/python/cpython/actions/runs/38330186853). At the top is the workflow graph, if you scroll past that you see a job summary per failed job. If there are many failures they are "collapsed". Just a few will start expanded. The tracebacks are colored with Python traceback syntax highlighting.
Error Annotations
If you click on "Files Changed" in a PR there is a button to "Open Alerts Panel". This will show all the error annotations on all the files. Individual test failures are mapped back to the source line when possible, and that line in the Diff view will show an indicator that there is an error which can be expanded. Clicking on "show details" will take you to the specific job logs.
On individual test jobs all the annotations from that job are included at the top of the job page.
Unfortunately this does result in redundant printing of the assertion failures as we print both the
libregrtestform and a form of the github annotation line is presented. I chose for this to be just after the existing failure message which I think reads reasonably.edit: Rewrote with screenshots to show features.