Skip to content

Upstream sync: 112 commits through upstream a099d00643, and CI on self-hosted runners - #6

Open
Sam Gammon (sgammon) wants to merge 129 commits into
elide-25.5.xfrom
upstream/sync-2026-10-13
Open

Upstream sync: 112 commits through upstream a099d00643, and CI on self-hosted runners#6
Sam Gammon (sgammon) wants to merge 129 commits into
elide-25.5.xfrom
upstream/sync-2026-10-13

Conversation

@sgammon

@sgammon Sam Gammon (sgammon) commented Oct 13, 2026

Copy link
Copy Markdown
Member

Brings elide-25.5.x up to graalvm/graalpython master fa6ebf6beb (2026-10-13): all 109 upstream non-merge commits after the current base 154205b7511. They are cherry-picked with -x in topological order, on top of the 8 fork patches. Please rebase-merge so the (cherry picked from commit …) trailers are kept.

Skipped

None. No upstream commit in the range was already on the branch (checked the cherry picked from commit trailers and git cherry).

Conflicts (2, both in PosixModuleBuiltins.postInitialize)

Fork patch 4b503ff03b (pre-init and sys modules) split postInitialize into a pre-init-safe part plus populateEnviron(). populateEnviron() runs at post-init or, for a pre-initialized context, at patchPostInitialize. Upstream's POSIX refactor edited the old, unsplit method body.

  1. bdac2d9d13 Convert PosixSupportLibrary.(getBackend|strerror|getpid|umask): kept the fork's split. Applied upstream's change inside it: Object posixSupport became PosixSupport posixSupport, and posixLib.getBackend(posixSupport) became posixSupport.getBackend(). The java-backend geteuid/getegid/WNOHANG block stays in postInitialize, where the fork had moved it. Dropped the posixLib local from postInitialize, which was now unused.
  2. c84c43dbd8 Move shared POSIX types and remove generated dispatch: this commit deletes PosixSupportLibrary. Kept the fork's split and removed the last PosixSupportLibrary posixLib local from populateEnviron(). The rest of populateEnviron() (posixSupport.createWideStringFromString/createPathFromString/setenv) auto-merged to upstream's direct PosixSupport calls.

After the sync, git diff origin/master HEAD touches the same 9 files as the fork's original delta, so the fork's behavior is preserved.

Graal compatibility

  • Upstream's mx.graalpython/suite.py imports tools/regex (and truffle/sdk via versionFrom) at oracle/graal 5727d2bee28 (2026-10-05). None of the 109 commits changes it, so the pin is the same as on the current base.
  • 5727d2bee28 is an ancestor of upstream graal 0a530a3d066. Its non-merge commits are in both graal elide-25.5.x and upstream/wave-10 (Upstream wave 10: 17 commits through upstream c38b86ec5c6 graal#132).

Testing

Sibling layout: graal = elide/upstream/wave-10 (14403bb2cda), graalpython = this branch, labsjdk-ce-latest-jvmci-25.4-b23.

  • mx build --build-logs=silent: OK.
  • Smoke: mx python -c "import sys, json, re, os; …" prints 3.13.14 … [Graal, Interpreted, Java 25.0.4.1.1]. json/re work, and os.getpid(), os.strerror(), os.umask() and os.environ work. These go through the conflict-resolved code.
  • mx graalpytest with tests/test_json.py, test_re.py, test_list.py, test_posix.py, test_sys.py and test_builtin.py: 292 run, 282 passed, 10 skipped, 0 failed.
  • mx graalpytest --tagged lib-python/3/test/test_os.py lib-python/3/test/test_posix.py: 259 run, all passed.

Picked (109)

  • dae499f897 Report loop count wherever the count can be derived in O(1) and without any virtual or boundary call and without additional code in the loop body
  • 5022a3ab9b Fix PBaseExceptionGroup copyright year
  • ff9eddd706 Implement basic Windows console support
  • 2bedaad9a8 Fix async generator aclose
  • bfa12ea861 Keep native method descriptors unbound
  • 7436733b09 Enable Cython fast calling conventions
  • 92ce61b2a5 Use vectorcall for native tp_call slots
  • e36ab50103 Optimize vectorcall method dispatch
  • c9d96fef07 Handle native method descriptors in vectorcall dispatch
  • b529092951 Optimize vectorcall transitions
  • 3eed965d7d Convert GraalPyPrivate_Object_VectorcallMethod to static upcall
  • aa9fb342c0 Match CPython password lookup error handling
  • f2e2546a77 Add patches for Cython up to 3.3.0
  • b04f5a9ce6 Remove useless TruffleNode.EqualNode param from OSErrorEnum.fromMessage and callers
  • 145916fd44 Convert PosixSupportLibrary.read to a bare Java method
  • 2d49eda893 Convert PosixSupportLibrary.write to a bare Java method
  • 68b4fa22de Convert PosixSupportLibrary.(getWindowsConsoleType|writeWindowsConsole)
  • 827aad9836 Introduce Path abstraction to PosixSupport
  • 58bfbc6054 Accept Java String in PythonContext.getPublicTruffleFileRelaxed
  • 5f8b5f6c05 Use j.l.String as backend path type in EmulatedPosixSupport
  • 94bcbbd997 Remove InlinedConditionProfile from EmulatedPosixSupport.resolvePath
  • 4be2a16189 Convert PosixSupportLibrary.(fstatat|fstat|statvfs|fstatvfs) to a bare Java methods
  • 4182cb848f Convert PosixSupportLibrary descriptor inheritance methods
  • bdac2d9d13 Convert PosixSupportLibrary.(getBackend|strerror|getpid|umask)
  • 638dadd511 Convert PosixSupportLibrary file descriptor operations
  • c4de063fef Convert PosixSupportLibrary Windows descriptor operations
  • c2fae6cb2b Convert PosixSupportLibrary pipes and readiness
  • f76c88df79 Convert PosixSupportLibrary positioning operations
  • 09d8a12067 Convert PosixSupportLibrary synchronization and locking operations
  • a4dd588cd0 Convert PosixSupportLibrary descriptor modes and terminal size
  • ff0c9aeded Convert PosixSupportLibrary system information operations
  • ae199352a3 Convert PosixSupportLibrary path namespace operations
  • 54aadbd21b Convert PosixSupportLibrary working directory and TTY operations
  • ab66a9d7ce Convert PosixSupportLibrary directory stream lifecycle operations
  • 51857683db Convert PosixSupportLibrary directory entry access operations
  • 26147a9beb Convert PosixSupportLibrary file time operations
  • 36b4d0d210 Convert PosixSupportLibrary renaming operations
  • d6a29bb5e1 Convert PosixSupportLibrary access and ownership operations
  • 222d20974a Convert PosixSupportLibrary signals and timers
  • 6a74bf7b7e Convert PosixSupportLibrary process waiting
  • 46f5ca3334 Convert PosixSupportLibrary wait-status decoding
  • 0b2121879a Convert PosixSupportLibrary user and group identity
  • 896e0a3de7 Convert PosixSupportLibrary process groups and usage
  • 5d09ed0833 Convert PosixSupportLibrary terminal identifiers
  • b6a6b01560 Convert PosixSupportLibrary environment operations
  • 5361def3af Convert PosixSupportLibrary process execution
  • 6c899bffe5 Convert PosixSupportLibrary mapping and byte access
  • 8a4a22087f Convert PosixSupportLibrary mapping completion
  • 478331c272 Convert PosixSupportLibrary semaphore and shared-memory lifetime operations
  • 5f98d0798a Convert PosixSupportLibrary semaphore operations
  • 4497d512a9 Convert PosixSupportLibrary password database operations
  • 3926ec2860 Convert PosixSupportLibrary path conversions
  • 98210d8f30 Convert PosixSupportLibrary narrow and wide strings
  • 8bd2dcbcb3 Convert PosixSupportLibrary socket lifecycle and addresses
  • 2af6d4f454 Convert PosixSupportLibrary socket data and options
  • a93890c8b7 Convert PosixSupportLibrary numeric Internet addresses
  • 969f7dde22 Convert PosixSupportLibrary name resolution and address-info cursor
  • ad2c4308bf Convert PosixSupportLibrary device control
  • 9670d74e7e Convert PosixSupport socket-address factories and access
  • c84c43dbd8 Move shared POSIX types and remove generated dispatch
  • cc4a2096ad Generic cleanups
  • b8ff82eb99 Move POSIX string conversions into nodes
  • 570be18ba6 Fix style
  • ed267f9fe6 Disable transiently failing test_os.FwalkTests.test_fd_finalization
  • 66b223a1a7 Address review comments
  • 213c26e78a Fix swapped failed and unknown states on compatibility page
  • 4eb4ef478c Fix threshold value in tooltips
  • cedf51156d Change wording of Incompatible category
  • c9121ec536 Don't include metadata entries in aggregate test results
  • ecdd07c611 Set sysconfig TZPATH on POSIX so zoneinfo finds system time zones
  • 5edd93e7e8 Reduce temporary bytearray allocations in PPickler
  • 34df922e10 Lazily initialize fastMemo table
  • 58067cbc9f Simplify pickling builtins
  • ce1f410d4d Profile version and fast mode when pickling
  • 964bd5ed2e Add a recursion limit to PPickler
  • 1011631ac9 Add more profiling to PPickler
  • f1044d3caf Add fast path for pickling sequences of primitives
  • ba662f23ae Add more pickler optimizations
  • b60b36c563 Push down helper nodes into PPickler.SaveNode
  • afc6dd2e34 Make pickler write helpers static
  • a9f6108fd1 Avoid graph size explosion
  • ad3897172f Get rid of MemoIterator
  • d867056e40 Profile buffer bypass
  • c31f65db54 Keep size estimates for buffer and memo table
  • bc0e552e5d Use more specific nodes when working with collections
  • 2d7de94d69 Fix emitting OPCODE_BYTEARRAY8
  • 2600c46f2b Reintroduce BasePickleNode again
  • c275c1d692 Organize SaveNode fields
  • 7848db81fb Move write helpers to PPickler
  • 7bfa0b69dc Get rid of PPickler.maxOutputLen
  • 5f78bd2280 Get rid of ByteArrayView
  • eb14674fa6 Add unpickler fast path for decoding UTF-8
  • 9e2d6e5c63 Add profiling for seen unpickler opcodes
  • 2102b34e2f Pre-size dictionaries when unpickling
  • 2212a831a8 Add a few branch probabilities
  • 16e83bf34b Avoid copying buffer in pickle.loads
  • 1a5c6500cd Profile file reads in unpickle
  • 3047bf5450 Add fast paths for pickling/unpickling large longs
  • 538816c84f Avoid boundary in getDottedPath
  • d0c6a40826 Cache unpickled names
  • 2934d6d06f Use ImportName in unpickler
  • 427c4fb120 Push members down from BasePickleNode
  • a75550eaae Use the same import mechanism in pickler and unpickler
  • 29988e99e9 Improve pickler graph size
  • 90dc553c5a Use separate recursive SaveNodes
  • dff33e71d9 Report loop counts in pickle code
  • c99ab1c55f Avoid out-of-bounds access on list mutation in pickler
  • 50ce7eebbe Downsize oversized output bytes
  • a1b186e482 Cap the max pickle buffer/memo size estimate at 15 MiB

@steve-s @sgammon
Report loop count wherever the count can be derived in O(1) and witho…
5554362 
…ut any virtual or boundary call and without additional code in the loop body

(cherry picked from commit dae499f)
@steve-s @sgammon
Fix PBaseExceptionGroup copyright year
41154e5 
(cherry picked from commit 5022a3a)
@msimacek @sgammon
Implement basic Windows console support
408297c 
(cherry picked from commit ff9eddd)
@msimacek @sgammon
Fix async generator aclose
d1c9939 
Fixes oracle#1196

(cherry picked from commit 2bedaad)
@fangerer @sgammon
Keep native method descriptors unbound
5ee2973 
(cherry picked from commit bfa12ea)
@fangerer @sgammon
Enable Cython fast calling conventions
0e49dc8 
(cherry picked from commit 7436733)
@fangerer @sgammon
Use vectorcall for native tp_call slots
654fdd4 
(cherry picked from commit 92ce61b)
@fangerer @sgammon
Optimize vectorcall method dispatch
a350288 
(cherry picked from commit e36ab50)
@fangerer @sgammon
Handle native method descriptors in vectorcall dispatch
6cf7dfb 
(cherry picked from commit c9d96fe)
@fangerer @sgammon
Optimize vectorcall transitions
e04e981 
(cherry picked from commit b529092)
@fangerer @sgammon
Convert GraalPyPrivate_Object_VectorcallMethod to static upcall
b62331c 
(cherry picked from commit 3eed965)
@fangerer @sgammon
Match CPython password lookup error handling
46aa854 
(cherry picked from commit aa9fb34)
@msimacek @sgammon
Add patches for Cython up to 3.3.0
662de3d 
(cherry picked from commit f2e2546)
@fangerer @sgammon
Remove useless TruffleNode.EqualNode param from OSErrorEnum.fromMessa…
b634e6d 
…ge and callers

(cherry picked from commit b04f5a9)
@fangerer @sgammon
Convert PosixSupportLibrary.read to a bare Java method
dc18eb7 
(cherry picked from commit 145916f)
@fangerer @sgammon
Convert PosixSupportLibrary.write to a bare Java method
840fc97 
(cherry picked from commit 2d49eda)
@fangerer @sgammon
Convert PosixSupportLibrary.(getWindowsConsoleType|writeWindowsConsole)
1851a46 
(cherry picked from commit 68b4fa2)
@fangerer @sgammon
Introduce Path abstraction to PosixSupport
2cd95f1 
(cherry picked from commit 827aad9)
@fangerer @sgammon
Accept Java String in PythonContext.getPublicTruffleFileRelaxed
dd82c10 
(cherry picked from commit 58bfbc6)
@fangerer @sgammon
Use j.l.String as backend path type in EmulatedPosixSupport
0af9ee3 
(cherry picked from commit 5f8b5f6)
@fangerer @sgammon
Remove InlinedConditionProfile from EmulatedPosixSupport.resolvePath
b470812 
(cherry picked from commit 94bcbbd)
@fangerer @sgammon
Convert PosixSupportLibrary.(fstatat|fstat|statvfs|fstatvfs) to a bar…
a72938c 
…e Java methods

(cherry picked from commit 4be2a16)
@fangerer @sgammon
Convert PosixSupportLibrary descriptor inheritance methods
eea387f 
(cherry picked from commit 4182cb8)
@fangerer @sgammon
Convert PosixSupportLibrary.(getBackend|strerror|getpid|umask)
b75b30c 
(cherry picked from commit bdac2d9)
@fangerer @sgammon
Convert PosixSupportLibrary file descriptor operations
4ed212c 
(cherry picked from commit 638dadd)
@fangerer @sgammon
Convert PosixSupportLibrary Windows descriptor operations
e808757 
(cherry picked from commit c4de063)
@fangerer @sgammon
Convert PosixSupportLibrary pipes and readiness
4869778 
(cherry picked from commit c2fae6c)
@fangerer @sgammon
Convert PosixSupportLibrary positioning operations
d5e2dcc 
(cherry picked from commit f76c88d)
@fangerer @sgammon
Convert PosixSupportLibrary synchronization and locking operations
f29abc6 
(cherry picked from commit 09d8a12)
@fangerer @sgammon
Convert PosixSupportLibrary descriptor modes and terminal size
89bac6d 
(cherry picked from commit a4dd588)
@msimacek @sgammon
Push members down from BasePickleNode
889c458 
(cherry picked from commit 427c4fb)
@msimacek @sgammon
Use the same import mechanism in pickler and unpickler
1943631 
(cherry picked from commit a75550e)
@msimacek @sgammon
Improve pickler graph size
d24ecaf 
(cherry picked from commit 29988e9)
@msimacek @sgammon
Use separate recursive SaveNodes
1e83918 
(cherry picked from commit 90dc553)
@msimacek @sgammon
Report loop counts in pickle code
b877bb5 
(cherry picked from commit dff33e7)
@msimacek @sgammon
Avoid out-of-bounds access on list mutation in pickler
c097bfd 
(cherry picked from commit c99ab1c)
@msimacek @sgammon
Downsize oversized output bytes
56e63f2 
(cherry picked from commit 50ce7ee)
@msimacek @sgammon
Cap the max pickle buffer/memo size estimate at 15 MiB
759f836 
(cherry picked from commit a1b186e)
@sgammon
Format PosixModuleBuiltins after the upstream sync's conflict resolution
4868cbf
@sgammon
ci: run builds and tests on Elide's self-hosted runners
27e7c31 
GitHub-hosted runners are only used for small jobs now: generating the job
matrices, merging test reports, post-merge notifications and the website.
Everything that builds or tests GraalPy runs on the self-hosted pool:

- the jobs generated from ci.jsonnet (extract_matrix.py maps linux-amd64,
  linux-aarch64 and windows-amd64 to linux-amd64-cipool, linux-arm64-cipool
  and windows-amd64-cipool),
- abi-check, the wheel builds, the weekly downstream tests and the exotic
  platform tests.
@sgammon
ci: set up Python with uv on self-hosted Linux and macOS runners
11d3a5e 
actions/setup-python only has interpreters for GitHub's runner images, so it
failed on the self-hosted Ubuntu 26.04 runners ("The version '3.12.8' with
architecture 'x64' was not found for Ubuntu 26.04"). Install the requested
version with uv (python-build-standalone) into a virtualenv with pip and put it
first on the PATH; Windows keeps setup-python.
@sgammon
ci: make the build steps work on persistent self-hosted runners
ec010da 
Start every build job from an empty workspace (self-hosted runners keep it, so
cloning mx next to the checkout failed with "destination path 'mx' already
exists"), set a UTF-8 locale as GitHub's runners have (the JDK otherwise can't
open the CA certificates with non-ASCII file names), and read the matrix
environment with python3 instead of jq, which not every runner has.
@sgammon
Update the copyright years of two files the fork changed in 2026
0f7cb5a
@sgammon
ci: keep uv's Python, cache and links inside the job's temp directory
4e4626a 
On some self-hosted runners uv picked up a broken interpreter ("Python
installation is missing a _sysconfigdata_ file"). Use only uv-managed
interpreters, a per-job cache and copied (not hard-linked) files.
@sgammon
ci: install uv's Python afresh in every job
c034e12 
The runner temp directory is not emptied between jobs on the self-hosted
runners, so a partial interpreter left by an earlier job was reused without
downloading ("Python installation is missing a _sysconfigdata_ file").
@steve-s @sgammon
Add megamorphic attribute access benchmark
b75b303 
(cherry picked from commit 706d733)
@msimacek @sgammon
Add a range specialization to tuple constructor
f976561 
(cherry picked from commit d742f86)
@msimacek @sgammon
Untag transient test
cda0f00 
(cherry picked from commit 71c7d34)
@sgammon Sam Gammon (sgammon) changed the title Upstream sync: 109 commits through upstream fa6ebf6beb Upstream sync: 112 commits through upstream a099d00643, and CI on self-hosted runners Oct 13, 2026
@sgammon
Re-create native compression support when patching a pre-initialized …
0d91f61 
…context

patchRuntimeInformation() only re-seeded the hash secret and patched the
package paths. The native zlib, bz2 and lzma supports are created during
image build without native access, so their native functions are null, and
in native standalones (which patch a pre-initialized context) BZ2Compressor
and BZ2Decompressor failed with a NullPointerException in
NativeBz2Support.createStream. This broke the tagged test_tarfile
CommandLineTest tests, which run `python -m tarfile` on .tar.bz2 files in
the native standalone.

Re-create the native supports, the locale, the time zone and the int max
str digits from the run-time environment, as upstream's
setupRuntimeInformation does when patching.
@sgammon
ci: skip the GraalOS resources job and make the Windows Maven gate ru…
a706659 
…nnable

python-graalos-resources needs GRAALPY_GRAALOS_ARTIFACT_BASE_URL, an
Oracle-internal overlay import that is empty in the public ci.jsonnet, so
it fails identically in upstream's GitHub Actions; python-resource-variants
consumes its artifact. Exclude both from the generated matrix.

python-junit-maven-gate-windows failed because the self-hosted Windows
runner has no Chocolatey and no preinstalled Maven; download Maven directly
when choco is missing. The gate then builds a delvewheel venv, which needs
CPython >= 3.12, so give that job Python 3.12.8 instead of ci.jsonnet's 3.8
(upstream's run fails at that point).
@sgammon
ci: only apt-get abigail-tools in abi-check when abidiff is missing
8b2e58e 
The self-hosted runners already have it, and an apt-get install there fails
whenever dpkg has a half-configured package left by another job (the Maven
install pulled openjdk-25-jre-headless, whose postinst fails on a runner
without /usr/share/man/man1).
@sgammon
ci: do not run mvn --version right after downloading Maven on Windows
66f322e 
The JDK is only fetched in a later step, so the check failed with 'No
external JVM found in PATH' and took the step with it.
@sgammon
Initialize the hash secret for fresh (non-pre-initialized) contexts a…
fc2e796 
…gain

Fork commit 4b503ff moved the patching of pre-initialized contexts into
patchRuntimeInformation() and, in doing so, dropped upstream's
initializeHashSecret() call from the fresh-context path of
setupRuntimeInformation(). Contexts that are not patched from a
pre-initialized image, such as the JVM standalone, therefore kept an
all-zero hash secret: hash randomization was effectively disabled there and
the InitializationEntropySource was never consulted for it, which is what
tests/test_entropy_subprocess.py detects (the child never opens the entropy
device, or consumes 24 bytes fewer than expected).

Restore upstream's call, guarded by !env.isPreInitialization() as before;
the patching path keeps its existing call.
@sgammon
ci: download Maven on Linux when apt-get leaves no mvn on the PATH
97beb9e 
python-junit-maven-gate-linux-amd64 failed with "No such file or directory:
'mvn'" on a self-hosted runner whose dpkg state is broken: the maven
package is unpacked but unconfigured there, and the apt-get failure is
tolerated by the install loop. Fall back to a direct download of Maven for
jobs that list it, as the Windows step already does.
@sgammon
ci: give the Windows JUnit gate CPython 3.12 as well
079e5ff 
python-junit-gate-windows runs punittest, which also goes through
get_path_with_patchelf and so needs CPython >= 3.12 for the delvewheel
venv, like the Maven JUnit gate.
@sgammon
Untag two hanging asyncio subprocess tests on linux-aarch64-github
3eaa759 
PollEventLoopTests.test_subprocess_close_after_finish and
test_subprocess_wait_no_same_group time out after 600s in
python-tagged-unittest-gate-batch1-linux-aarch64 on the self-hosted
runners. test_subprocess_close_after_finish fails the same way in
upstream's own GitHub Actions runs (oracle/graalpython run 38434376885,
batch1 on x86_64), so these are known-flaky under the GitHub tag set; the
linux-x86_64-github tag is kept where the test passes.
@sgammon
Untag OS-vs-JDK tzdata dependent system-transition tests for the GitH…
ac8a660 
…ub tag sets

ZoneInfoTest[...].test_system_transitions reads the transitions from the OS
tzdata in /usr/share/zoneinfo (the path is hard-coded in datetimetester) and
compares them with datetime.fromtimestamp(), which GraalPy backs with the
JDK's tz rules. The self-hosted linux-aarch64 runners carry Ubuntu 26.04's
tzdata 2026c, which changed Africa/Casablanca, Africa/El_Aaiun and
America/Edmonton for 2026 relative to the labsjdk's bundled tzdata, so
those six cases fail there (python-tagged-unittest-gate-batch6); the amd64
runners will follow once their tzdata is updated. Drop the *-github tags
for exactly those cases; the darwin and non-GitHub linux tags are kept.
@sgammon
Untag the hanging asyncio no-pipe subprocess tests on linux-amd64-git…
ec71709 
…hub too

PollEventLoopTests.test_subprocess_close_after_finish also timed out in
python-tagged-unittest-gate-batch1-linux-amd64 (run 38513924640), so the
hang is not specific to aarch64. It reproduces locally on a JVM standalone
(8 of 48 runs under parallel load): the test waits for a subprocess with no
pipes, so only asyncio's SIGCHLD handler can wake the event loop, but
signal.set_wakeup_fd is a no-op stub in GraalPy and the Python handler only
runs as an async action once the main thread leaves its native poll(),
which only happens when the kernel delivers the signal to that very thread.
A real set_wakeup_fd implementation is being prepared separately on
fix/signal-set-wakeup-fd.
@sgammon Sam Gammon (sgammon) mentioned this pull request Oct 14, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sgammon @steve-s @msimacek @fangerer @graemerocher