Repository navigation
asyncio: Add guest mode for running asyncio inside external event loops #145342
Description
Feature request
Add guest mode to asyncio, enabling it to run cooperatively inside a host event loop such as Tkinter, Qt, GTK, or any other framework that owns the main thread.
Motivation
GUI toolkits (Tk, Qt, GTK, pygame, ...) each have their own event loop that must run on the main thread. Today there is no standard way to use asyncio coroutines inside a GUI application without:
- Polling hacks (periodic
root.after()calls) - Replacing the event loop entirely (fragile, framework-specific)
- Running asyncio in a separate thread and manually marshalling results
Trio solved this problem with trio.lowlevel.start_guest_run() and the approach has proven robust across many GUI frameworks (see trio-guest).
Proposed API
import asyncio
task = asyncio.start_guest_run(
async_fn, *args,
run_sync_soon_threadsafe=host.schedule_callback,
done_callback=host.on_done,
)The implementation decomposes BaseEventLoop._run_once() into three new public methods — poll_events(), process_events(), and process_ready() — then uses a dual-thread architecture where the host thread processes callbacks and a daemon I/O thread polls for events.
Prior art and discussion
- Trio guest mode
- trio-guest — multi-framework reference implementation
- asyncio-guest — proof-of-concept for asyncio (this proposal is derived from it)
- discuss.python.org: Connecting asyncio and tkinter event loops
A reference implementation with tests (12 tests passing, full asyncio test suite regression-free) is ready to submit as a PR.
Activity
pythongh-145342: asyncio: Add guest mode for running inside external …
pythongh-145342: asyncio: Add docs and NEWS entry for guest mode
x42005e1f commentedon Mar 4, 2026
Note that there is already a guest mode inspired by Trio: oremanj/aioguest. There is also my gist, which works without greenlets (and which I am currently refactoring/improving/etc. to support Curio and release as a standalone package). If you want to implement a reliable guest mode, you should also consider related issues such as #85093 (and also handle the wakeup fd, otherwise you will have issues with signal handlers).
x42005e1f commentedon Mar 4, 2026
I believe that this is insufficient, especially in terms of performance. Let us suppose we have a set of tasks, and at least one of them is active at any point in time. In this case, the timeout will always be zero. But since in your model, events are always polled by a separate thread, even a zero timeout (even with a single task that simply does
asyncio.sleep(0)in an infinite loop) will result in quite expensive ping-ponging between the two threads, and we will also lose determinism between guest tasks and host tasks.The problem can be solved by providing a method for obtaining an effective timeout and delegating polling to a separate thread only when it is non-zero. The entire
poll_events()code before theselect()call is responsible for this.However, these methods are too low-level. Any guest mode implementation will have to repeat almost the entire
asyncio.Runner.run()(both thread-local state management and event loop shutdown; I do not see the latter or evenloop._run_forever_setup()/loop._run_forever_cleanup()in the linked PR).seifertm commentedon May 15, 2026
The feature request and the thread on discuss.python.org both emphasize the usefulness of an asyncio guest mode in the context of GUI applications. I'd like to highlight the benefits for another use case, which is application testing.
For example, a developer of an ASGI web application may want to start a web server as part of their test suite and issue requests against it. These requests would typically be distributed over several test cases. Since there can only be one async event loop per thread, the same event loop running the web server will also have to run the test cases. This weakens the isolation between test cases and can lead to side effects through the shared event loop. Starting the webserver in a dedicated thread or process instead of a shared loop can introduce a lot of complexity (mixed concurrency approaches, signal handling, …) or may not be an option at all due to the lack of thread-safety of the test runner (e.g. pytest).
In this example, the proposed asyncio guest mode would enable stronger isolation between individual test cases and between tests and the web server. Other approaches, such as patching an asyncio event loop with the nest_asyncio package, are not able to provide the same guarantees.
pythongh-145342: Add _guest_mode flag and guest-mode signal-handler g…
pythongh-145342: Make the guest I/O thread non-daemonic with graceful…
pythongh-145342: Expand asyncio guest mode tests
pythongh-145342: Update asyncio guest mode docs and NEWS
pythongh-145342: Adapt guest mode to current main and fix docs CI
pythongh-145342: Reference the asyncio-guest example collection in th…
pythongh-145342: Add _guest_mode flag and guest-mode signal-handler g…
pythongh-145342: Make the guest I/O thread non-daemonic with graceful…
pythongh-145342: Expand asyncio guest mode tests
pythongh-145342: Update asyncio guest mode docs and NEWS
pythongh-145342: Adapt guest mode to current main and fix docs CI
pythongh-145342: Reference the asyncio-guest example collection in th…
pythongh-145342: Avoid the I/O thread round trip for non-blocking polls
congzhangzh commentedon Oct 13, 2026
@x42005e1f Sorry for the very late reply here. Both of your comments shaped the current version of #145343, so here is where each point stands.
Ping-pong with zero timeouts. Fixed in 42c8f71763, essentially as you suggested.
poll_events()is now split into a private_poll_timeout()(the code beforeselect()) and theselect()call. After each batch the host computes the timeout and does a non-blockingselect(0). If the timeout is zero or events are pending, the next batch is scheduled on the host itself; the I/O thread is only used for polls that would block. Busy phases therefore no longer involve the thread at all, and guest steps stay ordered with host callbacks through the host's own queue. On a--with-pydebugbuild with a mock host, asleep(0)loop went from 4.6× to 1.2–1.4× the cost ofasyncio.run(), and a socketpair ping-pong from 3.6× to 1.1×. A test asserts that 1000sleep(0)steps cause fewer than 10 I/O-thread polls (it saw 1005 before the change).Too low-level / repeating
Runner.run(). The guest run now usesloop._run_forever_setup()/_run_forever_cleanup()instead of hand-rolled state management. When the main task finishes, it does the same shutdown asasyncio.run()(_cancel_all_tasks(),shutdown_asyncgens(),shutdown_default_executor(),close()) before callingdone_callback. I've also offered to makepoll_events()/process_events()/process_ready()private, so the only new public API would bestart_guest_run().Wakeup fd. The guest loop never takes over
signal.set_wakeup_fd(): on Windows the host's fd is saved and restored around the proactor's__init__andclose(), andadd_signal_handler()/remove_signal_handler()raiseRuntimeErrorin guest mode.Async generator hooks (gh-85093). This one is only partly solved, and I'd like your opinion.
_run_forever_setup()installs the guest's asyncgen hooks for the whole run, because the guest loop is the thread's running loop for the whole run. That is fine for GUI hosts, which don't use asyncgen hooks. But a host that does (e.g. Trio as the host) would have its hooks replaced, and async generators first iterated by host code would be finalized by the guest loop. The alternative is what you described on the PR: swap the hooks (and the running loop) in and out around each host batch. That isolates the host, butget_running_loop()would then be unset between batches. Do you think per-batch swapping is worth it for the first version, or is documenting the limitation acceptable?Thanks also for the gist. If your standalone package handles cases the PR doesn't, I'd be glad to cross-check against it.
pythongh-145342: Swap per-thread loop state around guest batches
x42005e1f commentedon Oct 13, 2026
Perhaps it depends on what the maintainers decide. Personally, I would like the standard implementation to be at least as good as the others (such as Trio's guest mode or oremanj/aioguest, both of which handle outside async generators; see
trio/_core/_asyncgens.pyas an example), because otherwise there will be a temptation to use alternatives. But alas, we do not have many opinions on this.congzhangzh commentedon Oct 13, 2026
Follow-up on the async generator question: I went ahead with per-batch swapping, in ff3800b10a. The running loop, the async generator hooks and the coroutine origin tracking depth are now switched to the guest's around each host batch and restored afterwards.
loop.is_running()stays true for the whole run.run_sync_soon_threadsafeinstead of running insidestart_guest_run(), because a guest task cannot start while a host task is running (enter_taskchecks this). Soasync_fnstarts afterstart_guest_run()returns; Trio likewise runs no user code before itsstart_guest_run()returns.loop.set_debug()toggling origin tracking, are carried over to later batches.The cost is a few C calls per batch: roughly +2 µs per
sleep(0)step and +8 µs per socket round trip on a debug build. If you see a hole in this, I'd be glad to hear it.