Skip to content

gh-159620: Fix remote asyncio stacks stopping at async generators - #159623

Merged
pablogsal merged 5 commits into
python:mainfrom
deadlovelll:gh-159620-remote
Oct 17, 2026
Merged

pablogsal merged 5 commits into
python:mainfrom
deadlovelll:gh-159620-remote

Conversation

@deadlovelll

Copy link
Copy Markdown
Contributor

Fix remote asyncio stacks stopping at async generators

See gh-159620

@pablogsal pablogsal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! 🤘

@pablogsal pablogsal added needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Oct 17, 2026
@pablogsal
pablogsal enabled auto-merge (squash) October 17, 2026 22:28
@pablogsal
pablogsal merged commit 2188f9b into python:main Oct 17, 2026
110 of 112 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @deadlovelll for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

@miss-islington-app

Copy link
Copy Markdown

Sorry, @deadlovelll and @pablogsal, I could not cleanly backport this to 3.14 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 2188f9b60abb7965548b1f0eac368d858edfff80 3.14

@bedevere-app

bedevere-app Bot commented Oct 17, 2026

Copy link
Copy Markdown

GH-159654 is a backport of this pull request to the 3.15 branch.

@bedevere-app

bedevere-app Bot commented Oct 17, 2026

Copy link
Copy Markdown

GH-159655 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Oct 17, 2026
pablogsal added a commit to pablogsal/cpython that referenced this pull request Oct 17, 2026
pablogsal added a commit that referenced this pull request Oct 17, 2026
…ors (GH-159623) (GH-159654)

Follow asynchronous-generator send/throw wrappers when unwinding suspended
and running asyncio tasks.
(cherry picked from commit 2188f9b)

Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
pablogsal added a commit that referenced this pull request Oct 18, 2026
…ors (GH-159623) (GH-159655)

Follow asynchronous-generator send/throw wrappers when unwinding suspended
and running asyncio tasks. Retry transient sampling errors in the existing
live-process stress test, matching main and 3.15.

(cherry picked from commit 2188f9b)

Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>
Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com>
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 CentOS9 Refleaks 3.x (tier-1) has failed when building commit 2188f9b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/840/builds/2864) and take a look at the build logs.
  4. Check if the failure is related to this commit (2188f9b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/840/builds/2864

Failed tests:

  • test.test_multiprocessing_fork.test_processes

Failed subtests:

  • test_interrupt - test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt
  • test_cprofile - test.test_profile.ProfileTest.test_cprofile

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/test_profile.py", line 70, in test_cprofile
    self.fail("\n".join(fail))
    ~~~~~~~~~^^^^^^^^^^^^^^^^^
AssertionError: 
Stats.print_stats output for Profile does not fit expectation:
--- 
+++ 
@@ -1,9 +1,7 @@
-       28   27.972    0.999   27.972    0.999 profilee.py:110(__getattr__)
-        1  269.996  269.996  999.769  999.769 profilee.py:25(testfunc)
-     23/3  149.937    6.519  169.917   56.639 profilee.py:35(factorial)
-       20   19.980    0.999   19.980    0.999 profilee.py:48(mul)
-        2   39.986   19.993  599.830  299.915 profilee.py:55(helper)
-        4  115.984   28.996  119.964   29.991 profilee.py:73(helper1)
-        2   -0.006   -0.003  139.946   69.973 profilee.py:84(helper2_indirect)
-        8  311.976   38.997  399.912   49.989 profilee.py:88(helper2)
-        8   63.976    7.997   79.960    9.995 profilee.py:98(subhelper)
+       12   11.988    0.999   11.988    0.999 profilee.py:110(__getattr__)
+        1   98.999   98.999  999.938  999.938 profilee.py:25(testfunc)
+        1   18.994   18.994  900.939  900.939 profilee.py:55(helper)
+        2   57.992   28.996   59.982   29.991 profilee.py:73(helper1)
+        1   -0.001   -0.001  671.996  671.996 profilee.py:84(helper2_indirect)
+        4  127.990   31.997  821.964  205.491 profilee.py:88(helper2)
+        3   23.991    7.997   29.985    9.995 profilee.py:98(subhelper)


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 632, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 628, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-3' pid=2218393 parent=2218387 started daemon>


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 579, in _sleep_some_event
    time.sleep(100)
    ~~~~~~~~~~^^^^^
KeyboardInterrupt
k


Traceback (most recent call last):
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/contextlib.py", line 116, in inner
    return func(*args, **kwds)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 632, in _kill_process
    self.assertEqual(join(), None)
                     ~~~~^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 305, in __call__
    return self.func(*args, **kwds)
           ~~~~~~~~~^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/process.py", line 156, in join
    res = self._popen.wait(timeout)
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/popen_fork.py", line 44, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/var/lib/buildbot/worker/cstratak-CentOS9-aarch64/3.x.cstratak-CentOS9-aarch64.refleak/build/Lib/test/_test_multiprocessing.py", line 628, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-1369' pid=2133025 parent=2096221 started daemon>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants