Skip to content

fix(circuit_breaker): await async coroutines inside circuit breaker failure tracking - #8703

Merged
leandrodamascena merged 4 commits into
aws-powertools:developfrom
ASHMEET555:fix/circuit-breaker-async-coroutine
Oct 29, 2026
Merged

leandrodamascena merged 4 commits into
aws-powertools:developfrom
ASHMEET555:fix/circuit-breaker-async-coroutine

Conversation

@ASHMEET555

Copy link
Copy Markdown
Contributor

Issue number: closes #8697

Summary

Adds async coroutine function support to @circuit_breaker. When wrapping an async def function, the decorator now provides an async wrapper that awaits the coroutine inside the circuit breaker's state machine, allowing qualifying failures to be caught and counted towards failure_threshold.

Changes

  • In aws_lambda_powertools/utilities/circuit_breaker/circuit_breaker.py:
    • Detect whether the decorated function is a coroutine function with inspect.iscoroutinefunction.
    • Return an async_wrapper for async functions that invokes await handler.handle_async().
    • In development disabled mode (POWERTOOLS_CIRCUIT_BREAKER_DISABLED), await the async function directly.
  • In aws_lambda_powertools/utilities/circuit_breaker/base.py:
    • Added handle_async, _call_closed_async, _call_probe_async, and _open_response_async to CircuitBreakerHandler.
    • Awaits the protected call inside the failure and probe handlers to properly transition circuit states.
    • Supports both synchronous and asynchronous on_circuit_open callbacks.
  • In tests/functional/circuit_breaker/test_circuit_breaker.py:
    • Added 8 async functional tests covering closed healthy calls, failure threshold tripping to OPEN, sync and async on_circuit_open callbacks, open circuit errors, half-open probe success/failure, and disabled environment bypass.

User experience

Decorating async def functions with @circuit_breaker now tracks failures as expected:

from aws_lambda_powertools.utilities.circuit_breaker import circuit_breaker, CircuitBreakerConfig

config = CircuitBreakerConfig(failure_threshold=2)

@circuit_breaker(name="payment-service", persistence_store=store, config=config)
async def call_payment():
    raise ConnectionError("Service unavailable")

# After 2 awaited failures, the circuit transitions from CLOSED to OPEN:
await call_payment()

Verification

  • uv run pytest tests/functional/circuit_breaker/ tests/unit/circuit_breaker/ --cov=aws_lambda_powertools.utilities.circuit_breaker --cov-report=term-missing — 93 passed, 95.01% coverage.
  • uv run ruff check aws_lambda_powertools/utilities/circuit_breaker/ tests/functional/circuit_breaker/ — Passed.
  • uv run ruff format --check aws_lambda_powertools/utilities/circuit_breaker/ tests/functional/circuit_breaker/ — Passed (14 files already formatted).
  • uv run mypy aws_lambda_powertools/utilities/circuit_breaker/ — Passed (no issues found in 10 source files).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@ASHMEET555
ASHMEET555 requested a review from a team as a code owner October 29, 2026 16:24
@ASHMEET555
ASHMEET555 requested a review from svozza October 29, 2026 16:24
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 29, 2026
@boring-cyborg boring-cyborg Bot added the tests label Oct 29, 2026
…sync paths

_route() decides CLOSED/PROBE/OPEN once; _on_closed_*/_on_probe_* hold the
bookkeeping; the sync and async call methods only differ in how they invoke the
protected function. Also dedupes the decorator wrappers and documents async support.
@boring-cyborg boring-cyborg Bot added the documentation Improvements or additions to documentation label Oct 29, 2026
@leandrodamascena leandrodamascena added this to the 3.39.0 milestone Oct 29, 2026

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ASHMEET555, good work, and fast. I tested it against a real DynamoDB table: on develop an async def never trips the circuit, with your change it opens on the second failure, awaits the async callback and closes again after the probe.

One thing I changed, and I pushed it to your branch so you can see it: the PR copied handle, _call_closed and _call_probe into async versions, which means two copies of the state machine to keep in sync. I split the routing into a single _route() and moved the bookkeeping into small _on_closed_* / _on_probe_* helpers, so the sync and async paths only differ in the await. Your tests pass unchanged. I also added a short docs section with an async example.

Approving, this goes out in 3.39.0, credit is yours in the release notes.

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.00%. Comparing base (13dbc55) to head (6bcc9f3).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8703      +/-   ##
===========================================
+ Coverage    96.99%   97.00%   +0.01%     
===========================================
  Files          321      321              
  Lines        16332    16415      +83     
  Branches      1473     1482       +9     
===========================================
+ Hits         15841    15924      +83     
  Misses         350      350              
  Partials       141      141              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…call the circuit concurrently

The probe lease is per thread, so concurrent coroutines in one event loop all passed
the owner check and probed together. Track in-flight probes per circuit name and give
the other callers the open response, matching the single-prober guarantee threads
already had.
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2026
@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ASHMEET555 I pushed two more commits, take a look when you can.

While testing I noticed the probe owner id lives in thread-local storage. Coroutines share the thread, so when I ran asyncio.gather with 20 calls on a circuit that was due for a probe, all 20 hit the backend and the circuit closed on the first 3 successes. Threads don't have this problem because each one gets its own id.

I fixed it by tracking which circuits have a probe running in the process. The first coroutine probes, the rest get the open response. Added two tests for it. Sync code paths are untouched, I ran the develop test file against the new code to be sure.

@leandrodamascena
leandrodamascena merged commit ec1bec2 into aws-powertools:develop Oct 29, 2026
20 of 21 checks passed
@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@ASHMEET555

Copy link
Copy Markdown
Contributor Author

Thank @leandrodamascena! I really appreciate the detailed review, testing against real DynamoDB, and the clean refactoring.

Extracting _route() and the bookkeeping helpers is much cleaner than duplicating the state machine definitely a great pattern for keeping sync/async logic unified.

Also, the catch on thread-local storage with concurrent coroutines in asyncio.gather is fantastic. Because coroutines share the same thread and identical probe owner ID, in-process probe tracking was the exact right fix to prevent probe stampedes while keeping thread isolation intact.

Thanks for refining this and getting it over the finish line for 3.39.0! Looking forward to contributing more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: @circuit_breaker decorator does not track exceptions raised by async coroutines

2 participants