fix(billing): stop free-plan workflow runs at their usage limit mid-run by waleedlatif1 · Pull Request #9207 · simstudioai/sim

waleedlatif1 · 2026-10-18T22:55:21Z

Summary

Workflow runs were only checked against the usage limit at admission, and a run's cost reaches the ledger only when its log is written, so one free-plan Agent tool loop (or a Loop of Agent blocks) could keep calling hosted models long past the limit, unbilled
Every settled model step's billed cost (BYOK steps count $0) now adds to a run-wide unrecorded total, shared with child workflows and workflow tools billed on the same log; custom-block children keep their own
Before every model call (prepareConversationGeneration, which every provider loop already calls) and again after a compaction summary, a free-plan run re-judges its payer through the existing readMidRunUsageVerdict, now counting the run's unrecorded spend; it reads nothing until the run has spent and uses the cached execution gate
On the limit the run stops with RunUsageLimitError (non-retryable, attributed to the user so it logs at info), carrying what success would have billed: the request's settled steps, successful tool results and failed Function runs, compaction summaries, and journaled prior attempts (a continued pending call is billed once, through the journal). Providers already pass it through unwrapped via isConversationContextError, so fallbacks and block retries don't fire
An unreadable ledger or blocked account lets the run continue and isn't re-read for the near-limit TTL; paid plans (overage is billed), self-hosted, cancelled runs and calls with no workflow context are never read
Copilot callers of readMidRunUsageVerdict pass no spend and are unchanged
providerFailureLogLevel and the streaming tool loops now share one severity rule: a run abort first (timeout warn, Stop info), then classifyFailure attribution

Type of Change

Bug fix

Testing

providers/run-usage-limit.test.ts (21 failure-mode tests through the real hooks, executeProviderRequest and verdict code), plus compaction, workflow-tool and workflow-handler child-sharing tests; each guard reverted to confirm its test goes red
Provider, executor, agent/workflow handler and memory suites (2,537 tests)
bun run lint, check:comment-hygiene, check:test-patterns; type-check and check:audits left to CI

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-10-18T22:55:26Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 19, 2026 3:10am UTC

cubic-dev-ai

All reported issues were addressed across 12 files

_{This PR changes payment, billing, or quota logic. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-18T23:03:13Z

Confidence Score: 5/5

[Critical impact] The PR appears safe to merge; no new blocking issue remains.

Summary

Stops free-plan Agent runs before another model call once recorded usage plus the run’s unsettled model spend reaches the limit.

Free-plan runs stop before a model call that reaches their usage limit.
A run's unsettled spend includes model work and costs from children billed on its log.
Streaming tool loops and block streams log user stops at the right severity.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Settled model step] --> B[Add cost to request and shared run total]
  B --> C[Before next model call]
  C --> D{Free-plan run has spent?}
  D -- No --> G[Continue]
  D -- Yes --> E[Read recorded usage plus run total]
  E --> F{Limit reached?}
  F -- No --> G
  F -- Yes --> H[Stop without retry]
  H --> I[Bill completed request work]

_{Reviews (7) · Last reviewed commit: "fix(billing): charge a continued Functio..." · Reviewed by Greptile}

waleedlatif1 · 2026-10-18T23:10:00Z

@greptile

waleedlatif1 · 2026-10-18T23:10:01Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-18T23:10:05Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 13 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-18T23:23:03Z

@greptile

waleedlatif1 · 2026-10-18T23:23:04Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-18T23:23:09Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-18T23:35:22Z

@greptile

waleedlatif1 · 2026-10-18T23:35:23Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-18T23:35:29Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

…e usage stop

…spend with workflow tools

…ntinued calls once - RunUsageLimitError joins the errors every provider already rethrows unwrapped, replacing the per-request side channel that could mask an unrelated later failure - The stop is attributed to the user, so provider, stream-loop and stream-read logs record it at info; a run abort is classified before attribution so a run timeout still logs at warn - A pending call continued from an earlier attempt is billed through the journal only - The stop bills tool results the way success does: successful results and failed Function runs

waleedlatif1 · 2026-10-19T02:48:03Z

@greptile

waleedlatif1 · 2026-10-19T02:48:04Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-19T02:48:10Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 25 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

… limit once - A turn an earlier process journaled adds its model spend to the run's unrecorded total the first time this run opens it; reopening the turn (fallback model, block retry) never counts it again - Type the continued-call test's tool as a ProviderToolConfig

waleedlatif1 · 2026-10-19T03:00:15Z

@greptile

waleedlatif1 · 2026-10-19T03:00:16Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-19T03:00:22Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 26 files

Confidence score: 3/5

In apps/sim/providers/index.ts, a failed function_execute replay that proceeds to a model call is charged twice. Clear the failed-function accumulator before adding priorConversationUsage.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/providers/index.ts">

<violation number="1" location="apps/sim/providers/index.ts:353">
P1: A failed `function_execute` replay is charged twice when this request continues to a model call: once through `failedFunctionToolCost` and again through `priorConversationUsage`. Clear the failed-function accumulator after replaying pending calls, before the current provider response is assembled.

(Based on your team's feedback about journaled agent spend.)</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

…h the journal

waleedlatif1 · 2026-10-19T03:10:44Z

@greptile

waleedlatif1 · 2026-10-19T03:10:46Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-19T03:10:53Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 26 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts

Comment thread apps/sim/providers/conversation-history.ts

Comment thread apps/sim/providers/conversation-generation.ts

greptile-apps Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts Outdated

Comment thread apps/sim/providers/conversation-generation.ts

Comment thread apps/sim/providers/conversation-history.ts

waleedlatif1 force-pushed the fix/workflow-mid-run-usage-cap branch from 56d9599 to 2d47822 Compare October 18, 2026 23:09

vercel Bot temporarily deployed to Preview October 18, 2026 23:09 Inactive

cubic-dev-ai Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts Outdated

greptile-apps Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread apps/sim/providers/conversation-generation.ts

Comment thread apps/sim/executor/handlers/workflow/handler.ts

waleedlatif1 force-pushed the fix/workflow-mid-run-usage-cap branch from 2d47822 to e10de80 Compare October 18, 2026 23:22

vercel Bot temporarily deployed to Preview October 18, 2026 23:22 Inactive

cubic-dev-ai Bot reviewed Oct 18, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 18, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts Outdated

waleedlatif1 force-pushed the fix/workflow-mid-run-usage-cap branch from e10de80 to ebabfe5 Compare October 18, 2026 23:35

vercel Bot temporarily deployed to Preview October 18, 2026 23:35 Inactive

cubic-dev-ai Bot reviewed Oct 18, 2026

View reviewed changes

waleedlatif1 added 5 commits October 18, 2026 19:31

fix(billing): stop free-plan workflow runs at their usage limit mid-run

81dd823

fix(billing): recheck after compaction, bill settled tool costs on th…

59bd395

…e usage stop

fix(billing): bill compaction summaries on the usage stop, share run …

f6334e8

…spend with workflow tools

fix(billing): bill journaled attempts on the usage stop

f192164

waleedlatif1 force-pushed the fix/workflow-mid-run-usage-cap branch from ebabfe5 to 9d7aa68 Compare October 19, 2026 02:47

vercel Bot temporarily deployed to Preview October 19, 2026 02:47 Inactive

cubic-dev-ai Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts

Comment thread apps/sim/providers/index.ts

greptile-apps Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread apps/sim/providers/run-usage-limit.ts

vercel Bot temporarily deployed to Preview October 19, 2026 02:59 Inactive

cubic-dev-ai Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread apps/sim/providers/index.ts

fix(billing): charge a continued Function call's failure once, throug…

ec98c11

…h the journal

vercel Bot temporarily deployed to Preview October 19, 2026 03:10 Inactive

cubic-dev-ai Bot reviewed Oct 19, 2026

View reviewed changes

waleedlatif1 merged commit c0cf608 into staging Oct 19, 2026
53 checks passed

waleedlatif1 deleted the fix/workflow-mid-run-usage-cap branch October 19, 2026 03:21

waleedlatif1 mentioned this pull request Oct 19, 2026

v0.9.27: tools expansion, aeo and db query improvements #9222

Merged

Conversation

waleedlatif1 commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

waleedlatif1 commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 18, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 19, 2026

Uh oh!

waleedlatif1 commented Oct 19, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 19, 2026

Uh oh!

waleedlatif1 commented Oct 19, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

waleedlatif1 commented Oct 18, 2026 •

edited

Loading

vercel Bot commented Oct 18, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 18, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading