Repository navigation
feat(twilio): drop a reply not yet started when the caller goes on talking - #1470
Merged
yileicn merged 2 commits intoOct 30, 2026
Merged
Conversation
…lking
ConversationRelay ends the caller's turn on a pause, so a caller who
stops mid-thought ("Tomorrow." ... "Pretty open, I'm around most of the
day.") has the first half sent to the agent on its own. Until now that
turn always ran to the end: its reply was spoken, and the rest of what
the caller said was kept for the next turn, arriving as a cut-in.
Now, when more caller speech comes in while a turn has neither said
anything (a streamed piece, a filler, an indication) nor called a tool,
the session cancels it and answers what the caller said in the next
turn. The turn's words are already in the history, and its reply is
stored marked as never said ("[the caller went on talking before
anything was said]"), so the next turn reads both halves. A transfer or
hang-up the dropped turn asked for still goes ahead; keypad input never
drops a turn.
Cancelling goes through IConversationCancellationService, which the chat
providers already honor while streaming: TwilioConversationProcessor
registers each streamed turn there, and AgentTurnCallbacks.Cancel /
OnCancel connect the session to it, including a cancel that comes before
the turn could take it. So it only applies with StreamReplies on.
Known gap: a tool the model asked for in the moment before the cancel
still runs; its result is kept, and the reply after it is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
A turn is dropped only when it has neither said anything nor called a tool, but a tool call shows only when the model asks for it. A flow can also call one in code: a phone flow that fetches the confirmation number from a conversation hook, once the model has ended its reply, does that with nothing to show for it, and the caller going on talking meanwhile dropped the turn with the number in it. Such a flow already keeps the agent's words uninterrupted while that line is coming up (twilio_relay_keep_uninterrupted), so a turn it keeps uninterrupted is no longer dropped. The state is read on the turn only, which alone may touch the conversation's states: at its start through the new AgentTurnCallbacks.OnTurnStarted (called once the states are loaded), as each response starts streaming and with each piece, and with the reply. The session keeps the last value across turns, so in the moment before a turn has read it the last turn's answer holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
|
reviewed |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved cancellation and supersession races can cancel a later turn, discard replies after tool side effects, and speak superseded content.
6 open issues
What changed in this PR
This PR lets Twilio ConversationRelay cancel a pending reply when the caller resumes speaking, so the next turn can answer their combined input.
Changes:
- Track speech, tool calls, keypad input, and uninterrupted-flow state to determine cancellation eligibility.
- Connect turn callbacks to conversation streaming cancellation.
- Mark dropped replies—including empty replies—as unheard in conversation history.
| File | Description |
|---|---|
src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioConversationProcessor.cs |
Registers streaming cancellation, notifies turn start, and permits empty-reply history rewrites. |
src/Plugins/BotSharp.Plugin.Twilio/Services/ConversationRelaySession.cs |
Tracks turn eligibility, drops pending replies, and records unheard responses. |
src/Plugins/BotSharp.Plugin.Twilio/Models/AgentTurnCallbacks.cs |
Adds turn-start notification and cancellation callbacks with deferred delivery. |
Review effort: Balanced
| OnStreamStarted = stream ? StartStreamAsync : null, | ||
| // A line said before a slow tool ("Okay, that's everything I need") is spoken now, and the watchdog | ||
| // counts the silence after it instead of waiting for the next response to close it | ||
| OnStreamEndedWithToolCall = stream ? CloseUtteranceBeforeToolAsync : null, |
| if (cancellation != null) | ||
| { | ||
| cancellation.RegisterConversation(message.ConversationId); | ||
| callbacks!.OnCancel(() => cancellation.CancelStreaming(message.ConversationId)); |
Comment on lines
+327
to
+328
| return _settings.ConversationRelay.StreamReplies && _turnCallbacks != null && !_turnFromKeypad | ||
| && !_turnReplied && !_superseded && !_streamedThisTurn && !_spokeThisTurn && !_toolCalledThisTurn |
| lock (_sync) | ||
| { | ||
| superseded = _superseded; | ||
| _keptUninterrupted = reply.KeepUninterrupted; |
| continue; | ||
| } | ||
|
|
||
| await SpeakAsync(GetUnstreamedText(reply.Content), keepUninterrupted: reply.KeepUninterrupted); |
| string sent; | ||
| lock (_sync) | ||
| { | ||
| if (_superseded) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ConversationRelay ends the caller's turn on a pause. A caller who stops mid-thought ("Tomorrow." ... "Pretty open, I'm around most of the day.") has the first half sent to the agent on its own. That turn always ran to the end: its reply was spoken, and the rest of what the caller said waited for the next turn and arrived as a cut-in. On a live call this sounds like the agent answering before the caller has finished.
Change
When more caller speech comes in while the current turn has neither said anything (a streamed piece, a filler, an indication) nor called a tool,
ConversationRelaySessioncancels that turn and answers what the caller said in the next one.[the caller went on talking before anything was said], or... they did not hear: "..."when the model had written something), so the next turn reads both halves.twilio_relay_keep_uninterrupted) is never dropped: such a flow has a line the caller must hear coming (a confirmation number) and may fetch it in code from a conversation hook, where no tool call shows. The state is read on the turn only — at its start through the newAgentTurnCallbacks.OnTurnStarted, as each response starts streaming, with each piece and with the reply — and the session keeps the last value across turns, so the last turn's answer holds until the new one has read it.IConversationCancellationService, which the chat providers already honor while streaming:TwilioConversationProcessorregisters each streamed turn, andAgentTurnCallbacks.Cancel/OnCancelconnect the session to it (a cancel that arrives before the turn could take it is applied as soon as it can be). So this only applies withConversationRelay:StreamReplieson.RewriteForHistorynow also offers an empty reply toRewriteReply, so a cancelled turn with no text can still be marked.Known gap
A tool the model asked for in the moment before the cancel still runs; its result is kept, and the reply after it is dropped.
Tests
The session is covered by the relay session test suite in our downstream app (7 new cases): a reply not started yet is dropped and marked (with and without text written); a reply already going out, a turn that called a tool, a turn that does not stream and a turn the flow keeps uninterrupted are not taken back; and the last turn's keep-uninterrupted holds until the new turn has read it. That suite passes 24/24 on five runs in a row.
🤖 Generated with Claude Code