Prompt cache is not saved when a request ends on a stop sequence or disconnect (regression from b620ae8 bump) #245

Summary

After the mlx-swift-lm bump to b620ae8 (#243, merged as 57ea94b), the prompt KV cache is saved only on the producer's .info event. A request that ends early, either on a text-level stop sequence or on a client disconnect, never reaches .info, so its prompt is not saved. The next request that shares the prefix re-prefills the whole prompt.

Before the bump, the save ran on the first decode token, so an early stop still kept the prompt.

Where (at 57ea94b)

• Sources/SwiftLM/Server.swift:3418: the streaming path saves in case .info, and the comment there says an early exit "never reaches .info and skips the save".
• Sources/SwiftLM/Server.swift:3594: the non-streaming path has the same behavior at the stop-sequence break.

Impact

• Any request that stops on a stop sequence or a disconnect leaves no prompt cache entry. The next turn of that conversation re-prefills the full history, so the cost grows with conversation length.
• Not verified: whether the built-in stop strings (<|im_end|>, <end_of_turn>, and others) fire in practice before the EOS token id. If they do, most ordinary turns are affected, not only aborted ones.
• Not measured: the time-to-first-token cost.

Suggested direction

Save the prompt on the first decode token again, or add a save on the stop and disconnect paths. The .info move exists to avoid a race with the decode step that is still writing the same KVCacheSimple, so any fix has to keep that race closed.

References

• chore: bump mlx-swift-lm to b620ae8 #243 chore: bump mlx-swift-lm to b620ae8, merged as 57ea94b.