Add external body storage for audit by rbev · Pull Request #6029 · Particular/ServiceControl

rbev · 2026-10-20T06:20:26Z

Problem

Audit's SQL Server and PostgreSQL persisters discard oversized and binary message bodies. Oversized text retains a searchable prefix, but the body API cannot return the complete body. Error's SQL persisters already support external storage for these bodies.

Change

Reuse Error's FileSystem, Azure Blob, and S3 providers for Audit by extracting them into ServiceControl.Persistence.BodyStorage.

Keep small valid text bodies inline. Store oversized, binary, content-encoded, non-UTF-8, and NUL-containing bodies externally.
Retain a UTF-8-safe inline search prefix for oversized text and serve the complete body through the body API.
Await external writes before inserting message rows, including when disposing an abandoned ingestion batch.
Delete external bodies before removing expired rows or partitions. A storage failure leaves the database records available for the next sweep.
Provision storage during setup and register the existing disk-space custom check for FileSystem storage.
Update persistence manifests, test fixtures, development configuration, and the testing tool.

RavenDB behavior is unchanged. Existing TooLarge and NotText rows retain their previous API responses; discarded bodies cannot be recovered. The new external-body state uses the existing integer column, so no schema migration is needed.

Required configuration

External storage is required for Audit SQL Server and PostgreSQL instances. Existing instances must add ServiceControl.Audit/MessageBody/StorageType and the selected provider's settings before restarting after an upgrade.

<add key="ServiceControl.Audit/MessageBody/StorageType" value="FileSystem" />
<add key="ServiceControl.Audit/MessageBody/FileSystem/StoragePath" value="D:\ServiceControl.Audit\bodies" />

MaxBodySizeToStore becomes the inline threshold, not a limit on the complete stored body. Azure defaults to the audit-bodies container; S3 defaults to the audit-bodies/ key prefix.

Each instance must use a separate directory, container, or key prefix. Sharing a location can let one instance's retention process delete another instance's bodies.

Design constraints

Sharing the providers avoids duplicating their storage formats, compression, authentication, and stream handling. Error retains its existing configuration and behavior.

Audit retention deletes by hourly prefixes on SQL Server and daily prefixes on PostgreSQL. This preserves PostgreSQL's partition-drop path without scanning message rows. SQL Server only visits hours containing rows, so bodies orphaned by failed inserts into otherwise empty hours can require manual cleanup.

The current design and configuration are documented in Message body storage.

…into audit-external-bodies

warwickschroeder · 2026-10-23T04:14:03Z

 | `--tag <tag>` | Test a specific ServiceControl image tag, for example a PR prerelease like `pr-1234` |
 | `--persistence <type>` | Persistence for the error instance: `RavenDb` (default), `SqlServer`, or `PostgreSql` |
 | `--error-ingestion-scale-unit <n>` | Start `n` additional error-ingestion-only ServiceControl instances |
+| `--azureblob-body-storage "<connection string>"` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only to `SqlServer` and `PostgreSql`. The error instance and its scale-out units share one container, and each audit instance gets its own |


is this missing a "-" between azure and blob?

I was treating it as a single noun, you are right it might be more "correct" to have a dash.

warwickschroeder · 2026-10-23T04:46:51Z

+
+External storage is required. The instance fails to start when `MessageBody/StorageType` is missing, which includes existing SQL Server and PostgreSQL Audit instances that upgrade without adding it. Setup creates the directory, container, or bucket.
+
+Settings use the `ServiceControl.Audit/` prefix in the application settings file. The validation matches the Error instance, whose settings use the `ServiceControl/` prefix.


Need to add to docs PR, if not already

warwickschroeder · 2026-10-23T04:50:48Z

+| `MessageBody/MinCompressionSize` | all, minimum body size in bytes to compress | 4096 |
+| `MessageBody/FileSystem/StoragePath` | `FileSystem`, required | none |
+| `MessageBody/FileSystem/DataSpaceRemainingThreshold` | `FileSystem`, free space percentage below which the custom check fails | 15 |
+| `MessageBody/Azure/ConnectionString` | `AzureBlob`, shared key or SAS | none |


So we are only supporting authentication via SAS token? What about Managed Identity or other Entra identities?

warwickschroeder · 2026-10-23T04:51:40Z

@@ -0,0 +1,79 @@
+# Message body storage


Any ADRs worth extracting for the message body design?

warwickschroeder · 2026-10-23T05:00:32Z

+            var storage = new RecordingBodyStorage();
+            await configuration.CreateRetention(storage).SweepNow();
+
+            Assert.That(storage.DeletedPrefixes, Has.Some.Matches<string>(bodyId.StartsWith));


Has.Some.Matches(bodyId.StartsWith) also passes for a day prefix, or "2", which would delete every body still in retention.

warwickschroeder · 2026-10-23T05:02:14Z

+                return external is null
+                    ? MessageBodyView.NoContent()
+                    : MessageBodyView.FromStream(external.Stream, external.ContentType, external.BodySize, version);


When the row says External but storage has nothing, this answers 204 with no log. A LogWarning with the body id and row here would make it traceable.

warwickschroeder · 2026-10-23T05:06:12Z

+        await foreach (var blob in container.GetBlobsAsync(BlobTraits.None, BlobStates.None, prefix, cancellationToken: cancellationToken))
+        {
+            await container.GetBlobClient(blob.Name).DeleteIfExistsAsync(cancellationToken: cancellationToken);
+        }


This deletes one blob per round trip, in sequence. An hour of audit traffic with binary bodies can be hundreds of thousands of blobs, so the sweep can take as long as the hour it is clearing, and the rows (and on PostgreSQL the partition drop) wait behind it. S3 already deletes 1,000 per call.

BlobBatchClient.DeleteBlobsAsync (256 per batch) would keep Azure in the same range.

warwickschroeder · 2026-10-23T05:08:29Z

+        foreach (var filePath in Directory.EnumerateFiles(StoragePath, $"{prefix}*.body"))
+        {
+            cancellationToken.ThrowIfCancellationRequested();
+            if (Path.GetFileName(filePath).StartsWith(prefix, StringComparison.Ordinal))
+            {
+                TryDelete(filePath);
+            }
+        }


Bodies are one flat directory, and this enumeration reads every file in it to find one hour's worth. With a month of audit bodies that's tens of millions of entries per hourly sweep.

Would per-hour subdirectories ({StoragePath}/{yyyyMMddHH}/{guid}.body) work for audit, so the delete becomes Directory.Delete?

Exactly, here is something I prepared earlier - https://github.com/Particular/ServiceControl/pull/5318/changes#diff-4d7b8cefc9a08fc2d1c1480d29b16e47cbf4d4c926dcbb4b6f0c54a3355c0c26R162-R181

warwickschroeder · 2026-10-23T05:11:16Z

        catch (RequestFailedException ex) when (ex.Status is (int)HttpStatusCode.Conflict or (int)HttpStatusCode.PreconditionFailed)
        {
            // A blob for this immutable body already exists.
        }


This will swallow every 409 and 412 as "already exists". 409 is also ContainerBeingDeleted and ContainerDisabled, so writes during a container recreate succeed silently and audit rows point at bodies that were never stored.

Filtering on ex.ErrorCode is BlobErrorCode.BlobAlreadyExists or BlobErrorCode.ConditionNotMet would keep the intended case.

johnsimons · 2026-10-23T06:19:07Z

So this would be a new feature?
Because if I correctly recall, Raven does the same, correct?

johnsimons · 2026-10-23T06:20:48Z

So MaxBodySizeToStore is the setting that controls whether it is stored or not?

ramonsmits · 2026-10-26T17:11:06Z

+
+Ingestion writes external bodies before the batch transaction that inserts the rows. A failed write fails the batch, and no row is inserted. A failed insert can leave an unreferenced body.
+
+A body is stored under its audit body id, `yyyyMMddHH-{unique message id}`, the same id used in the body URL. Storage keeps the first write of an id and ignores later writes.


Why is this prefixed with yyyyMMddHH-?

When we store audit message we generate a new uuid, can't we use uuid v7 so that these already contain a timestamp?

Based on the retention section it seems this is done for cleanup based on the retention window?

ramonsmits · 2026-10-26T17:12:27Z

+
+The unit differs by provider, so the prefix is part of `IAuditPartitionManager`:
+
+- SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`.


Is that internval configurable? Will it to chunks? Is that a native SQL server feature or an application query?

ramonsmits · 2026-10-26T17:16:12Z

+- SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`.
+- PostgreSQL drops whole day partitions without scanning rows. The prefix is the day, `yyyyMMdd`, and it matches every hour of that day. A day is only dropped once all of it is older than the retention period, so no body is removed earlier than its row would be.
+
+Prefixes must not be empty and must not contain wildcard or path characters. SQL Server only visits hours that have rows, so an unreferenced body in an hour with no rows stays until that location is cleaned manually. PostgreSQL visits every expired day partition, including pre-created partitions without rows, so it also cleans unreferenced bodies of those days.


How to the other implementation deal with retention?

ramonsmits · 2026-10-26T17:17:08Z

+
+An invalid value, or missing settings for the chosen type, still fails at startup. Setup creates the directory, container, or bucket.
+
+Audit settings use the `ServiceControl.Audit/` prefix in the application settings file. Provider-specific validation matches the Error instance, whose settings use the `ServiceControl/` prefix.


Is it a conscious decision that primary and audit do not share the same body store?

Yes, the problem with sharing the large message storage is that you can no longer clean up unused bodies without cooperation between the instances that have different retention windows.

The implementation allows them to share the same storage account, but the bodies themselves are not shared.

ramonsmits · 2026-10-26T17:19:13Z

+<add key="ServiceControl.Audit/MessageBody/S3/KeyPrefix" value="audit-instance-1/" />
+```
+
+## One location per instance


I miss decisions like body compression and encryption

johnsimons · 2026-10-26T23:57:26Z

@rbev the delimiter for the prefix needs to be set by the implementation, example for Azure Blobs, we want the prefix to be HOUR_TIMESTAMP/, for file system we want hourly folder structures ( HOUR_TIMESTAMP ) and for S3 it may just be HOUR_TIMESTAMP-.

Have a look at my implementation of this in https://github.com/Particular/ServiceControl/pull/5318/changes#diff-985fb65dab014cdcb60c6d03ef95370d1059107f6887c478fdd2c175f9decab1 and https://github.com/Particular/ServiceControl/pull/5318/changes#diff-44f054b50a09b2ab13d8985465c7db9b57fd103451e40fc3030ba889c28624e7R24.
The trick is that you need to pass the DateTime createdOn 😉

Add external body storage for audit

0b9b1b9

rbev requested review from abparticular, johnsimons and warwickschroeder October 20, 2026 06:20

rbev self-assigned this Oct 20, 2026

rbev added 2 commits October 20, 2026 14:53

Add load to trigger external body storage

9ddc908

Allow running with azure body storage

1e0a965

rbev marked this pull request as ready for review October 21, 2026 05:29

rbev added 5 commits October 21, 2026 13:30

Merge branch 'master' of https://github.com/Particular/ServiceControl …

5c4baca

…into audit-external-bodies

Add s3 large body support to test tool

0a86806

Fix persistence test compilation after body storage changes

bb1b459

Add body storage to audit tests

157be91

Don't fail the test on failed cleanup

fbe726b

warwickschroeder requested changes Oct 23, 2026

View reviewed changes

Make external storage optional in audit

db6cc41

ramonsmits reviewed Oct 26, 2026

View reviewed changes


		External storage is required. The instance fails to start when `MessageBody/StorageType` is missing, which includes existing SQL Server and PostgreSQL Audit instances that upgrade without adding it. Setup creates the directory, container, or bucket.

		Settings use the `ServiceControl.Audit/` prefix in the application settings file. The validation matches the Error instance, whose settings use the `ServiceControl/` prefix.


		Ingestion writes external bodies before the batch transaction that inserts the rows. A failed write fails the batch, and no row is inserted. A failed insert can leave an unreferenced body.

		A body is stored under its audit body id, `yyyyMMddHH-{unique message id}`, the same id used in the body URL. Storage keeps the first write of an id and ignores later writes.


		The unit differs by provider, so the prefix is part of `IAuditPartitionManager`:

		- SQL Server deletes rows one hour at a time. The prefix is the hour, `yyyyMMddHH-`.


		An invalid value, or missing settings for the chosen type, still fails at startup. Setup creates the directory, container, or bucket.

		Audit settings use the `ServiceControl.Audit/` prefix in the application settings file. Provider-specific validation matches the Error instance, whose settings use the `ServiceControl/` prefix.

Conversation

rbev commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Change

Required configuration

Design constraints

Uh oh!

warwickschroeder Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

johnsimons commented Oct 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnsimons commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rbev commented Oct 20, 2026 •

edited

Loading

warwickschroeder Oct 23, 2026 •

edited

Loading