diff --git a/docs/README.md b/docs/README.md index 690f42fa00..cf6104c7d6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,6 +24,7 @@ This section points to sources that explain why ServiceControl is designed the w - [Ingestion pipeline](ingestion-pipeline.md) — why batch parallelism is a storage decision, not an instance decision - [Error ingestion design](error-ingestion-design.md) — why the relational persisters write failed messages with hand-written SQL +- [Message body storage](message-body-storage.md) — where the Audit SQL Server and PostgreSQL persisters keep message bodies, how retention removes them, and which settings are optional - [Bulk retries design](bulk-retries-design.md) — how ServiceControl retries failed messages in bulk - [Retries over Azure Storage Queues transport](retries-asq-transport.md) — how retries behave with one or several storage accounts - [Data versioning design](data-versioning-design.md) — which invariant keeps a cached API response from going stale diff --git a/docs/load-testing.md b/docs/load-testing.md index a16ac48585..7ed99b96f1 100644 --- a/docs/load-testing.md +++ b/docs/load-testing.md @@ -21,6 +21,11 @@ Useful flags, passed after `--`: | `--tag ` | Test a specific ServiceControl image tag, for example a PR prerelease like `pr-1234` | | `--persistence ` | Persistence for the error instance: `RavenDb` (default), `SqlServer`, or `PostgreSql` | | `--error-ingestion-scale-unit ` | Start `n` additional error-ingestion-only ServiceControl instances | +| `--azure-blob-body-storage ""` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only to `SqlServer` and `PostgreSql`. The error instance and its scale-out units share one container, and each audit instance gets its own | +| `--s3-body-storage ` | Store message bodies in S3 instead of the container file system. Applies only to `SqlServer` and `PostgreSql`, and is mutually exclusive with `--azure-blob-body-storage`. The error instance and its scale-out units share the `error-bodies/` prefix, and each audit instance gets its own | +| `--s3-region ` | Region for `--s3-body-storage` | +| `--s3-service-url ` | S3-compatible endpoint, such as MinIO or LocalStack. It must be reachable from inside the containers, so `localhost` does not work | +| `--s3-access-key-id `, `--s3-secret-access-key ` | Static credentials, set both or neither. Without them, the containers use the default AWS credential chain; host credentials are not forwarded | ## Run against an existing ServiceControl @@ -35,7 +40,7 @@ Open http://localhost:5290. The tool defaults to a ServiceControl instance at `h Everything is controllable from the web UI, or via the HTTP API: -- Scenarios: the five built-in failure scenarios (`third-party-outage`, `timeout-spike`, `poison-message`, `deserialization-failure`, `background-noise`), each producing errors that group naturally. Start with `POST /api/scenarios/{name}/start` (optional `rate` and `durationSeconds`). +- Scenarios: the six built-in scenarios (`third-party-outage`, `timeout-spike`, `poison-message`, `deserialization-failure`, `background-noise`, `large-messages`), each producing errors that group naturally. `large-messages` starts at 3 messages per second unless `rate` is set, sends bodies of about 256 KiB (the wire envelope is slightly larger), fails 20% of them to the error queue, and audits the rest. The bypass writer fails every message by design, so it sends the large bodies only as failures. Start with `POST /api/scenarios/{name}/start` (optional `rate` and `durationSeconds`). - Bypass writer: high-throughput load that writes failed-message envelopes directly to the ServiceControl error queue, skipping the message handler (`POST /api/bypass/start`). - Jobs: retry, archive, search, retention sweep, and custom-check-failure jobs exercise ServiceControl's recoverability, FTS search, and retention pipelines against the load. Jobs do not start automatically. Start them from the UI or `/api/jobs`. - Release-test presets: named presets mapped from the [testing scenarios](testing-scenarios.md), for example `POST /api/release-tests/ingestion-load/start`. diff --git a/docs/message-body-storage.md b/docs/message-body-storage.md new file mode 100644 index 0000000000..404bc4fb9e --- /dev/null +++ b/docs/message-body-storage.md @@ -0,0 +1,105 @@ +# Message body storage + +The SQL Server and PostgreSQL persisters for ServiceControl.Audit can keep every message body. A small valid text body stays in the database row. With external storage configured, any other body goes there: a directory, an Azure Blob container, or an S3 bucket. Without it, the instance discards bodies over the size limit, as RavenDB does, and bodies that cannot be stored as text. The Audit providers live in `src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage`. The Error (primary) persisters have their own, in `src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage`, and the two do not share code. + +## Placement rules + +`MaxBodySizeToStore` (default 100 KB) is the inline threshold. With external storage configured, it does not limit the size of a stored body. The table describes that case; the next section describes the case without it. + +| Body | Where it goes | `BodyState` | Inline `BodyText` | +| --- | --- | --- | --- | +| Empty | Nowhere | `None` | none | +| Valid UTF-8 text, at most `MaxBodySizeToStore` bytes | Database row | `Stored` | whole body | +| Valid UTF-8 text, over `MaxBodySizeToStore` bytes | External storage | `External` | first `MaxBodySizeToStore` bytes, as a search prefix | +| Binary content type, `Content-Encoding` header, invalid UTF-8, or a NUL character | External storage | `External` | none | + +External storage always holds the complete body. Only the inline prefix is searchable, as before. + +Rows written before external storage existed keep their meaning: `TooLarge` answers 204 and `NotText` answers 404. The bodies behind those rows were discarded at ingestion and cannot be recovered. `External` is a new value of the existing integer column, so there is no schema change. + +## Without external storage + +When `MessageBody/StorageType` is not set, nothing is written outside the database and startup needs no provider. Valid UTF-8 text within `MaxBodySizeToStore` stays inline as above. Text over the limit is dropped with the same inline search prefix (`TooLarge`, answers 204). Other bodies are dropped (`NotText`, answers 404; a binary body over the limit is `TooLarge`). + +Removing the storage settings from an instance that used them leaves its `External` rows in place. Their bodies answer 204, and retention no longer deletes them from the old location. Clean that location manually. + +## Ordering and failures + +Ingestion writes external bodies before the batch transaction that inserts the rows. A failed write fails the batch, and no row is inserted. A failed insert can leave an unreferenced body. + +Audit stores a body through `IAuditBodyStoragePersistence` under the UTC ingestion hour of its batch and the unique message id: `yyyyMMddHH/{unique message id}`. The file system store writes `{StoragePath}/yyyyMMddHH/{unique message id}.body`, the Azure Blob store uses the blob name `yyyyMMddHH/{unique message id}`, and the S3 store prepends its key prefix. Callers pass the hour explicitly and the stores do not parse it from the id, so reads use the `CreatedOn` of the row. Storage keeps the first write for an hour and id and ignores later writes. + +The body URL keeps the audit body id, `yyyyMMddHH-{unique message id}`, which routes the request to the row. The Error (primary) persisters use `IBodyStoragePersistence` with the body id alone; their file system layout remains `{StoragePath}/{body id}.body`. + +Reading a body streams it from storage with the stored content type, size, and a version derived from the row. A body missing from storage answers 204. + +## Compression + +The Audit and Error external storage providers use Brotli compression when the original body size is at least `MessageBody/MinCompressionSize` bytes (default 4096). Smaller bodies are stored uncompressed. This setting applies only to external bodies, not inline `BodyText` or its searchable prefix. Compression does not depend on whether the result is smaller than the original body. + +The file system provider uses `CompressionLevel.Fastest`; Azure Blob and S3 use Brotli quality 1 with a 22-bit window. If the cloud encoder reports that compression failed, those providers store the original bytes instead. A file system compression error fails the write. + +Each provider records the compression flag and original body size in the file header or object metadata. Reads decompress the body as a stream and return its original bytes, content type, and size. API clients do not need to decompress the storage format. + +## Encryption + +The external storage providers do not encrypt bodies before writing them. They set no server-side encryption options on Azure Blob or S3 uploads and expose no encryption-key settings. Compression does not protect body contents, and storage authentication does not configure encryption. + +Configure encryption at rest and key management in the underlying storage account, S3 bucket, or file system volume. Configure database encryption separately for inline bodies and searchable prefixes, and transport encryption for remote storage and database connections. Whether data is encrypted depends on those systems' settings, not the body storage provider. + +## Retention + +Retention removes bodies before it removes rows. For each expired unit it calls `IAuditPartitionManager.DeleteBodies`, and it drops the rows only after that call succeeds. If storage fails, the rows and partitions stay and the next sweep retries. + +The unit differs by provider, so the delete call is part of `IAuditPartitionManager`: + +- SQL Server deletes rows one hour at a time and calls `DeleteBodiesForHour`. +- PostgreSQL drops whole day partitions without scanning rows and calls `DeleteBodiesForDay`, which covers every hour of that day. A day is only dropped once all of it is older than the retention period, so no body is removed earlier than its row would be. + +The file system store deletes the hour directory recursively, so it also removes leftover temporary files in that directory. The day call deletes the 24 hour directories of that day without listing the storage root. The Azure Blob store lists the blobs of the hour or day and deletes them in batches of at most 256. A batch is not atomic, so the store checks the status of every delete, treats a missing blob as deleted, and fails the call on any other error. The S3 store deletes objects in batches of up to 1,000 and checks for per-object failures. + +SQL Server only visits hours that have rows, so an unreferenced body in an hour with no rows stays until that location is cleaned manually. PostgreSQL visits every expired day partition, including pre-created partitions without rows, so it also cleans unreferenced bodies of those days. + +## Configuration + +External storage is optional for Audit instances using SQL Server or PostgreSQL. An Audit instance without `MessageBody/StorageType` starts without it. External storage remains required for Error (primary) instances using these persisters; they fail to start without `ServiceControl/MessageBody/StorageType`. See [Error body placement](error-ingestion-design.md#body-placement). + +An invalid value, or missing settings for the chosen type, still fails at startup. Setup creates the directory, container, or bucket. + +Audit settings use the `ServiceControl.Audit/` prefix in the application settings file. Provider-specific validation matches the Error instance, whose settings use the `ServiceControl/` prefix. + +| Key | Applies to | Default | +| --- | --- | --- | +| `MessageBody/StorageType` | all, optional: `FileSystem`, `AzureBlob`, or `S3` | none | +| `MessageBody/MinCompressionSize` | all external stores, minimum original body size in bytes to compress | 4096 | +| `MessageBody/FileSystem/StoragePath` | `FileSystem`, required | none | +| `MessageBody/FileSystem/DataSpaceRemainingThreshold` | `FileSystem`, free space percentage below which the custom check fails | 15 | +| `MessageBody/Azure/ConnectionString` | `AzureBlob`, shared key or SAS | none | +| `MessageBody/Azure/ServiceUri` | `AzureBlob`, managed identity | none | +| `MessageBody/Azure/ManagedIdentityClientId` | `AzureBlob`, managed identity | none | +| `MessageBody/Azure/AuthorityHost` | `AzureBlob`, managed identity | SDK default | +| `MessageBody/Azure/ContainerName` | `AzureBlob` | `audit-bodies` | +| `MessageBody/S3/BucketName` | `S3`, required | none | +| `MessageBody/S3/KeyPrefix` | `S3` | `audit-bodies/` | +| `MessageBody/S3/Region` | `S3` | SDK default | +| `MessageBody/S3/ServiceUrl` | `S3` | SDK default | +| `MessageBody/S3/AccessKeyId`, `MessageBody/S3/SecretAccessKey` | `S3`, set both or neither | default AWS credential chain | + +Azure requires exactly one of the connection string and the service URI. `MaxBodySizeToStore` keeps its existing top-level setting. + +```xml + + +``` + +```xml + + + +``` + +## One location per instance + +Give each Audit instance its own directory, container, or key prefix, and never share one with an Error instance. Retention deletes whole hour or day groups, so a shared location lets one instance delete bodies that another still references. The Audit defaults differ from the Error defaults (`error-bodies`, `error-bodies/`) for this reason, but they are the same for every Audit instance, so a second Audit instance in the same account must set its own container name or key prefix. + +With `FileSystem` storage, an instance registers the `ServiceControl body storage` custom check that reports low free space on the drive. diff --git a/src/Directory.Packages.props b/src/Directory.Packages.props index 5594f5a888..1ea18344b5 100644 --- a/src/Directory.Packages.props +++ b/src/Directory.Packages.props @@ -11,6 +11,7 @@ + diff --git a/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs b/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs index 09977fdebe..7dba9a1c84 100644 --- a/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs +++ b/src/ServiceControl.AcceptanceTests/ClockRegistrationTests.cs @@ -3,10 +3,11 @@ namespace ServiceControl.AcceptanceTests using System; using System.Runtime.Loader; using System.Threading.Tasks; + using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.DependencyInjection; - using Microsoft.Extensions.Hosting; using NServiceBus; using NUnit.Framework; + using global::ServiceControl.Infrastructure.WebApi; using Particular.ServiceControl; using ServiceBus.Management.Infrastructure.Settings; @@ -41,9 +42,10 @@ public void Should_keep_the_clock_the_host_registered() var endpointConfiguration = new EndpointConfiguration(settings.InstanceName); endpointConfiguration.AssemblyScanner().Disable = true; - var hostBuilder = Host.CreateApplicationBuilder(); + var hostBuilder = WebApplication.CreateBuilder(); hostBuilder.Services.AddSingleton(hostClock); hostBuilder.AddServiceControl(settings, endpointConfiguration); + hostBuilder.AddServiceControlApi(settings.CorsSettings); using var host = hostBuilder.Build(); diff --git a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs index c6aeb6f385..82c9ed933b 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/AcceptanceTestStorageConfiguration.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Audit.AcceptanceTests.PostgreSql; using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.IO; using System.Threading; using System.Threading.Tasks; using ServiceControl.Audit.AcceptanceTests.TestSupport; @@ -22,18 +23,39 @@ public async Task> CustomizeSettings(CancellationTok schemas.Add(schema); + var bodyStoragePath = Directory.CreateTempSubdirectory("sc_audit_at_bodies_").FullName; + bodyStoragePaths.Add(bodyStoragePath); + return new Dictionary { [EFPersistenceConfigurationBase.ConnectionStringKey] = connectionString, - [EFPersistenceConfigurationBase.SchemaKey] = schema + [EFPersistenceConfigurationBase.SchemaKey] = schema, + ["MessageBody/StorageType"] = "FileSystem", + ["MessageBody/FileSystem/StoragePath"] = bodyStoragePath }; } public async Task Cleanup(CancellationToken cancellationToken = default) { - while (schemas.TryTake(out var schema)) + try + { + while (schemas.TryTake(out var schema)) + { + await TestSchema.Drop(connectionString, schema, cancellationToken); + } + } + finally { - await TestSchema.Drop(connectionString, schema, cancellationToken); + while (bodyStoragePaths.TryTake(out var bodyStoragePath)) + { + try + { + Directory.Delete(bodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + } } } @@ -51,5 +73,6 @@ sealed class Release : IDisposable static readonly SemaphoreSlim initializationLock = new(1, 1); readonly ConcurrentBag schemas = []; + readonly ConcurrentBag bodyStoragePaths = []; string connectionString; } diff --git a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj index 4836ca191d..8682623de6 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj +++ b/src/ServiceControl.Audit.AcceptanceTests.PostgreSql/ServiceControl.Audit.AcceptanceTests.PostgreSql.csproj @@ -30,12 +30,16 @@ + + + + diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs index 52485fc4a1..54c4b0a87d 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/AcceptanceTestStorageConfiguration.cs @@ -3,6 +3,7 @@ namespace ServiceControl.Audit.AcceptanceTests.SqlServer; using System; using System.Collections.Concurrent; using System.Collections.Generic; +using System.IO; using System.Threading; using System.Threading.Tasks; using ServiceControl.Audit.AcceptanceTests.TestSupport; @@ -22,18 +23,39 @@ public async Task> CustomizeSettings(CancellationTok schemas.Add(schema); + var bodyStoragePath = Directory.CreateTempSubdirectory("sc_audit_at_bodies_").FullName; + bodyStoragePaths.Add(bodyStoragePath); + return new Dictionary { [EFPersistenceConfigurationBase.ConnectionStringKey] = connectionString, - [EFPersistenceConfigurationBase.SchemaKey] = schema + [EFPersistenceConfigurationBase.SchemaKey] = schema, + ["MessageBody/StorageType"] = "FileSystem", + ["MessageBody/FileSystem/StoragePath"] = bodyStoragePath }; } public async Task Cleanup(CancellationToken cancellationToken = default) { - while (schemas.TryTake(out var schema)) + try + { + while (schemas.TryTake(out var schema)) + { + await TestSchema.Drop(connectionString, schema, cancellationToken); + } + } + finally { - await TestSchema.Drop(connectionString, schema, cancellationToken); + while (bodyStoragePaths.TryTake(out var bodyStoragePath)) + { + try + { + Directory.Delete(bodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + } } } @@ -51,5 +73,6 @@ sealed class Release : IDisposable static readonly SemaphoreSlim initializationLock = new(1, 1); readonly ConcurrentBag schemas = []; + readonly ConcurrentBag bodyStoragePaths = []; string connectionString; } diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj index 0294b97928..c596961a39 100644 --- a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/ServiceControl.Audit.AcceptanceTests.SqlServer.csproj @@ -36,6 +36,9 @@ + + + diff --git a/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs new file mode 100644 index 0000000000..3d4f3f91e8 --- /dev/null +++ b/src/ServiceControl.Audit.AcceptanceTests.SqlServer/When_messages_with_big_bodies_are_ingested.ExternalBody.cs @@ -0,0 +1,49 @@ +namespace ServiceControl.Audit.AcceptanceTests.Auditing +{ + using System.Text.Json; + using System.Threading.Tasks; + using AcceptanceTesting; + using AcceptanceTesting.EndpointTemplates; + using Audit.Auditing.MessagesView; + using NServiceBus; + using NServiceBus.AcceptanceTesting; + using NUnit.Framework; + + partial class When_messages_with_big_bodies_are_ingested + { + [Test] + public async Task Should_serve_the_full_body_when_it_is_larger_than_MaxBodySizeToStore() + { + SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; + + var expected = new byte[MAX_BODY_SIZE + 1000]; + byte[] body = null; + + await Define() + .WithEndpoint(c => c.When(b => b.SendLocal(new BigFatMessage { BigFatBody = expected }))) + .Done( + async c => + { + if (c.MessageId == null) + { + return false; + } + + var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); + MessagesView auditMessage = result; + if (!result) + { + return false; + } + + body = await this.DownloadData($"/api{auditMessage.BodyUrl}"); + + return true; + }) + .Run(); + + using var document = JsonDocument.Parse(body); + Assert.That(document.RootElement.GetProperty(nameof(BigFatMessage.BigFatBody)).GetBytesFromBase64(), Is.EqualTo(expected)); + } + } +} diff --git a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs new file mode 100644 index 0000000000..a01155630b --- /dev/null +++ b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.MaxBodySize.cs @@ -0,0 +1,55 @@ +namespace ServiceControl.Audit.AcceptanceTests.Auditing +{ + using System.Net; + using System.Threading.Tasks; + using AcceptanceTesting; + using AcceptanceTesting.EndpointTemplates; + using Audit.Auditing.MessagesView; + using NServiceBus; + using NServiceBus.AcceptanceTesting; + using NUnit.Framework; + + partial class When_messages_with_big_bodies_are_ingested + { + [Test] + public async Task Should_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_less_then_message_size() + { + //Arrange + SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; + + byte[] body = null; + + //Act + await Define() + .WithEndpoint(c => c.When(b => b.SendLocal( + new BigFatMessage // An endpoint that is configured for audit + { + BigFatBody = new byte[MAX_BODY_SIZE + 1000] + })) + ) + .Done( + async c => + { + if (c.MessageId == null) + { + return false; + } + + var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); + MessagesView auditMessage = result; + if (!result) + { + return false; + } + + body = await this.DownloadData($"/api{auditMessage.BodyUrl}", HttpStatusCode.NoContent); + + return true; + }) + .Run(); + + //Assert + Assert.That(body, Is.Empty); + } + } +} diff --git a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs index 990b705566..5c4d7fa55b 100644 --- a/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs +++ b/src/ServiceControl.Audit.AcceptanceTests/Auditing/When_messages_with_big_bodies_are_ingested.cs @@ -9,7 +9,7 @@ using NServiceBus.AcceptanceTesting; using NUnit.Framework; - class When_messages_with_big_bodies_are_ingested : AcceptanceTest + partial class When_messages_with_big_bodies_are_ingested : AcceptanceTest { [Test] public async Task Should_not_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_greater_then_message_size() @@ -52,47 +52,6 @@ await Define() Assert.That(body, Is.Not.Null); } - [Test] - public async Task Should_get_an_empty_audit_message_body_when_configured_MaxBodySizeToStore_is_less_then_message_size() - { - //Arrange - SetSettings = settings => settings.MaxBodySizeToStore = MAX_BODY_SIZE; - - byte[] body = null; - - //Act - await Define() - .WithEndpoint(c => c.When(b => b.SendLocal( - new BigFatMessage // An endpoint that is configured for audit - { - BigFatBody = new byte[MAX_BODY_SIZE + 1000] - })) - ) - .Done( - async c => - { - if (c.MessageId == null) - { - return false; - } - - var result = await this.TryGetSingle("/api/messages", r => r.MessageId == c.MessageId); - MessagesView auditMessage = result; - if (!result) - { - return false; - } - - body = await this.DownloadData($"/api{auditMessage.BodyUrl}", HttpStatusCode.NoContent); - - return true; - }) - .Run(); - - //Assert - Assert.That(body, Is.Empty); - } - [Test] public async Task Should_not_get_an_empty_audit_message_body_when_body_is_above_loh_but_below_max_body_size() { diff --git a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs index 1c310dd52f..da74858666 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/PostgreSqlAuditPartitionManager.cs @@ -52,6 +52,10 @@ public async Task> ListExpired(AuditDbContext dbContext, return [.. days]; } + // A partition holds a whole day, so the bodies are deleted by day rather than by hour. + public Task DeleteBodies(IAuditBodyStoragePersistence bodyStorage, DateTime day, CancellationToken cancellationToken = default) => + bodyStorage.DeleteBodiesForDay(day, cancellationToken); + public async Task DropExpired(AuditDbContext dbContext, DateTime day, int batchSize, CancellationToken cancellationToken = default) { var sql = string.Concat(Tables(dbContext).Select(table => $"DROP TABLE IF EXISTS {table.Partition(day)};\n")); diff --git a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest index 2da848c499..93363f97c6 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest +++ b/src/ServiceControl.Audit.Persistence.EFCore.PostgreSql/persistence.manifest @@ -17,6 +17,66 @@ { "Name": "ServiceControl.Audit/Database/CommandTimeout", "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/StorageType", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/StoragePath", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/DataSpaceRemainingThreshold", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ConnectionString", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ServiceUri", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ManagedIdentityClientId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/AuthorityHost", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ContainerName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/BucketName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/KeyPrefix", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/Region", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/ServiceUrl", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/AccessKeyId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/SecretAccessKey", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/MinCompressionSize", + "Mandatory": false } ] } diff --git a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs index 65e436b7f1..2c4d19865e 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/SqlServerAuditPartitionManager.cs @@ -18,6 +18,9 @@ public async Task> ListExpired(AuditDbContext dbContext, return await messageHours.Union(snapshotHours).OrderBy(hour => hour).ToListAsync(cancellationToken); } + public Task DeleteBodies(IAuditBodyStoragePersistence bodyStorage, DateTime hour, CancellationToken cancellationToken = default) => + bodyStorage.DeleteBodiesForHour(hour, cancellationToken); + public async Task DropExpired(AuditDbContext dbContext, DateTime hour, int batchSize, CancellationToken cancellationToken = default) { var messagesDeleted = await dbContext.AuditMessages diff --git a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest index 6a36218884..475fec6814 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest +++ b/src/ServiceControl.Audit.Persistence.EFCore.SqlServer/persistence.manifest @@ -17,6 +17,66 @@ { "Name": "ServiceControl.Audit/Database/CommandTimeout", "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/StorageType", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/StoragePath", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/FileSystem/DataSpaceRemainingThreshold", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ConnectionString", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ServiceUri", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ManagedIdentityClientId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/AuthorityHost", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/Azure/ContainerName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/BucketName", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/KeyPrefix", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/Region", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/ServiceUrl", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/AccessKeyId", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/S3/SecretAccessKey", + "Mandatory": false + }, + { + "Name": "ServiceControl.Audit/MessageBody/MinCompressionSize", + "Mandatory": false } ] } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs new file mode 100644 index 0000000000..edb77bc5a0 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/AzureBlobBodyStorageSettings.cs @@ -0,0 +1,28 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +public sealed class AzureBlobBodyStorageSettings : BodyStorageSettings +{ + public const string DefaultContainerName = "audit-bodies"; + + public required AzureBlobAuthentication Authentication { get; set; } + public string ContainerName { get; set; } = DefaultContainerName; +} + +// Shared-key and managed-identity auth are mutually exclusive, and the managed identity options are +// meaningless alongside a connection string. +public abstract class AzureBlobAuthentication; + +public sealed class AzureBlobSharedKeyAuthentication : AzureBlobAuthentication +{ + public required string ConnectionString { get; set; } +} + +public sealed class AzureBlobManagedIdentityAuthentication : AzureBlobAuthentication +{ + public required Uri ServiceUri { get; set; } + public string? ClientId { get; set; } + + // Steers the login endpoint for sovereign clouds; when unset the SDK honours the + // AZURE_AUTHORITY_HOST environment variable. + public Uri? AuthorityHost { get; set; } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettings.cs new file mode 100644 index 0000000000..1791093dde --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettings.cs @@ -0,0 +1,17 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +/// +/// Settings for the selected body storage type. +/// +/// +/// One subclass per storage type, so a store only ever receives the settings it can act on and the +/// configuration layer's validation is carried by the type rather than re-asserted at the point of use. +/// +public abstract class BodyStorageSettings +{ + public const int DefaultMinCompressionSize = 4096; + public const int DefaultMaxBodySizeToStore = 102400; // 100 kb + + public int MinCompressionSize { get; set; } = DefaultMinCompressionSize; + public int MaxBodySizeToStore { get; set; } = DefaultMaxBodySizeToStore; +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs new file mode 100644 index 0000000000..a71c2b7878 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageSettingsParser.cs @@ -0,0 +1,175 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +using Microsoft.Extensions.Logging; +using ServiceControl.Infrastructure; + +// Mirrors the Error persister's MessageBody/* parsing, which reads a settings root rather than a dictionary. +static class BodyStorageSettingsParser +{ + internal const string StorageTypeKey = "MessageBody/StorageType"; + internal const string FileSystemStoragePathKey = "MessageBody/FileSystem/StoragePath"; + internal const string FileSystemDataSpaceRemainingThresholdKey = "MessageBody/FileSystem/DataSpaceRemainingThreshold"; + internal const string AzureConnectionStringKey = "MessageBody/Azure/ConnectionString"; + internal const string AzureServiceUriKey = "MessageBody/Azure/ServiceUri"; + internal const string AzureManagedIdentityClientIdKey = "MessageBody/Azure/ManagedIdentityClientId"; + internal const string AzureAuthorityHostKey = "MessageBody/Azure/AuthorityHost"; + internal const string AzureContainerNameKey = "MessageBody/Azure/ContainerName"; + internal const string S3BucketNameKey = "MessageBody/S3/BucketName"; + internal const string S3KeyPrefixKey = "MessageBody/S3/KeyPrefix"; + internal const string S3RegionKey = "MessageBody/S3/Region"; + internal const string S3ServiceUrlKey = "MessageBody/S3/ServiceUrl"; + internal const string S3AccessKeyIdKey = "MessageBody/S3/AccessKeyId"; + internal const string S3SecretAccessKeyKey = "MessageBody/S3/SecretAccessKey"; + internal const string MinCompressionSizeKey = "MessageBody/MinCompressionSize"; + + public static readonly string[] Keys = + [ + StorageTypeKey, + FileSystemStoragePathKey, + FileSystemDataSpaceRemainingThresholdKey, + AzureConnectionStringKey, + AzureServiceUriKey, + AzureManagedIdentityClientIdKey, + AzureAuthorityHostKey, + AzureContainerNameKey, + S3BucketNameKey, + S3KeyPrefixKey, + S3RegionKey, + S3ServiceUrlKey, + S3AccessKeyIdKey, + S3SecretAccessKeyKey, + MinCompressionSizeKey + ]; + + // No storage type means no external storage. A configured but invalid value still fails. + public static BodyStorageSettings? Parse(IDictionary specific, int maxBodySizeToStore) + { + if (ReadStorageType(specific) is not { } storageType) + { + return null; + } + + BodyStorageSettings bodyStorage = storageType switch + { + BodyStorageType.FileSystem => new FileSystemBodyStorageSettings + { + StoragePath = GetRequired(specific, FileSystemStoragePathKey), + DataSpaceRemainingThreshold = ReadDataSpaceRemainingThreshold(specific) + }, + BodyStorageType.AzureBlob => new AzureBlobBodyStorageSettings + { + Authentication = ReadAzureBlobAuthentication(specific), + ContainerName = Get(specific, AzureContainerNameKey) ?? AzureBlobBodyStorageSettings.DefaultContainerName + }, + BodyStorageType.S3 => new S3BodyStorageSettings + { + BucketName = GetRequired(specific, S3BucketNameKey), + KeyPrefix = Get(specific, S3KeyPrefixKey) ?? S3BodyStorageSettings.DefaultKeyPrefix, + Region = Get(specific, S3RegionKey), + ServiceUrl = Get(specific, S3ServiceUrlKey), + Credentials = ReadS3Credentials(specific) + }, + var unknown => throw new ArgumentOutOfRangeException(nameof(specific), unknown, "Unknown body storage type.") + }; + + bodyStorage.MinCompressionSize = ReadInt(specific, MinCompressionSizeKey, BodyStorageSettings.DefaultMinCompressionSize); + bodyStorage.MaxBodySizeToStore = ReadMaxBodySizeToStore(maxBodySizeToStore); + + return bodyStorage; + } + + static BodyStorageType? ReadStorageType(IDictionary specific) + { + var raw = Get(specific, StorageTypeKey); + + if (raw is null) + { + return null; + } + + if (!Enum.TryParse(raw, ignoreCase: true, out var value) || !Enum.IsDefined(value)) + { + throw new InvalidOperationException($"{StorageTypeKey} value '{raw}' is not valid. Valid values: {string.Join(", ", Enum.GetNames())}."); + } + + return value; + } + + static AzureBlobAuthentication ReadAzureBlobAuthentication(IDictionary specific) + { + var connectionString = Get(specific, AzureConnectionStringKey); + var serviceUri = Get(specific, AzureServiceUriKey); + + // A connection string carries shared-key/SAS auth; a service URI uses managed identity. + if ((connectionString is null) == (serviceUri is null)) + { + throw new InvalidOperationException($"Azure Blob body storage requires exactly one of {AzureConnectionStringKey} (shared key / SAS) or {AzureServiceUriKey} (managed identity). {(connectionString is null ? "Neither was set." : "Both were set.")}"); + } + + if (connectionString is not null) + { + return new AzureBlobSharedKeyAuthentication { ConnectionString = connectionString }; + } + + var authorityHost = Get(specific, AzureAuthorityHostKey); + + return new AzureBlobManagedIdentityAuthentication + { + ServiceUri = ReadAbsoluteUri(serviceUri!, AzureServiceUriKey), + ClientId = Get(specific, AzureManagedIdentityClientIdKey), + AuthorityHost = authorityHost is null ? null : ReadAbsoluteUri(authorityHost, AzureAuthorityHostKey) + }; + } + + static S3StaticCredentials? ReadS3Credentials(IDictionary specific) + { + var accessKeyId = Get(specific, S3AccessKeyIdKey); + var secretAccessKey = Get(specific, S3SecretAccessKeyKey); + + if ((accessKeyId is null) != (secretAccessKey is null)) + { + throw new InvalidOperationException($"S3 body storage requires both {S3AccessKeyIdKey} and {S3SecretAccessKeyKey} together, or neither to use the default AWS credential chain (IAM role)."); + } + + return accessKeyId is null ? null : new S3StaticCredentials { AccessKeyId = accessKeyId, SecretAccessKey = secretAccessKey! }; + } + + static Uri ReadAbsoluteUri(string raw, string key) => + Uri.TryCreate(raw, UriKind.Absolute, out var uri) + ? uri + : throw new InvalidOperationException($"{key} value '{raw}' is not a valid absolute URI."); + + static int ReadDataSpaceRemainingThreshold(IDictionary specific) + { + var threshold = ReadInt(specific, FileSystemDataSpaceRemainingThresholdKey, FileSystemBodyStorageSettings.DefaultDataSpaceRemainingThreshold); + + return threshold is < 0 or > 100 + ? throw new InvalidOperationException($"{FileSystemDataSpaceRemainingThresholdKey} value '{threshold}' is not valid. The value is a percentage between 0 and 100.") + : threshold; + } + + internal static int ReadMaxBodySizeToStore(int maxBodySizeToStore) + { + if (maxBodySizeToStore > 0) + { + return maxBodySizeToStore; + } + + logger.LogError("MaxBodySizeToStore setting is invalid, 1 is the minimum value. Defaulting to {MaxBodySizeToStoreDefault}", BodyStorageSettings.DefaultMaxBodySizeToStore); + + return BodyStorageSettings.DefaultMaxBodySizeToStore; + } + + static int ReadInt(IDictionary specific, string key, int defaultValue) => + Get(specific, key) is not { } value ? defaultValue + : int.TryParse(value, out var parsed) ? parsed + : throw new InvalidOperationException($"{key} must be an integer."); + + static string GetRequired(IDictionary specific, string key) => + Get(specific, key) ?? throw new InvalidOperationException($"{key} must be specified."); + + static string? Get(IDictionary specific, string key) => + specific.TryGetValue(key, out var value) && !string.IsNullOrWhiteSpace(value) ? value : null; + + static readonly ILogger logger = LoggerUtil.CreateStaticLogger(); +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageType.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageType.cs new file mode 100644 index 0000000000..5b7b97ad09 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/BodyStorageType.cs @@ -0,0 +1,8 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +public enum BodyStorageType +{ + FileSystem, + AzureBlob, + S3 +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs index 90cb94678b..0799edd11d 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceBase.cs @@ -4,6 +4,8 @@ namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; using Microsoft.Extensions.DependencyInjection.Extensions; using NServiceBus; using ServiceControl.Audit.Persistence.EFCore.Implementation; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; +using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.UnitOfWork; @@ -13,6 +15,7 @@ public void AddPersistence(IServiceCollection services) { AddCommon(services); AddQueryServices(services); + AddBodyStorage(services); services.AddSingleton(); services.AddSingleton(provider => provider.GetRequiredService()); @@ -32,6 +35,11 @@ public void AddPersistence(IServiceCollection services) { endpointConfiguration.AddCustomCheck(); AddCustomChecks(endpointConfiguration); + + if (settings.BodyStorage is FileSystemBodyStorageSettings) + { + endpointConfiguration.AddCustomCheck(); + } } } @@ -49,6 +57,31 @@ void AddCommon(IServiceCollection services) AddPartitionManager(services); } + // The settings type picks the store, so each store resolves only the settings it can act on. + void AddBodyStorage(IServiceCollection services) + { + switch (settings.BodyStorage) + { + case FileSystemBodyStorageSettings fileSystem: + services.AddSingleton(fileSystem); + services.AddSingleton(); + services.AddSingleton(); + break; + case AzureBlobBodyStorageSettings azureBlob: + services.AddSingleton(azureBlob); + services.AddSingleton(); + break; + case S3BodyStorageSettings s3: + services.AddSingleton(s3); + services.AddSingleton(); + break; + case null: + break; + default: + throw new ArgumentOutOfRangeException(nameof(settings), settings.BodyStorage, "Unknown body storage type."); + } + } + protected abstract void AddDbContext(IServiceCollection services, EFPersisterSettings settings); protected abstract void AddPartitionManager(IServiceCollection services); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs index 3ab25809af..38d1d6e8c6 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersistenceConfigurationBase.cs @@ -15,7 +15,7 @@ public abstract class EFPersistenceConfigurationBase : IPersistenceConfiguration public abstract string Name { get; } - public IEnumerable ConfigurationKeys => [ConnectionStringKey, SchemaKey, CommandTimeoutKey, QueryTimeoutInSecondsKey]; + public IEnumerable ConfigurationKeys => [ConnectionStringKey, SchemaKey, CommandTimeoutKey, QueryTimeoutInSecondsKey, .. BodyStorageSettingsParser.Keys]; public IPersistence Create(PersistenceSettings settings) => Create(CreateSettings(settings)); @@ -37,7 +37,8 @@ public static EFPersisterSettings CreateSettings(PersistenceSettings settings) CommandTimeout = ReadInt(specific, CommandTimeoutKey, EFPersisterSettings.DefaultCommandTimeout), QueryTimeout = QueryTimeLimit.Validate(ReadInt(specific, QueryTimeoutInSecondsKey, QueryTimeLimit.DefaultSeconds), QueryTimeoutSettingName, logger), AuditRetentionPeriod = settings.AuditRetentionPeriod, - MaxBodySizeToStore = settings.MaxBodySizeToStore + BodyStorage = BodyStorageSettingsParser.Parse(specific, settings.MaxBodySizeToStore), + MaxBodySizeToStore = BodyStorageSettingsParser.ReadMaxBodySizeToStore(settings.MaxBodySizeToStore) }; } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs index cee4ba0869..b09bf24a18 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/EFPersisterSettings.cs @@ -20,7 +20,8 @@ public string? Schema public int CommandTimeout { get; init; } = DefaultCommandTimeout; public TimeSpan QueryTimeout { get; init; } = QueryTimeLimit.Default; public required TimeSpan AuditRetentionPeriod { get; init; } - public required int MaxBodySizeToStore { get; init; } + public BodyStorageSettings? BodyStorage { get; init; } + public int MaxBodySizeToStore { get; init; } = BodyStorageSettings.DefaultMaxBodySizeToStore; public int MaxRetryCount { get; init; } = 5; public int MaxRetryDelayInSeconds { get; init; } = 30; public bool EnableRetryOnFailure { get; init; } = true; diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs new file mode 100644 index 0000000000..3f1d6aeb81 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/FileSystemBodyStorageSettings.cs @@ -0,0 +1,10 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +public sealed class FileSystemBodyStorageSettings : BodyStorageSettings +{ + public const int DefaultDataSpaceRemainingThreshold = 15; + + public required string StoragePath { get; set; } + + public int DataSpaceRemainingThreshold { get; set; } = DefaultDataSpaceRemainingThreshold; +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs new file mode 100644 index 0000000000..8864f5516b --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Abstractions/S3BodyStorageSettings.cs @@ -0,0 +1,20 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Abstractions; + +public sealed class S3BodyStorageSettings : BodyStorageSettings +{ + public const string DefaultKeyPrefix = "audit-bodies/"; + + public required string BucketName { get; set; } + public string KeyPrefix { get; set; } = DefaultKeyPrefix; + public string? Region { get; set; } + public string? ServiceUrl { get; set; } + + // Null resolves the ambient IAM role through the SDK's default credential chain. + public S3StaticCredentials? Credentials { get; set; } +} + +public sealed class S3StaticCredentials +{ + public required string AccessKeyId { get; set; } + public required string SecretAccessKey { get; set; } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs b/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs index 59d773602b..d7d8055d86 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Entities/BodyState.cs @@ -5,5 +5,6 @@ public enum BodyState None = 0, Stored = 1, TooLarge = 2, - NotText = 3 + NotText = 3, + External = 4 } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs index 89daf8cd42..68ec395375 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditDataStore.cs @@ -2,6 +2,7 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; using ServiceControl.Audit.Auditing; using ServiceControl.Audit.Auditing.MessagesView; using ServiceControl.Audit.Infrastructure; @@ -15,7 +16,9 @@ sealed class AuditDataStore( IServiceScopeFactory scopeFactory, EFPersisterSettings settings, IFullTextSearchDialect fullTextSearch, - TimeProvider timeProvider) : DataStoreBase(scopeFactory, settings), IAuditMessagesViewDataStore, ISagaHistoryDataStore + TimeProvider timeProvider, + ILogger logger, + IAuditBodyStoragePersistence? bodyStorage = null) : DataStoreBase(scopeFactory, settings), IAuditMessagesViewDataStore, ISagaHistoryDataStore { public const int TotalCountCap = 100_000; @@ -43,8 +46,9 @@ public Task>> QueryMessagesByReceivingEndpoint(b public Task>> QueryMessagesByConversationId(string conversationId, PagingInfo pagingInfo, SortInfo sortInfo, CancellationToken cancellationToken = default) => Page(messages => messages.Where(message => message.ConversationId == conversationId), pagingInfo, sortInfo, cancellationToken); - public Task GetMessageBody(string messageId, CancellationToken cancellationToken = default) => - ExecuteQueryWithDbContext(async (dbContext, token) => + public async Task GetMessageBody(string messageId, CancellationToken cancellationToken = default) + { + var body = await ExecuteQueryWithDbContext(async (dbContext, token) => { IQueryable candidates; if (AuditBodyId.TryParse(messageId, out var createdOn, out var uniqueMessageId)) @@ -57,29 +61,53 @@ public Task GetMessageBody(string messageId, CancellationToken } else { - return MessageBodyView.NotFound(); + return null; } - var body = await candidates + return await candidates .AsNoTracking() .OrderByDescending(message => message.CreatedOn) .ThenByDescending(message => message.Id) - .Select(message => new { message.CreatedOn, message.Id, message.BodyState, message.BodyText, message.BodyContentType, message.BodySize }) + .Select(message => new BodyRow(message.CreatedOn, message.Id, message.UniqueMessageId, message.BodyState, message.BodyText, message.BodyContentType, message.BodySize)) .FirstOrDefaultAsync(token); + }, cancellationToken); - if (body is null) - { - return MessageBodyView.NotFound(); - } + if (body is null) + { + return MessageBodyView.NotFound(); + } - return body.BodyState switch - { - BodyState.Stored => MessageBodyView.FromString(body.BodyText!, body.BodyContentType ?? "text/plain", body.BodySize, DataVersion.Compose(("row", $"{body.CreatedOn.Ticks}-{body.Id}"))), - BodyState.None or BodyState.TooLarge => MessageBodyView.NoContent(), - BodyState.NotText => MessageBodyView.NotFound(), - _ => throw new InvalidOperationException($"Unknown body state {body.BodyState}") - }; - }, cancellationToken); + var version = DataVersion.Compose(("row", $"{body.CreatedOn.Ticks}-{body.Id}")); + + switch (body.BodyState) + { + case BodyState.Stored: + return MessageBodyView.FromString(body.BodyText!, body.BodyContentType ?? "text/plain", body.BodySize, version); + case BodyState.None or BodyState.TooLarge: + return MessageBodyView.NoContent(); + case BodyState.NotText: + return MessageBodyView.NotFound(); + case BodyState.External: + // Read outside the query so the stream is not tied to the query timeout. + // The store can be removed after rows were written to it. + if (bodyStorage is null) + { + return MessageBodyView.NoContent(); + } + + var external = await bodyStorage.ReadBody(body.UniqueMessageId.ToString(), body.CreatedOn, cancellationToken); + + if (external is null) + { + logger.LogWarning("Message body {MessageId} is marked External in audit row {RowId} created on {CreatedOn:u}, but external storage returned nothing", messageId, body.Id, body.CreatedOn); + return MessageBodyView.NoContent(); + } + + return MessageBodyView.FromStream(external.Stream, external.ContentType, external.BodySize, version); + default: + throw new InvalidOperationException($"Unknown body state {body.BodyState}"); + } + } public Task>> QueryAuditCounts(string endpointName, CancellationToken cancellationToken = default) => ExecuteQueryWithDbContext(async (dbContext, token) => @@ -183,4 +211,6 @@ IQueryable Search(IQueryable source, str OutgoingMessages = SagaSnapshotJson.ReadOutgoingMessages(snapshot.OutgoingMessagesJson), Endpoint = snapshot.Endpoint }; + + sealed record BodyRow(DateTime CreatedOn, long Id, Guid UniqueMessageId, BodyState BodyState, string? BodyText, string? BodyContentType, int BodySize); } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs index 588631ee32..ac76823776 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/AuditRetention.cs @@ -14,7 +14,8 @@ sealed class AuditRetention( AuditRetentionCustomCheck.State retentionState, EFPersisterSettings settings, TimeProvider timeProvider, - ILogger logger) : BackgroundService + ILogger logger, + IAuditBodyStoragePersistence? bodyStorage = null) : BackgroundService { // Stays under the 5,000 locks at which SQL Server escalates a delete to a table lock. const int BatchSize = 4_000; @@ -114,6 +115,12 @@ async Task Sweep(bool pace, CancellationToken cancellationToken) foreach (var expired in await partitions.ListExpired(dbContext, keepFrom, cancellationToken)) { + // Bodies go first: a failure here leaves the rows in place, so the next sweep retries the same hour. + if (bodyStorage is not null) + { + await partitions.DeleteBodies(bodyStorage, expired, cancellationToken); + } + DropResult drop; do { diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobBodyStorage.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobBodyStorage.cs new file mode 100644 index 0000000000..5e0c398681 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobBodyStorage.cs @@ -0,0 +1,148 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using System.IO.Compression; +using System.Net; +using Azure; +using Azure.Storage.Blobs; +using Azure.Storage.Blobs.Models; +using Azure.Storage.Blobs.Specialized; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +public class AuditAzureBlobBodyStorage : IAuditBodyStoragePersistence +{ + const string FormatVersion = "1"; + const int MaxBatchSize = 256; // Azure Blob Batch limit + + readonly BlobContainerClient container; + readonly int minBodySizeForCompression; + + public AuditAzureBlobBodyStorage(AzureBlobBodyStorageSettings settings) + { + container = AuditAzureBlobClientFactory.CreateContainerClient(settings); + minBodySizeForCompression = settings.MinCompressionSize; + } + + internal AuditAzureBlobBodyStorage(BlobContainerClient container, int minBodySizeForCompression) + { + this.container = container; + this.minBodySizeForCompression = minBodySizeForCompression; + } + + public async Task WriteBody(string bodyId, DateTime createdOn, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) + { + var blob = container.GetBlobClient(AuditBodyStorageKey.For(createdOn, bodyId)); + + var compressed = body.Length >= minBodySizeForCompression ? BodyCompression.TryCompress(body) : null; + var data = compressed is null ? BinaryData.FromBytes(body) : BinaryData.FromBytes(compressed); + + var options = new BlobUploadOptions + { + // Bodies are immutable, so create the blob only if it does not already exist. + Conditions = new BlobRequestConditions { IfNoneMatch = ETag.All }, + Metadata = new Dictionary + { + ["FormatVersion"] = FormatVersion, + ["ContentType"] = Uri.EscapeDataString(contentType), + ["BodySize"] = body.Length.ToString(), + ["IsCompressed"] = (compressed is not null).ToString() + } + }; + + try + { + await blob.UploadAsync(data, options, cancellationToken); + } + catch (RequestFailedException ex) when (ex.ErrorCode == BlobErrorCode.BlobAlreadyExists || ex.ErrorCode == BlobErrorCode.ConditionNotMet) + { + // A blob for this immutable body already exists. + } + } + + public async Task ReadBody(string bodyId, DateTime createdOn, CancellationToken cancellationToken = default) + { + var blob = container.GetBlobClient(AuditBodyStorageKey.For(createdOn, bodyId)); + + try + { + var content = (await blob.DownloadStreamingAsync(cancellationToken: cancellationToken)).Value; + try + { + var metadata = content.Details.Metadata; + + if (metadata.TryGetValue("FormatVersion", out var version) && version != FormatVersion) + { + throw new InvalidOperationException($"Unsupported blob format version {version} for {bodyId}."); + } + + var contentType = metadata.TryGetValue("ContentType", out var ct) ? Uri.UnescapeDataString(ct) : "application/octet-stream"; + var bodySize = metadata.TryGetValue("BodySize", out var sizeText) && int.TryParse(sizeText, out var size) ? size : 0; + var isCompressed = metadata.TryGetValue("IsCompressed", out var compressedText) && bool.TryParse(compressedText, out var compressed) && compressed; + Stream stream = isCompressed + ? new ExpectedLengthStream(new BrotliStream(content.Content, CompressionMode.Decompress), bodySize) + : content.Content; + + return new MessageBodyFileResult + { + Stream = stream, + ContentType = contentType, + BodySize = bodySize + }; + } + catch + { + content.Dispose(); + throw; + } + } + catch (RequestFailedException ex) when (ex.Status == (int)HttpStatusCode.NotFound) + { + return null; + } + } + + public Task DeleteBodiesForHour(DateTime hour, CancellationToken cancellationToken = default) => + DeleteBodiesWithPrefix(AuditBodyStorageKey.HourPrefix(hour), cancellationToken); + + public Task DeleteBodiesForDay(DateTime day, CancellationToken cancellationToken = default) => + DeleteBodiesWithPrefix(AuditBodyStorageKey.DayPrefix(day), cancellationToken); + + async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken) + { + var batchClient = container.GetBlobBatchClient(); + var batch = batchClient.CreateBatch(); + var responses = new List(MaxBatchSize); + + await foreach (var blob in container.GetBlobsAsync(BlobTraits.None, BlobStates.None, prefix, cancellationToken: cancellationToken)) + { + responses.Add(batch.DeleteBlob(container.Name, blob.Name)); + + if (responses.Count == MaxBatchSize) + { + await Submit(); + } + } + + if (responses.Count > 0) + { + await Submit(); + } + + async Task Submit() + { + // A batch is not atomic: a successful outer response does not mean every delete succeeded. + await batchClient.SubmitBatchAsync(batch, throwOnAnyFailure: false, cancellationToken); + + foreach (var response in responses) + { + if (response.Status is >= 400 and not (int)HttpStatusCode.NotFound) + { + throw new RequestFailedException(response); + } + } + + responses.Clear(); + batch = batchClient.CreateBatch(); + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobClientFactory.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobClientFactory.cs new file mode 100644 index 0000000000..1b7bc73538 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditAzureBlobClientFactory.cs @@ -0,0 +1,38 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using Azure.Core; +using Azure.Identity; +using Azure.Storage.Blobs; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; + +static class AuditAzureBlobClientFactory +{ + public static BlobContainerClient CreateContainerClient(AzureBlobBodyStorageSettings settings) + { + var serviceClient = settings.Authentication switch + { + AzureBlobSharedKeyAuthentication sharedKey => new BlobServiceClient(sharedKey.ConnectionString), + AzureBlobManagedIdentityAuthentication managedIdentity => new BlobServiceClient(managedIdentity.ServiceUri, CreateCredential(managedIdentity)), + _ => throw new ArgumentOutOfRangeException(nameof(settings), settings.Authentication, "Unknown Azure Blob authentication.") + }; + + return serviceClient.GetBlobContainerClient(settings.ContainerName); + } + + static TokenCredential CreateCredential(AzureBlobManagedIdentityAuthentication authentication) + { + var options = new DefaultAzureCredentialOptions(); + + if (authentication.AuthorityHost is { } authorityHost) + { + options.AuthorityHost = authorityHost; + } + + if (authentication.ClientId is { Length: > 0 } clientId) + { + options.ManagedIdentityClientId = clientId; + } + + return new DefaultAzureCredential(options); + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorage.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorage.cs new file mode 100644 index 0000000000..7270626b14 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorage.cs @@ -0,0 +1,179 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using System.IO.Compression; +using System.Text; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +/// +/// Stores Audit message bodies as files, each one a small header followed by the body, in one directory per ingestion hour. +/// +/// +/// Bodies are immutable, so an existing file is left untouched. +/// +public class AuditFileSystemBodyStorage(FileSystemBodyStorageSettings settings) : IAuditBodyStoragePersistence +{ + const int FormatVersion = 1; + + string StoragePath => settings.StoragePath; + + public async Task WriteBody(string bodyId, DateTime createdOn, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) + { + Directory.CreateDirectory(HourDirectory(createdOn)); + + var filePath = GetBodyFilePath(createdOn, bodyId); + + if (File.Exists(filePath)) + { + return; + } + + // A unique temp name lets concurrent writers of the same body race without clobbering. + var tempFilePath = $"{filePath}.{Guid.NewGuid():N}.tmp"; + + try + { + var fileStream = new FileStream(tempFilePath, FileMode.CreateNew, FileAccess.Write, FileShare.None, bufferSize: 4096, useAsync: true); + await using (fileStream.ConfigureAwait(false)) + { + var shouldCompress = body.Length >= settings.MinCompressionSize; + + using (var writer = new BinaryWriter(fileStream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write(FormatVersion); + writer.Write(contentType); + writer.Write(body.Length); + writer.Write(shouldCompress); + writer.Flush(); + } + + if (shouldCompress) + { + var brotliStream = new BrotliStream(fileStream, CompressionLevel.Fastest, leaveOpen: true); + await using (brotliStream.ConfigureAwait(false)) + { + await brotliStream.WriteAsync(body, cancellationToken).ConfigureAwait(false); + } + } + else + { + await fileStream.WriteAsync(body, cancellationToken).ConfigureAwait(false); + } + + await fileStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + + try + { + File.Move(tempFilePath, filePath, overwrite: false); + } + catch (IOException) when (File.Exists(filePath)) + { + // A concurrent writer already produced the (immutable) body; discard our copy. + TryDelete(tempFilePath); + } + } +#pragma warning disable PS0019 // The catch rethrows everything, cancellation included, so splitting + // out OperationCanceledException would duplicate the cleanup for no change in behaviour. + catch + { + TryDelete(tempFilePath); + throw; + } +#pragma warning restore PS0019 + } + + public Task ReadBody(string bodyId, DateTime createdOn, CancellationToken cancellationToken = default) + { + var filePath = GetBodyFilePath(createdOn, bodyId); + + if (!File.Exists(filePath)) + { + return Task.FromResult(null); + } + + FileStream? fileStream = null; + try + { + fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize: 4096, useAsync: true); + + var reader = new BinaryReader(fileStream, Encoding.UTF8, leaveOpen: true); + + var formatVersion = reader.ReadInt32(); + if (formatVersion != FormatVersion) + { + throw new InvalidOperationException($"Unsupported body file format version {formatVersion} for {bodyId}."); + } + + var contentType = reader.ReadString(); + var bodySize = reader.ReadInt32(); + var isCompressed = reader.ReadBoolean(); + + // The returned stream owns fileStream and is disposed by the caller. Both branches are + // wrapped so the body is never exposed as a seekable stream: fileStream is positioned + // past the header, but its Length still counts the header, and ASP.NET Core would set + // Content-Length from that and then write only the bytes after the current position. + Stream bodyStream = isCompressed + ? new ExpectedLengthStream(new BrotliStream(fileStream, CompressionMode.Decompress, leaveOpen: false), bodySize) + : new ExpectedLengthStream(fileStream, bodySize); + + return Task.FromResult(new MessageBodyFileResult + { + Stream = bodyStream, + ContentType = contentType, + BodySize = bodySize + }); + } + catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException) + { + fileStream?.Dispose(); + return Task.FromResult(null); + } + catch + { + fileStream?.Dispose(); + throw; + } + } + + // The hour directory is owned by Audit storage, so deleting it recursively also removes leftover temp files. + public Task DeleteBodiesForHour(DateTime hour, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + try + { + Directory.Delete(HourDirectory(hour), recursive: true); + } + catch (DirectoryNotFoundException) + { + // Nothing stored for this hour. + } + + return Task.CompletedTask; + } + + public async Task DeleteBodiesForDay(DateTime day, CancellationToken cancellationToken = default) + { + for (var hourIndex = 0; hourIndex < 24; hourIndex++) + { + await DeleteBodiesForHour(day.Date.AddHours(hourIndex), cancellationToken).ConfigureAwait(false); + } + } + + string HourDirectory(DateTime hour) => Path.Combine(StoragePath, AuditBodyStorageKey.HourPrefix(hour)); + + string GetBodyFilePath(DateTime createdOn, string bodyId) => Path.Combine(StoragePath, $"{AuditBodyStorageKey.For(createdOn, bodyId)}.body"); + + static void TryDelete(string filePath) + { + try + { + File.Delete(filePath); + } + catch (DirectoryNotFoundException) + { + // Nothing to delete. + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorageCustomCheck.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorageCustomCheck.cs new file mode 100644 index 0000000000..d73724a86a --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditFileSystemBodyStorageCustomCheck.cs @@ -0,0 +1,53 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using NServiceBus.CustomChecks; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; + +public class AuditFileSystemBodyStorageCustomCheck(FileSystemBodyStorageSettings settings, IDriveSpaceProvider driveSpaceProvider, ILogger logger) + : CustomCheck("ServiceControl body storage", "Storage space", TimeSpan.FromMinutes(15)) +{ + public override Task PerformCheck(CancellationToken cancellationToken = default) + { + logger.LogDebug("Check ServiceControl body storage drive space remaining custom check starting. Threshold {PercentageThreshold:P0}", percentageThreshold); + + if (string.IsNullOrEmpty(dataPathRoot)) + { + return CheckResult.Failed($"Unable to find the root of the message body storage path '{settings.StoragePath}'. An absolute path is required."); + } + + DriveSpace driveSpace; + + try + { + driveSpace = driveSpaceProvider.GetDriveSpace(dataPathRoot); + } + catch (Exception ex) + { + // Custom checks report an exception as an opaque failure, so the reason is captured here instead. + logger.LogError(ex, "Unable to read the free space of drive '{DataPathRoot}' for the message body storage path '{StoragePath}'", dataPathRoot, settings.StoragePath); + + return CheckResult.Failed($"Unable to read the free space of drive '{dataPathRoot}' for the message body storage path '{settings.StoragePath}' on '{Environment.MachineName}': {ex.Message}"); + } + + if (driveSpace.TotalSize <= 0) + { + return CheckResult.Failed($"Unable to determine the size of drive '{driveSpace.Name}' for the message body storage path '{settings.StoragePath}' on '{Environment.MachineName}'."); + } + + var percentRemaining = (decimal)driveSpace.AvailableFreeSpace / driveSpace.TotalSize; + + logger.LogDebug("Free space: {FreeSpaceTotalBytesFree:N0}B | Total: {FreeSpaceTotalBytesAvailable:N0}B | Remaining {PercentRemaining:P1}", driveSpace.AvailableFreeSpace, driveSpace.TotalSize, percentRemaining); + + return percentRemaining > percentageThreshold + ? CheckResult.Pass + : CheckResult.Failed($"{percentRemaining:P0} disk space remaining on the message body storage drive '{driveSpace.Name}' ({settings.StoragePath}) on '{Environment.MachineName}'."); + } + + readonly string? dataPathRoot = Path.GetPathRoot(settings.StoragePath); + readonly decimal percentageThreshold = settings.DataSpaceRemainingThreshold / 100m; +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3BodyStorage.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3BodyStorage.cs new file mode 100644 index 0000000000..88a89d7aeb --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3BodyStorage.cs @@ -0,0 +1,143 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using System.IO.Compression; +using System.Net; +using Amazon.S3; +using Amazon.S3.Model; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Infrastructure; +using ServiceControl.Infrastructure; + +public class AuditS3BodyStorage : IAuditBodyStoragePersistence +{ + const string FormatVersion = "1"; + + readonly IAmazonS3 client; + readonly string bucketName; + readonly string keyPrefix; + readonly int minBodySizeForCompression; + + public AuditS3BodyStorage(S3BodyStorageSettings settings) + { + client = AuditS3ClientFactory.Create(settings); + bucketName = settings.BucketName; + keyPrefix = settings.KeyPrefix; + minBodySizeForCompression = settings.MinCompressionSize; + } + + public async Task WriteBody(string bodyId, DateTime createdOn, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) + { + var key = Key(AuditBodyStorageKey.For(createdOn, bodyId)); + + // Bodies are immutable, so an existing object is already correct. + if (await Exists(key, cancellationToken)) + { + return; + } + + var compressed = body.Length >= minBodySizeForCompression ? BodyCompression.TryCompress(body) : null; + + var payload = compressed is null ? body : compressed.AsMemory(); + var request = new PutObjectRequest + { + BucketName = bucketName, + Key = key, + InputStream = new ReadOnlyStream(payload), + ContentType = contentType + }; + request.Metadata.Add("format-version", FormatVersion); + request.Metadata.Add("body-size", body.Length.ToString()); + request.Metadata.Add("is-compressed", (compressed is not null).ToString()); + + await client.PutObjectAsync(request, cancellationToken); + } + + public async Task ReadBody(string bodyId, DateTime createdOn, CancellationToken cancellationToken = default) + { + try + { + var response = await client.GetObjectAsync(bucketName, Key(AuditBodyStorageKey.For(createdOn, bodyId)), cancellationToken); + try + { + var metadata = response.Metadata; + + var version = metadata["format-version"]; + if (!string.IsNullOrEmpty(version) && version != FormatVersion) + { + throw new InvalidOperationException($"Unsupported object format version {version} for {bodyId}."); + } + + var bodySize = int.TryParse(metadata["body-size"], out var size) ? size : 0; + var isCompressed = bool.TryParse(metadata["is-compressed"], out var compressed) && compressed; + var contentType = response.Headers.ContentType ?? "application/octet-stream"; + Stream stream = isCompressed + ? new ExpectedLengthStream(new BrotliStream(response.ResponseStream, CompressionMode.Decompress), bodySize) + : response.ResponseStream; + + return new MessageBodyFileResult + { + Stream = new OwnedStream(stream, response), + ContentType = contentType, + BodySize = bodySize + }; + } + catch + { + response.Dispose(); + throw; + } + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + return null; + } + } + + public Task DeleteBodiesForHour(DateTime hour, CancellationToken cancellationToken = default) => + DeleteBodiesWithPrefix(AuditBodyStorageKey.HourPrefix(hour), cancellationToken); + + public Task DeleteBodiesForDay(DateTime day, CancellationToken cancellationToken = default) => + DeleteBodiesWithPrefix(AuditBodyStorageKey.DayPrefix(day), cancellationToken); + + async Task DeleteBodiesWithPrefix(string prefix, CancellationToken cancellationToken) + { + var request = new ListObjectsV2Request { BucketName = bucketName, Prefix = Key(prefix) }; + ListObjectsV2Response page; + do + { + page = await client.ListObjectsV2Async(request, cancellationToken); + + if (page.S3Objects is { Count: > 0 }) + { + var deleted = await client.DeleteObjectsAsync(new DeleteObjectsRequest + { + BucketName = bucketName, + Objects = page.S3Objects.Select(o => new KeyVersion { Key = o.Key }).ToList() + }, cancellationToken); + + if (deleted.DeleteErrors is { Count: > 0 }) + { + var failed = deleted.DeleteErrors[0]; + throw new InvalidOperationException($"Failed to delete {deleted.DeleteErrors.Count} object(s) with prefix '{request.Prefix}', first '{failed.Key}': {failed.Code} {failed.Message}"); + } + } + + request.ContinuationToken = page.NextContinuationToken; + } while (page.IsTruncated == true); + } + + async Task Exists(string key, CancellationToken cancellationToken) + { + try + { + await client.GetObjectMetadataAsync(bucketName, key, cancellationToken); + return true; + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + return false; + } + } + + string Key(string bodyId) => $"{keyPrefix}{bodyId}"; +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3ClientFactory.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3ClientFactory.cs new file mode 100644 index 0000000000..a166542e1a --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/AuditS3ClientFactory.cs @@ -0,0 +1,39 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using Amazon; +using Amazon.Runtime; +using Amazon.S3; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; + +static class AuditS3ClientFactory +{ + public static IAmazonS3 Create(S3BodyStorageSettings settings) + { + var config = new AmazonS3Config(); + + var serviceUrl = settings.ServiceUrl; + if (!string.IsNullOrEmpty(serviceUrl)) + { + config.ServiceURL = serviceUrl; + config.ForcePathStyle = true; // Required for S3-compatible endpoints (MinIO, LocalStack). + } + + var region = settings.Region; + if (!string.IsNullOrEmpty(region)) + { + if (!string.IsNullOrEmpty(serviceUrl)) + { + config.AuthenticationRegion = region; + } + else + { + config.RegionEndpoint = RegionEndpoint.GetBySystemName(region); + } + } + + // With no static keys the SDK's default credential chain resolves the ambient IAM role. + return settings.Credentials is { } credentials + ? new AmazonS3Client(new BasicAWSCredentials(credentials.AccessKeyId, credentials.SecretAccessKey), config) + : new AmazonS3Client(config); + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs new file mode 100644 index 0000000000..67ddac0df5 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/BodyCompression.cs @@ -0,0 +1,39 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +using System.Buffers; +using System.IO.Compression; + +/// +/// Compresses message bodies on their way into cloud storage. +/// +/// +/// Brotli ships in the BCL, so it costs no extra dependency, and at quality 1 it compresses about as +/// quickly as gzip or deflate while producing a smaller payload. The quality is pinned deliberately +/// low because this runs on the ingestion path, where throughput matters more than the last few +/// percent of ratio; window 22 is the .NET default. +/// produces the same format through BrotliStream, which maps CompressionLevel.Fastest onto exactly +/// these parameters, so the two paths stay interchangeable without sharing this buffer-based helper. +/// Decompression is not here on purpose: every read path streams, so nothing needs the whole body in +/// memory at once. +/// +static class BodyCompression +{ + /// + /// Returns the Brotli-compressed bytes, or null when compression fails and the caller should + /// store the body uncompressed. + /// + public static byte[]? TryCompress(ReadOnlyMemory body) + { + var buffer = ArrayPool.Shared.Rent(BrotliEncoder.GetMaxCompressedLength(body.Length)); + try + { + return BrotliEncoder.TryCompress(body.Span, buffer, out var written, quality: 1, window: 22) + ? buffer.AsSpan(0, written).ToArray() + : null; + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs new file mode 100644 index 0000000000..b0d50f3c9d --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/DriveInfoSpaceProvider.cs @@ -0,0 +1,11 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +public class DriveInfoSpaceProvider : IDriveSpaceProvider +{ + public DriveSpace GetDriveSpace(string pathRoot) + { + var driveInfo = new DriveInfo(pathRoot); + + return new DriveSpace(driveInfo.Name, driveInfo.AvailableFreeSpace, driveInfo.TotalSize); + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs new file mode 100644 index 0000000000..daccd6b57d --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/BodyStorage/IDriveSpaceProvider.cs @@ -0,0 +1,8 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +public interface IDriveSpaceProvider +{ + DriveSpace GetDriveSpace(string pathRoot); +} + +public readonly record struct DriveSpace(string Name, long AvailableFreeSpace, long TotalSize); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs index 41c0966eb7..6c97dad0d8 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/DatabaseSetup.cs @@ -1,21 +1,30 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation; +using Amazon.S3.Model; +using Amazon.S3.Util; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.DbContexts; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; sealed class DatabaseSetup( IServiceScopeFactory scopeFactory, IAuditPartitionManager partitions, + EFPersisterSettings settings, TimeProvider timeProvider, ILogger logger) : IHostedService { public async Task StartAsync(CancellationToken cancellationToken = default) { + if (settings.BodyStorage is not null) + { + await ProvisionBodyStorage(settings.BodyStorage, cancellationToken); + } + await using var scope = scopeFactory.CreateAsyncScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); dbContext.Database.SetCommandTimeout(EFPersisterSettings.MigrationCommandTimeout); @@ -30,5 +39,32 @@ public async Task StartAsync(CancellationToken cancellationToken = default) logger.LogInformation("Audit database migration completed"); } + static Task ProvisionBodyStorage(BodyStorageSettings bodyStorage, CancellationToken cancellationToken) + { + switch (bodyStorage) + { + case FileSystemBodyStorageSettings fileSystem: + cancellationToken.ThrowIfCancellationRequested(); + Directory.CreateDirectory(fileSystem.StoragePath); + return Task.CompletedTask; + case AzureBlobBodyStorageSettings azureBlob: + return AuditAzureBlobClientFactory.CreateContainerClient(azureBlob).CreateIfNotExistsAsync(cancellationToken: cancellationToken); + case S3BodyStorageSettings s3: + return ProvisionS3Bucket(s3, cancellationToken); + default: + throw new ArgumentOutOfRangeException(nameof(bodyStorage), bodyStorage, "Unknown body storage type."); + } + } + + static async Task ProvisionS3Bucket(S3BodyStorageSettings s3, CancellationToken cancellationToken) + { + using var client = AuditS3ClientFactory.Create(s3); + + if (!await AmazonS3Util.DoesS3BucketExistV2Async(client, s3.BucketName)) + { + await client.PutBucketAsync(new PutBucketRequest { BucketName = s3.BucketName }, cancellationToken); + } + } + public Task StopAsync(CancellationToken cancellationToken = default) => Task.CompletedTask; } diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs index ac5b2b928e..d479e4205e 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWork.cs @@ -13,18 +13,25 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.UnitOfWork; using ServiceControl.SagaAudit; -sealed class AuditIngestionUnitOfWork(IServiceScopeFactory scopeFactory, EFPersisterSettings settings, DateTime createdOn) : IAuditIngestionUnitOfWork +sealed class AuditIngestionUnitOfWork(IServiceScopeFactory scopeFactory, IAuditBodyStoragePersistence? bodyStorage, EFPersisterSettings settings, DateTime createdOn) : IAuditIngestionUnitOfWork { readonly List messages = []; readonly List snapshots = []; + readonly List bodyWrites = []; public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMemory body = default, CancellationToken cancellationToken = default) { var headers = processedMessage.Headers; var metadata = processedMessage.MessageMetadata; var uniqueMessageId = ToGuid(processedMessage.UniqueMessageId); - var (bodyState, bodyText) = MessageBodyClassifier.Classify(headers, body, settings.MaxBodySizeToStore); + var (bodyState, bodyText) = MessageBodyClassifier.Classify(headers, body, settings.MaxBodySizeToStore, bodyStorage is not null); var bodyId = AuditBodyId.Format(createdOn, uniqueMessageId); + var contentType = headers.GetValueOrDefault(Headers.ContentType, "text/plain"); + + if (bodyStorage is not null && bodyState == BodyState.External) + { + bodyWrites.Add(bodyStorage.WriteBody(uniqueMessageId.ToString(), createdOn, body, contentType, cancellationToken)); + } processedMessage.Id ??= bodyId; metadata["ContentLength"] = body.Length; @@ -60,7 +67,7 @@ public Task RecordProcessedMessage(ProcessedMessage processedMessage, ReadOnlyMe BodyText = bodyText, BodyState = bodyState, BodySize = body.Length, - BodyContentType = headers.GetValueOrDefault(Headers.ContentType, "text/plain") + BodyContentType = contentType }); return Task.CompletedTask; @@ -93,6 +100,9 @@ public async Task Complete(CancellationToken cancellationToken = default) return; } + // External bodies are written before the rows that point at them + await Task.WhenAll(bodyWrites); + await using var scope = scopeFactory.CreateAsyncScope(); var dbContext = scope.ServiceProvider.GetRequiredService(); @@ -105,7 +115,7 @@ await dbContext.Database.CreateExecutionStrategy().ExecuteAsync(async token => }, cancellationToken); } - public ValueTask DisposeAsync() => ValueTask.CompletedTask; + public async ValueTask DisposeAsync() => await Task.WhenAll(bodyWrites).ConfigureAwait(ConfigureAwaitOptions.SuppressThrowing); static Guid ToGuid(string? uniqueMessageId) => uniqueMessageId is null ? Guid.NewGuid() diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs index 4bae7be84d..92e2674317 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/AuditIngestionUnitOfWorkFactory.cs @@ -5,12 +5,12 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using ServiceControl.Audit.Persistence.EFCore.Infrastructure; using ServiceControl.Audit.Persistence.UnitOfWork; -sealed class AuditIngestionUnitOfWorkFactory(IServiceScopeFactory scopeFactory, EFPersisterSettings settings, TimeProvider timeProvider) : IAuditIngestionUnitOfWorkFactory +sealed class AuditIngestionUnitOfWorkFactory(IServiceScopeFactory scopeFactory, EFPersisterSettings settings, TimeProvider timeProvider, IAuditBodyStoragePersistence? bodyStorage = null) : IAuditIngestionUnitOfWorkFactory { // Fixed for the whole batch, so the body ids it records match the rows it writes. public ValueTask StartNew(int batchSize, CancellationToken cancellationToken = default) => ValueTask.FromResult( - new AuditIngestionUnitOfWork(scopeFactory, settings, AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime))); + new AuditIngestionUnitOfWork(scopeFactory, bodyStorage, settings, AuditHours.Truncate(timeProvider.GetUtcNow().UtcDateTime))); public bool CanIngestMore() => true; diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs index 6a20dab471..9b251ded23 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Implementation/UnitOfWork/MessageBodyClassifier.cs @@ -4,9 +4,15 @@ namespace ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; using NServiceBus; using ServiceControl.Audit.Persistence.EFCore.Entities; +/// +/// Decides whether a body lives inline in the database or in external storage. +/// +/// +/// Without external storage, a body that cannot be stored inline is dropped (TooLarge or NotText). +/// static class MessageBodyClassifier { - public static (BodyState State, string? Text) Classify(IReadOnlyDictionary headers, ReadOnlyMemory body, int maxBodySizeToStore) + public static (BodyState State, string? Text) Classify(IReadOnlyDictionary headers, ReadOnlyMemory body, int maxBodySizeToStore, bool hasExternalStorage) { if (body.IsEmpty) { @@ -14,7 +20,7 @@ public static (BodyState State, string? Text) Classify(IReadOnlyDictionary maxBodySizeToStore; - var notStored = tooLarge ? BodyState.TooLarge : BodyState.NotText; + var notStored = hasExternalStorage ? BodyState.External : tooLarge ? BodyState.TooLarge : BodyState.NotText; if (!MightBeText(headers)) { @@ -40,7 +46,7 @@ public static (BodyState State, string? Text) Classify(IReadOnlyDictionary headers) diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyStorageKey.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyStorageKey.cs new file mode 100644 index 0000000000..8f5a38144c --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/AuditBodyStorageKey.cs @@ -0,0 +1,17 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +using System.Globalization; + +// Keys format the clock fields as supplied and never convert the time zone: EF can return Unspecified kinds. +// The fixed-width hour followed by '/' is what lets the day prefix match every hour of that day. +static class AuditBodyStorageKey +{ + public static string For(DateTime createdOn, string bodyId) => + string.Create(CultureInfo.InvariantCulture, $"{createdOn:yyyyMMddHH}/{bodyId}"); + + public static string HourPrefix(DateTime hour) => + string.Create(CultureInfo.InvariantCulture, $"{hour:yyyyMMddHH}/"); + + public static string DayPrefix(DateTime day) => + string.Create(CultureInfo.InvariantCulture, $"{day:yyyyMMdd}"); +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs new file mode 100644 index 0000000000..3b7c446a2a --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/ExpectedLengthStream.cs @@ -0,0 +1,65 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +/// +/// Validates the byte count of a forward-only stream without buffering its contents. +/// +/// +/// Compressed body objects record their uncompressed length in metadata. Before cloud reads were +/// streamed, eager decompression verified that length before returning the body. This wrapper +/// preserves that integrity check at end of stream. A consumer that stops reading early does not +/// perform the final length check, which is necessary to keep response streaming and cancellation. +/// +public sealed class ExpectedLengthStream(Stream stream, long expectedLength) : Stream +{ + long bytesRead; + + public override bool CanRead => stream.CanRead; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => throw new NotSupportedException(); + + public override long Position + { + get => bytesRead; + set => throw new NotSupportedException(); + } + + public override void Flush() => throw new NotSupportedException(); + + public override int Read(byte[] buffer, int offset, int count) => ValidateRead(stream.Read(buffer, offset, count)); + + public override int Read(Span buffer) => ValidateRead(stream.Read(buffer)); + + public override async Task ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default) => + ValidateRead(await stream.ReadAsync(buffer, offset, count, cancellationToken)); + + public override async ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) => + ValidateRead(await stream.ReadAsync(buffer, cancellationToken)); + + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + + public override void SetLength(long value) => throw new NotSupportedException(); + + public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + + int ValidateRead(int count) + { + bytesRead += count; + if (bytesRead > expectedLength || (count == 0 && bytesRead != expectedLength)) + { + throw new InvalidOperationException("Decompressed body size does not match its metadata."); + } + + return count; + } + + protected override void Dispose(bool disposing) + { + if (disposing) + { + stream.Dispose(); + } + + base.Dispose(disposing); + } +} \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditBodyStoragePersistence.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditBodyStoragePersistence.cs new file mode 100644 index 0000000000..0500bae915 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditBodyStoragePersistence.cs @@ -0,0 +1,30 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +using System; +using System.Threading; +using System.Threading.Tasks; + +/// +/// Stores Audit bodies by explicit UTC ingestion hour and caller-supplied id. +/// +/// +/// Dates retain their supplied UTC clock fields, including Unspecified values read from the database. +/// Repeated writes to the same hour and id keep the first body. +/// +public interface IAuditBodyStoragePersistence +{ + Task WriteBody(string bodyId, DateTime createdOn, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default); + Task ReadBody(string bodyId, DateTime createdOn, CancellationToken cancellationToken = default); + + /// + /// Deletes every body written for the hour of . Implementations throw when a + /// delete fails, so callers never drop the rows that name a body still in the store. + /// + Task DeleteBodiesForHour(DateTime hour, CancellationToken cancellationToken = default); + + /// + /// Deletes every body written during the day of . Failure behavior matches + /// . + /// + Task DeleteBodiesForDay(DateTime day, CancellationToken cancellationToken = default); +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs index 2b6e7de104..1485477268 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/IAuditPartitionManager.cs @@ -8,6 +8,11 @@ interface IAuditPartitionManager Task> ListExpired(AuditDbContext dbContext, DateTime keepFrom, CancellationToken cancellationToken = default); + /// + /// Deletes every body stored for the rows that removes for . + /// + Task DeleteBodies(IAuditBodyStoragePersistence bodyStorage, DateTime expired, CancellationToken cancellationToken = default); + Task DropExpired(AuditDbContext dbContext, DateTime start, int batchSize, CancellationToken cancellationToken = default); Task ProvisionedUntil(AuditDbContext dbContext, CancellationToken cancellationToken = default); diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs new file mode 100644 index 0000000000..b82bcf9731 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/MessageBodyFileResult.cs @@ -0,0 +1,10 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +using System.IO; + +public class MessageBodyFileResult +{ + public required Stream Stream { get; set; } + public required string ContentType { get; set; } + public required int BodySize { get; set; } +} \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/OwnedStream.cs b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/OwnedStream.cs new file mode 100644 index 0000000000..032f3d4e2e --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.EFCore/Infrastructure/OwnedStream.cs @@ -0,0 +1,68 @@ +namespace ServiceControl.Audit.Persistence.EFCore.Infrastructure; + +/// +/// Couples a returned stream to another resource that must remain alive while the stream is read. +/// +/// +/// S3 returns a response whose response stream is consumed later by ASP.NET Core, after the +/// persistence method has returned. Disposing this wrapper disposes both the exposed stream and +/// the owning response, releasing the underlying HTTP connection. +/// +public sealed class OwnedStream(Stream stream, IDisposable owner) : Stream +{ + public override bool CanRead => stream.CanRead; + public override bool CanSeek => stream.CanSeek; + public override bool CanWrite => stream.CanWrite; + public override long Length => stream.Length; + + public override long Position + { + get => stream.Position; + set => stream.Position = value; + } + + public override void Flush() => stream.Flush(); + + public override Task FlushAsync(CancellationToken cancellationToken = default) => stream.FlushAsync(cancellationToken); + + public override int Read(byte[] buffer, int offset, int count) => stream.Read(buffer, offset, count); + + public override int Read(Span buffer) => stream.Read(buffer); + + public override Task ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default) => + stream.ReadAsync(buffer, offset, count, cancellationToken); + + public override ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) => + stream.ReadAsync(buffer, cancellationToken); + + public override long Seek(long offset, SeekOrigin origin) => stream.Seek(offset, origin); + + public override void SetLength(long value) => stream.SetLength(value); + + public override void Write(byte[] buffer, int offset, int count) => stream.Write(buffer, offset, count); + + public override void Write(ReadOnlySpan buffer) => stream.Write(buffer); + + public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken = default) => + stream.WriteAsync(buffer, offset, count, cancellationToken); + + public override ValueTask WriteAsync(ReadOnlyMemory buffer, CancellationToken cancellationToken = default) => + stream.WriteAsync(buffer, cancellationToken); + + protected override void Dispose(bool disposing) + { + if (disposing) + { + try + { + stream.Dispose(); + } + finally + { + owner.Dispose(); + } + } + + base.Dispose(disposing); + } +} diff --git a/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj b/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj index 6b51feb1d8..b663636aef 100644 --- a/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj +++ b/src/ServiceControl.Audit.Persistence.EFCore/ServiceControl.Audit.Persistence.EFCore.csproj @@ -16,6 +16,10 @@ + + + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt index 648f28180a..d536054718 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt @@ -1,2 +1,3 @@ ServiceControl.Audit Health: Audit partition provisioning -ServiceControl.Audit Health: Audit retention \ No newline at end of file +ServiceControl.Audit Health: Audit retention +Storage space: ServiceControl body storage \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyDeletionTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyDeletionTests.cs new file mode 100644 index 0000000000..4f2570fe79 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/BodyDeletionTests.cs @@ -0,0 +1,27 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Threading.Tasks; + using Microsoft.Extensions.Logging.Abstractions; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.PostgreSql; + + [TestFixture] + class BodyDeletionTests + { + [Test] + public async Task A_dropped_partition_deletes_the_bodies_of_its_whole_day() + { + var day = new DateTime(2026, 10, 20, 0, 0, 0, DateTimeKind.Utc); + var storage = new RecordingBodyStorage(); + + await new PostgreSqlAuditPartitionManager(NullLogger.Instance).DeleteBodies(storage, day); + + using (Assert.EnterMultipleScope()) + { + Assert.That(storage.DeletedDays, Is.EqualTo([day])); + Assert.That(storage.DeletedHours, Is.Empty); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs new file mode 100644 index 0000000000..00acbf5528 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PartlyExpiredDayRetentionTests.cs @@ -0,0 +1,47 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.IO; + using System.Threading.Tasks; + using NUnit.Framework; + using ServiceControl.Audit.Infrastructure; + + class PartlyExpiredDayRetentionTests : EFPersistenceTestFixture + { + public override Task Setup() + { + SetSettings = settings => + { + settings.AuditRetentionPeriod = TimeSpan.FromDays(1); + // FakeTimeProvider only moves forward; tomorrow's midday is independent of the current hour. + configuration.TimeProvider.SetUtcNow(new DateTimeOffset(configuration.TimeProvider.GetUtcNow().UtcDateTime.Date.AddDays(1).AddHours(12), TimeSpan.Zero)); + }; + return base.Setup(); + } + + [Test] + public async Task Keeps_the_body_and_row_until_the_whole_day_has_expired() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + var retention = configuration.CreateRetention(); + + // The ingestion hour has expired but the rest of its day has not + configuration.TimeProvider.Advance(TimeSpan.FromHours(25)); + await retention.SweepNow(); + var partlyExpired = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + var filesWhilePartlyExpired = Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories); + + configuration.TimeProvider.Advance(TimeSpan.FromHours(12)); + await retention.SweepNow(); + var expired = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(partlyExpired.Results, Has.Count.EqualTo(1)); + Assert.That(filesWhilePartlyExpired, Has.Length.EqualTo(1)); + Assert.That(expired.Results, Is.Empty); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories), Is.Empty); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs index eed0e5b76c..64bbcb0f6e 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/PostgreSqlEnvironmentDataTests.cs @@ -67,12 +67,16 @@ public async Task Measures_and_identifies_the_search_path_schema_when_none_is_co var services = new ServiceCollection(); services.AddLogging(); services.AddSingleton(configuration.TimeProvider); - new PostgreSqlPersistence(new EFPersisterSettings + + var settings = new EFPersisterSettings { ConnectionString = new NpgsqlConnectionStringBuilder(configuration.ConnectionString) { SearchPath = configuration.Schema }.ConnectionString, AuditRetentionPeriod = TimeSpan.FromDays(1), - MaxBodySizeToStore = 100000 - }).AddPersistence(services); + MaxBodySizeToStore = 100000, + BodyStorage = new FileSystemBodyStorageSettings { StoragePath = configuration.BodyStoragePath } + }; + + new PostgreSqlPersistence(settings).AddPersistence(services); await using var unconfigured = services.BuildServiceProvider(); var identity = await unconfigured.GetRequiredService().GetIdentity(TestTimeoutCancellationToken); diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs index 5bc10040be..da28d495ff 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/RetentionPeriodTests.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using NUnit.Framework; using ServiceControl.Audit.Persistence.EFCore.Abstractions; using ServiceControl.Audit.Persistence.EFCore.PostgreSql; @@ -26,6 +27,8 @@ static PersistenceSettings SettingsWith(TimeSpan retentionPeriod) { var settings = new PersistenceSettings(retentionPeriod, true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = "Host=localhost"; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = Path.GetTempPath(); return settings; } } diff --git a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj index 1ea1c0eec9..afb5f35f72 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj +++ b/src/ServiceControl.Audit.Persistence.Tests.PostgreSql/ServiceControl.Audit.Persistence.Tests.PostgreSql.csproj @@ -21,6 +21,8 @@ + + @@ -36,8 +38,8 @@ - - + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt index cd5cf1e630..8b38d4eddb 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ApprovalFiles/CustomCheckTests.VerifyCustomChecks.approved.txt @@ -1 +1,2 @@ -ServiceControl.Audit Health: Audit retention \ No newline at end of file +ServiceControl.Audit Health: Audit retention +Storage space: ServiceControl body storage \ No newline at end of file diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyDeletionTests.cs b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyDeletionTests.cs new file mode 100644 index 0000000000..ab238f0249 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/BodyDeletionTests.cs @@ -0,0 +1,26 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Threading.Tasks; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.SqlServer; + + [TestFixture] + class BodyDeletionTests + { + [Test] + public async Task An_expired_hour_deletes_the_bodies_of_that_hour() + { + var hour = new DateTime(2026, 10, 20, 13, 0, 0, DateTimeKind.Utc); + var storage = new RecordingBodyStorage(); + + await new SqlServerAuditPartitionManager().DeleteBodies(storage, hour); + + using (Assert.EnterMultipleScope()) + { + Assert.That(storage.DeletedHours, Is.EqualTo([hour])); + Assert.That(storage.DeletedDays, Is.Empty); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj index b2a6fe42b5..256cd9ab39 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/ServiceControl.Audit.Persistence.Tests.SqlServer.csproj @@ -21,6 +21,8 @@ + + @@ -36,8 +38,8 @@ - - + + diff --git a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs index 1ce23546f8..5d789adff9 100644 --- a/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests.SqlServer/SqlServerEnvironmentDataTests.cs @@ -71,7 +71,7 @@ public void Should_report_only_the_service_objective_tier(string serviceObjectiv [TestCase("not a connection string")] public async Task Should_report_edition_and_service_objective_as_unknown_when_the_server_cannot_be_asked(string connectionString) { - var settings = new EFPersisterSettings { ConnectionString = connectionString, AuditRetentionPeriod = TimeSpan.FromDays(7), MaxBodySizeToStore = 1024 }; + var settings = new EFPersisterSettings { ConnectionString = connectionString, AuditRetentionPeriod = TimeSpan.FromDays(7), BodyStorage = new FileSystemBodyStorageSettings { StoragePath = "unused" } }; var probe = new SqlServerDatabaseHostingProbe(settings, new UnreachableScopeFactory(), NullLogger.Instance); var hosting = await probe.Probe(TestTimeoutCancellationToken); diff --git a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs new file mode 100644 index 0000000000..4a256f7833 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.MaxBodySize.cs @@ -0,0 +1,34 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Threading.Tasks; + using NUnit.Framework; + + partial class AuditTests + { + [Test] + public async Task Does_respect_max_message_body() + { + await using var unitOfWork = await StartAuditUnitOfWork(1); + + var body = new byte[MAX_BODY_SIZE + 1000]; + Random.Shared.NextBytes(body); + var processedMessage = MakeMessage(); + + await unitOfWork.RecordProcessedMessage(processedMessage, body); + + await unitOfWork.Complete(); + + var bodyId = GetBodyId(processedMessage); + + var retrievedMessage = await MessagesViewStore.GetMessageBody(bodyId, TestContext.CurrentContext.CancellationToken); + + Assert.That(retrievedMessage, Is.Not.Null); + using (Assert.EnterMultipleScope()) + { + Assert.That(retrievedMessage.Found, Is.True); + Assert.That(retrievedMessage.HasContent, Is.False); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs index 83532c48c4..1926be1dc7 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/AuditTests.cs @@ -100,31 +100,6 @@ public async Task Message_body_validator_is_stable_across_reads() Assert.That(second.Version, Is.EqualTo(first.Version), "the body did not change, so neither may its validator"); } - [Test] - public async Task Does_respect_max_message_body() - { - await using var unitOfWork = await StartAuditUnitOfWork(1); - - var body = new byte[MAX_BODY_SIZE + 1000]; - Random.Shared.NextBytes(body); - var processedMessage = MakeMessage(); - - await unitOfWork.RecordProcessedMessage(processedMessage, body); - - await unitOfWork.Complete(); - - var bodyId = GetBodyId(processedMessage); - - var retrievedMessage = await MessagesViewStore.GetMessageBody(bodyId, TestContext.CurrentContext.CancellationToken); - - Assert.That(retrievedMessage, Is.Not.Null); - using (Assert.EnterMultipleScope()) - { - Assert.That(retrievedMessage.Found, Is.True); - Assert.That(retrievedMessage.HasContent, Is.False); - } - } - [Test] public async Task Does_not_deduplicate_with_different_processing_started_header() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBatchDeleteTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBatchDeleteTests.cs new file mode 100644 index 0000000000..0d3a6317cd --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBatchDeleteTests.cs @@ -0,0 +1,198 @@ +namespace ServiceControl.Audit.Persistence.Tests.EFCore; + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using System.Web; +using Azure; +using Azure.Core.Pipeline; +using Azure.Storage.Blobs; +using NUnit.Framework; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +[TestFixture] +class AuditAzureBlobBatchDeleteTests +{ + static readonly DateTime Hour = new(2026, 10, 20, 10, 0, 0, DateTimeKind.Utc); + + const string ConnectionString = "DefaultEndpointsProtocol=https;AccountName=fake;AccountKey=ZmFrZQ==;EndpointSuffix=core.windows.net"; + + [Test] + public async Task Deletes_in_batches_of_at_most_256_across_listing_pages() + { + var service = new FakeBlobService(Enumerable.Range(0, 600).Select(i => $"2026102010/{i}").Concat(["2026102009/a", "2026102011/a"])) { PageSize = 250 }; + + await CreatePersistence(service).DeleteBodiesForHour(Hour); + + Assert.Multiple(() => + { + Assert.That(service.BatchSizes, Is.EqualTo(new[] { 256, 256, 88 })); + Assert.That(service.Blobs, Is.EquivalentTo(new[] { "2026102009/a", "2026102011/a" })); + }); + } + + [Test] + public async Task Deletes_only_blobs_of_the_requested_hour() + { + var service = new FakeBlobService(["2026102010/a", "2026102011/a"]); + + await CreatePersistence(service).DeleteBodiesForHour(Hour); + + Assert.That(service.Blobs, Is.EqualTo(new[] { "2026102011/a" })); + } + + [Test] + public async Task Sends_no_batch_when_nothing_matches() + { + var service = new FakeBlobService([]); + + await CreatePersistence(service).DeleteBodiesForHour(Hour); + + Assert.That(service.BatchSizes, Is.Empty); + } + + [Test] + public void Treats_a_blob_that_is_already_gone_as_deleted() + { + var service = new FakeBlobService(["2026102010/a", "2026102010/b"]) { StatusFor = name => name.EndsWith("/a") ? 404 : null }; + + Assert.DoesNotThrowAsync(() => CreatePersistence(service).DeleteBodiesForHour(Hour)); + } + + [Test] + public async Task Fails_when_one_delete_in_a_successful_batch_fails_and_a_retry_completes() + { + var service = new FakeBlobService(["2026102010/a", "2026102010/b"]) { StatusFor = name => name.EndsWith("/b") ? 403 : null }; + var store = CreatePersistence(service); + + var ex = Assert.ThrowsAsync(() => store.DeleteBodiesForHour(Hour)); + + Assert.That(ex.Status, Is.EqualTo(403)); + Assert.That(service.Blobs, Is.EqualTo(new[] { "2026102010/b" })); + + service.StatusFor = _ => null; + await store.DeleteBodiesForHour(Hour); + + Assert.That(service.Blobs, Is.Empty); + } + + [Test] + public void Stops_after_a_failing_batch() + { + var service = new FakeBlobService(Enumerable.Range(0, 300).Select(i => $"2026102010/{i}")) { StatusFor = _ => 500 }; + + Assert.ThrowsAsync(() => CreatePersistence(service).DeleteBodiesForHour(Hour)); + Assert.That(service.BatchSizes, Has.Count.EqualTo(1)); + } + + [Test] + public async Task Deletes_every_hour_of_the_day_and_no_other_day() + { + var service = new FakeBlobService(["2026102000/a", "2026102023/a", "2026101923/a", "2026102100/a"]); + + await CreatePersistence(service).DeleteBodiesForDay(Hour); + + Assert.That(service.Blobs, Is.EquivalentTo(new[] { "2026101923/a", "2026102100/a" })); + } + + [Test] + public void Honors_cancellation_before_starting() + { + var service = new FakeBlobService(["2026102010/a"]); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + + Assert.CatchAsync(() => CreatePersistence(service).DeleteBodiesForHour(Hour, cts.Token)); + Assert.That(service.Blobs, Has.Count.EqualTo(1)); + } + + [Test] + public void Honors_cancellation_between_batches() + { + using var cts = new CancellationTokenSource(); + var service = new FakeBlobService(Enumerable.Range(0, 300).Select(i => $"2026102010/{i}")) { AfterBatch = cts.Cancel }; + + Assert.CatchAsync(() => CreatePersistence(service).DeleteBodiesForHour(Hour, cts.Token)); + Assert.That(service.BatchSizes, Has.Count.EqualTo(1), "no batch is sent after cancellation"); + Assert.That(service.Blobs, Has.Count.EqualTo(44)); + } + + static AuditAzureBlobBodyStorage CreatePersistence(FakeBlobService service) + { + var options = new BlobClientOptions + { + Transport = new HttpClientTransport(new HttpClient(service)), + Retry = { MaxRetries = 0 } + }; + return new AuditAzureBlobBodyStorage(new BlobContainerClient(ConnectionString, "bodies", options), 64); + } + + sealed class FakeBlobService(IEnumerable initialBlobs) : HttpMessageHandler + { + static readonly Regex ContentIdPattern = new(@"Content-ID: (\d+)"); + static readonly Regex DeleteTargetPattern = new(@"DELETE /bodies/(\S+?) HTTP/1.1"); + + public List Blobs { get; } = initialBlobs.ToList(); + public List BatchSizes { get; } = []; + public int Requests { get; private set; } + public int PageSize { get; set; } = 1000; + public Action AfterBatch { get; set; } = () => { }; + + // A status for the blob, or null to delete it, or report it missing when it is already gone. + public Func StatusFor { get; set; } = _ => null; + + protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken = default) + { + Requests++; + var query = HttpUtility.ParseQueryString(request.RequestUri!.Query); + + if (query["comp"] == "batch") + { + var response = BatchResponse(await request.Content!.ReadAsStringAsync(cancellationToken)); + AfterBatch(); + return response; + } + + // Stable name markers keep deletions from changing which test blobs appear on the next page. + var marker = query["marker"] ?? ""; + var matching = Blobs.Where(name => name.StartsWith(query["prefix"] ?? "", StringComparison.Ordinal) && string.CompareOrdinal(name, marker) > 0) + .OrderBy(name => name, StringComparer.Ordinal).ToList(); + var page = matching.Take(PageSize).ToList(); + var next = matching.Count > page.Count ? page[^1] : ""; + + var xml = new StringBuilder(""); + foreach (var name in page) + { + xml.Append($"{name}1"); + } + xml.Append($"{next}"); + + return new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent(xml.ToString(), Encoding.UTF8, "application/xml") }; + } + + HttpResponseMessage BatchResponse(string body) + { + var ids = ContentIdPattern.Matches(body).Select(m => m.Groups[1].Value).ToArray(); + var names = DeleteTargetPattern.Matches(body).Select(m => m.Groups[1].Value).ToArray(); + BatchSizes.Add(names.Length); + + var response = new StringBuilder(); + for (var i = 0; i < names.Length; i++) + { + var status = StatusFor(names[i]) ?? (Blobs.Remove(names[i]) ? 202 : 404); + response.Append($"--batchresponse_1\r\nContent-Type: application/http\r\nContent-ID: {ids[i]}\r\n\r\nHTTP/1.1 {status} X\r\nx-ms-request-id: r{i}\r\nx-ms-error-code: Fake\r\n\r\n"); + } + response.Append("--batchresponse_1--\r\n"); + + var content = new StringContent(response.ToString(), Encoding.UTF8); + content.Headers.ContentType = new("multipart/mixed") { Parameters = { new("boundary", "batchresponse_1") } }; + return new HttpResponseMessage(HttpStatusCode.Accepted) { Content = content }; + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBodyStorageTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBodyStorageTests.cs new file mode 100644 index 0000000000..1f27f6906d --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditAzureBlobBodyStorageTests.cs @@ -0,0 +1,214 @@ +namespace ServiceControl.Audit.Persistence.Tests.EFCore; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; +using Testcontainers.Azurite; + +[TestFixture] +[Platform(Exclude = "Win", Reason = "Azurite has no Windows container image")] +class AzureBlobBodyStorageTests +{ + static AzuriteContainer azurite; + + [OneTimeSetUp] + public static async Task StartAzurite() + { + // Azurite 3.36.0 does not implement the service version the current Azure.Storage.Blobs + // package negotiates, so the API version check has to be skipped. Support is milestoned for + // Azurite 3.37.0 (https://github.com/Azure/Azurite/issues/2623); when that ships, move the + // tag forward and drop --skipApiVersionCheck together. + azurite = new AzuriteBuilder("mcr.microsoft.com/azure-storage/azurite:3.36.0") + .WithCommand("--skipApiVersionCheck") + .Build(); + await azurite.StartAsync(); + } + + [OneTimeTearDown] + public static async Task StopAzurite() + { + if (azurite != null) + { + await azurite.DisposeAsync(); + } + } + + public async Task CreateContainer() + { + var settings = new AzureBlobBodyStorageSettings + { + MinCompressionSize = 64, + Authentication = new AzureBlobSharedKeyAuthentication { ConnectionString = azurite.GetConnectionString() }, + ContainerName = $"bodies-{Guid.NewGuid():n}" + }; + + await AuditAzureBlobClientFactory.CreateContainerClient(settings).CreateIfNotExistsAsync(); + return new AuditAzureBlobBodyStorage(settings); + } + + static readonly DateTime Hour10 = new(2026, 10, 20, 10, 0, 0, DateTimeKind.Utc); + static readonly DateTime Hour11 = Hour10.AddHours(1); + static readonly DateTime NextDay = Hour10.Date.AddDays(1); + + [Test] + public async Task Round_trips_a_small_uncompressed_body() + { + var store = await CreateContainer(); + var bodyId = Guid.NewGuid().ToString(); + var body = Encoding.UTF8.GetBytes("hello world"); + + await store.WriteBody(bodyId, Hour10, body, "text/plain"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + await using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + } + + Assert.Multiple(() => + { + Assert.That(result.ContentType, Is.EqualTo("text/plain")); + Assert.That(result.BodySize, Is.EqualTo(body.Length)); + }); + } + + [Test] + public async Task Round_trips_a_large_body_over_the_compression_threshold() + { + var store = await CreateContainer(); + var bodyId = Guid.NewGuid().ToString(); + var body = Encoding.UTF8.GetBytes(new string('a', 100_000)); + + await store.WriteBody(bodyId, Hour10, body, "application/json"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + await using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + } + + Assert.That(result.BodySize, Is.EqualTo(body.Length)); + } + + [Test] + public async Task Returns_null_for_a_missing_body() + { + var store = await CreateContainer(); + Assert.That(await store.ReadBody(Guid.NewGuid().ToString(), Hour10), Is.Null); + } + + [Test] + public async Task Read_for_another_hour_finds_nothing() + { + var store = await CreateContainer(); + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + + Assert.That(await store.ReadBody("a", Hour11), Is.Null); + } + + [Test] + public async Task Delete_for_hour_removes_only_that_hour() + { + var store = await CreateContainer(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10, body, "text/plain"); + await store.WriteBody("b", Hour10, body, "text/plain"); + await store.WriteBody("a", Hour11, body, "text/plain"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("a", Hour10), Is.Null); + Assert.That(await store.ReadBody("b", Hour10), Is.Null); + Assert.That(await store.ReadBody("a", Hour11), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_for_hour_removes_more_bodies_than_fit_in_one_batch() + { + var store = await CreateContainer(); + var body = Encoding.UTF8.GetBytes("payload"); + var ids = Enumerable.Range(0, 300).Select(_ => Guid.NewGuid().ToString()).ToArray(); + await Task.WhenAll(ids.Select(id => store.WriteBody(id, Hour10, body, "text/plain"))); + await store.WriteBody("other", Hour11, body, "text/plain"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(async () => + { + foreach (var id in ids) + { + Assert.That(await store.ReadBody(id, Hour10), Is.Null, id); + } + + Assert.That(await store.ReadBody("other", Hour11), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_for_day_removes_every_hour_of_that_day_only() + { + var store = await CreateContainer(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10.Date, body, "text/plain"); + await store.WriteBody("b", Hour10.Date.AddHours(23), body, "text/plain"); + await store.WriteBody("c", NextDay, body, "text/plain"); + await store.WriteBody("d", Hour10.Date.AddHours(-1), body, "text/plain"); + + await store.DeleteBodiesForDay(Hour10); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("a", Hour10.Date), Is.Null); + Assert.That(await store.ReadBody("b", Hour10.Date.AddHours(23)), Is.Null); + Assert.That(await store.ReadBody("c", NextDay), Is.Not.Null); + Assert.That(await store.ReadBody("d", Hour10.Date.AddHours(-1)), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_succeeds_when_nothing_is_stored() + { + var store = await CreateContainer(); + + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForHour(Hour10)); + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForDay(Hour10)); + } + + [Test] + public async Task Rewriting_an_existing_body_keeps_the_first_write() + { + var store = await CreateContainer(); + var bodyId = Guid.NewGuid().ToString(); + var original = Encoding.UTF8.GetBytes("original"); + + await store.WriteBody(bodyId, Hour10, original, "text/plain"); + await store.WriteBody(bodyId, Hour10, Encoding.UTF8.GetBytes("different"), "text/plain"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + await using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(original), "bodies are immutable, so the first write wins"); + } + } + + static byte[] ReadAll(Stream stream) + { + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + return buffer.ToArray(); + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditFileSystemBodyStorageTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditFileSystemBodyStorageTests.cs new file mode 100644 index 0000000000..7b307b64ab --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditFileSystemBodyStorageTests.cs @@ -0,0 +1,198 @@ +namespace ServiceControl.Audit.Persistence.Tests.EFCore; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + +[TestFixture] +class AuditFileSystemBodyStorageTests +{ + string tempDir; + AuditFileSystemBodyStorage store; + + [SetUp] + public void SetUp() + { + tempDir = Directory.CreateTempSubdirectory("sc-body-tests-").FullName; + store = new AuditFileSystemBodyStorage(new FileSystemBodyStorageSettings { StoragePath = tempDir, MinCompressionSize = 64 }); + } + + [TearDown] + public void TearDown() => Directory.Delete(tempDir, recursive: true); + + static readonly DateTime Hour10 = new(2026, 10, 20, 10, 0, 0, DateTimeKind.Utc); + static readonly DateTime Hour11 = Hour10.AddHours(1); + + [TestCase(11)] + [TestCase(100_000)] + public async Task Round_trips_compressed_and_uncompressed_bodies(int bodyLength) + { + var body = Encoding.UTF8.GetBytes(new string('a', bodyLength)); + await store.WriteBody("body-1", Hour10, body, "application/json"); + + var result = await store.ReadBody("body-1", Hour10); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.Multiple(() => + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + Assert.That(result.ContentType, Is.EqualTo("application/json")); + Assert.That(result.BodySize, Is.EqualTo(bodyLength)); + Assert.That(result.Stream.CanSeek, Is.False, "a seekable stream would advertise the header bytes in its Length"); + }); + } + } + + [Test] + public async Task Body_is_stored_in_the_directory_of_its_explicit_hour_whatever_the_id() + { + await store.WriteBody("opaque", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + + Assert.That(Relative(), Is.EqualTo(new[] { Path.Combine("2026102010", "opaque.body") })); + var result = await store.ReadBody("opaque", Hour10); + Assert.That(result, Is.Not.Null); + result.Stream.Dispose(); + } + + [Test] + public async Task Bodies_with_the_same_id_in_different_hours_are_distinct() + { + await store.WriteBody("same", Hour10, Encoding.UTF8.GetBytes("ten"), "text/plain"); + await store.WriteBody("same", Hour11, Encoding.UTF8.GetBytes("eleven"), "text/plain"); + + using var ten = (await store.ReadBody("same", Hour10)).Stream; + using var eleven = (await store.ReadBody("same", Hour11)).Stream; + + Assert.Multiple(() => + { + Assert.That(ReadAll(ten), Is.EqualTo(Encoding.UTF8.GetBytes("ten"))); + Assert.That(ReadAll(eleven), Is.EqualTo(Encoding.UTF8.GetBytes("eleven"))); + }); + } + + [Test] + public async Task Read_for_another_hour_finds_nothing() + { + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + + Assert.That(await store.ReadBody("a", Hour11), Is.Null); + Assert.That(await store.ReadBody("a", Hour10.AddDays(-1)), Is.Null); + } + + [Test] + public async Task Read_with_an_unspecified_kind_finds_the_body_of_the_same_clock_hour() + { + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + + var result = await store.ReadBody("a", DateTime.SpecifyKind(Hour10, DateTimeKind.Unspecified)); + + Assert.That(result, Is.Not.Null); + result.Stream.Dispose(); + } + + [Test] + public async Task Rewriting_a_body_keeps_the_first_write() + { + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("original"), "text/plain"); + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("different"), "text/plain"); + + using var result = (await store.ReadBody("a", Hour10)).Stream; + + Assert.That(ReadAll(result), Is.EqualTo(Encoding.UTF8.GetBytes("original"))); + } + + [Test] + public async Task Delete_for_hour_removes_only_that_hour() + { + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10, body, "text/plain"); + await store.WriteBody("b", Hour10, body, "text/plain"); + await store.WriteBody("a", Hour11, body, "text/plain"); + var unrelated = Path.Combine(tempDir, "flat.body"); + await File.WriteAllTextAsync(unrelated, "keep"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(() => + { + Assert.That(Relative(), Is.EquivalentTo(new[] { Path.Combine("2026102011", "a.body"), "flat.body" })); + Assert.That(Directory.Exists(Path.Combine(tempDir, "2026102010")), Is.False); + }); + } + + [Test] + public async Task Delete_for_hour_also_removes_temp_files_in_that_hour_only() + { + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + await store.WriteBody("a", Hour11, Encoding.UTF8.GetBytes("payload"), "text/plain"); + await File.WriteAllTextAsync(Path.Combine(tempDir, "2026102010", "a.body.abc.tmp"), "in progress"); + var otherTemp = Path.Combine(tempDir, "2026102011", "a.body.abc.tmp"); + await File.WriteAllTextAsync(otherTemp, "in progress"); + var rootFile = Path.Combine(tempDir, "unrelated.txt"); + await File.WriteAllTextAsync(rootFile, "keep"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(() => + { + Assert.That(Directory.Exists(Path.Combine(tempDir, "2026102010")), Is.False); + Assert.That(File.Exists(otherTemp), Is.True); + Assert.That(File.Exists(rootFile), Is.True); + }); + } + + [Test] + public async Task Delete_for_day_removes_every_hour_of_that_day_and_no_other() + { + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10.Date, body, "text/plain"); + await store.WriteBody("b", Hour10.Date.AddHours(23), body, "text/plain"); + await store.WriteBody("c", Hour10.Date.AddDays(1), body, "text/plain"); + await store.WriteBody("d", Hour10.Date.AddHours(-1), body, "text/plain"); + Directory.CreateDirectory(Path.Combine(tempDir, "20261020-unrelated")); + + await store.DeleteBodiesForDay(Hour10); + + Assert.Multiple(() => + { + Assert.That(Relative(), Is.EquivalentTo(new[] { Path.Combine("2026102100", "c.body"), Path.Combine("2026101923", "d.body") })); + Assert.That(Directory.Exists(Path.Combine(tempDir, "20261020-unrelated")), Is.True); + }); + } + + [Test] + public void Delete_succeeds_when_nothing_is_stored() + { + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForHour(Hour10)); + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForDay(Hour10)); + } + + [Test] + public async Task Delete_honors_cancellation() + { + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + using var cts = new CancellationTokenSource(); + await cts.CancelAsync(); + + Assert.ThrowsAsync(() => store.DeleteBodiesForHour(Hour10, cts.Token)); + Assert.ThrowsAsync(() => store.DeleteBodiesForDay(Hour10, cts.Token)); + Assert.That(Relative(), Has.Length.EqualTo(1)); + } + + string[] Relative() => Directory.GetFiles(tempDir, "*", SearchOption.AllDirectories).Select(f => Path.GetRelativePath(tempDir, f)).ToArray(); + + static byte[] ReadAll(Stream stream) + { + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + return buffer.ToArray(); + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs new file mode 100644 index 0000000000..3a0413262d --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditIngestionUnitOfWorkBodyTests.cs @@ -0,0 +1,154 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Text; + using System.Threading.Tasks; + using Auditing; + using Microsoft.Extensions.DependencyInjection; + using NServiceBus; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + + [TestFixture] + class AuditIngestionUnitOfWorkBodyTests + { + static readonly DateTime CreatedOn = new(2026, 10, 20, 13, 0, 0, DateTimeKind.Utc); + const int Max = 100; + + RecordingBodyStorage storage; + ThrowingScopeFactory scopes; + AuditIngestionUnitOfWork unitOfWork; + + [SetUp] + public void SetUp() + { + storage = new RecordingBodyStorage(); + scopes = new ThrowingScopeFactory(); + var settings = new EFPersisterSettings + { + ConnectionString = "unused", + AuditRetentionPeriod = TimeSpan.FromDays(1), + BodyStorage = new FileSystemBodyStorageSettings { StoragePath = "unused" }, + MaxBodySizeToStore = Max + }; + unitOfWork = new AuditIngestionUnitOfWork(scopes, storage, settings, CreatedOn); + } + + [TearDown] + public async Task TearDown() => await unitOfWork.DisposeAsync(); + + static ProcessedMessage Message(string contentType) + { + var headers = new Dictionary + { + [Headers.MessageId] = Guid.NewGuid().ToString(), + [Headers.ProcessingEndpoint] = "Receiver", + [Headers.ContentType] = contentType + }; + + return new ProcessedMessage(headers, []); + } + + [Test] + public async Task Small_text_is_not_written_externally() + { + await unitOfWork.RecordProcessedMessage(Message("application/json"), Encoding.UTF8.GetBytes("{}")); + + Assert.That(storage.Written, Is.Empty); + } + + [Test] + public async Task An_external_body_is_written_with_the_batch_hour_and_unique_message_id() + { + var message = Message("application/octet-stream"); + byte[] body = [1, 2, 3]; + + await unitOfWork.RecordProcessedMessage(message, body); + + using (Assert.EnterMultipleScope()) + { + Assert.That(storage.Written, Has.Count.EqualTo(1)); + Assert.That(message.Id, Is.EqualTo($"2026102013-{storage.Written[0].BodyId}")); + Assert.That(Guid.TryParse(storage.Written[0].BodyId, out _), Is.True, "the storage id is the raw unique message id"); + Assert.That(storage.Written[0].CreatedOn, Is.EqualTo(CreatedOn)); + Assert.That(storage.Written[0].Body, Is.EqualTo(body)); + Assert.That(storage.Written[0].ContentType, Is.EqualTo("application/octet-stream")); + } + } + + [Test] + public async Task An_oversized_text_body_is_written_in_full() + { + var body = Encoding.UTF8.GetBytes(new string('a', Max * 3)); + + await unitOfWork.RecordProcessedMessage(Message("text/plain"), body); + + Assert.That(storage.Written[0].Body, Is.EqualTo(body)); + } + + [Test] + public async Task Without_a_body_store_messages_can_be_recorded_and_disposed() + { + var noStore = new AuditIngestionUnitOfWork(scopes, null, new EFPersisterSettings { ConnectionString = "unused", AuditRetentionPeriod = TimeSpan.FromDays(1), MaxBodySizeToStore = Max }, CreatedOn); + + await noStore.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + await noStore.RecordProcessedMessage(Message("text/plain"), Encoding.UTF8.GetBytes(new string('a', Max * 3))); + + Assert.That(async () => await noStore.DisposeAsync(), Throws.Nothing); + } + + [Test] + public async Task A_failed_body_write_stops_the_batch_before_any_row_is_inserted() + { + storage.WriteFailure = new InvalidOperationException("storage is down"); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var exception = Assert.ThrowsAsync(() => unitOfWork.Complete()); + + using (Assert.EnterMultipleScope()) + { + Assert.That(exception.Message, Is.EqualTo("storage is down")); + Assert.That(scopes.Used, Is.False, "the database must not be touched once a body write failed"); + } + } + + [Test] + public async Task Disposing_without_completing_waits_for_the_started_body_writes() + { + storage.PendingWrite = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var dispose = unitOfWork.DisposeAsync().AsTask(); + Assert.That(dispose.IsCompleted, Is.False); + + storage.PendingWrite.SetResult(); + await dispose; + } + + [Test] + public async Task Disposing_does_not_throw_when_a_started_body_write_failed() + { + storage.PendingWrite = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await unitOfWork.RecordProcessedMessage(Message("application/octet-stream"), new byte[] { 1 }); + + var dispose = unitOfWork.DisposeAsync().AsTask(); + storage.PendingWrite.SetException(new InvalidOperationException("storage is down")); + + Assert.DoesNotThrowAsync(async () => await dispose); + } + + sealed class ThrowingScopeFactory : IServiceScopeFactory + { + public bool Used { get; private set; } + + public IServiceScope CreateScope() + { + Used = true; + throw new InvalidOperationException("The database was used."); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditS3BodyStorageTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditS3BodyStorageTests.cs new file mode 100644 index 0000000000..864a84a304 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/AuditS3BodyStorageTests.cs @@ -0,0 +1,215 @@ +namespace ServiceControl.Audit.Persistence.Tests.EFCore; + +using System; +using System.IO; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Audit.Persistence.EFCore.Abstractions; +using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; +using Testcontainers.LocalStack; + +[TestFixture] +[Platform(Exclude = "Win", Reason = "LocalStack has no Windows container image")] +class S3BodyStorageTests +{ + static LocalStackContainer localStack; + + [OneTimeSetUp] + public static async Task StartLocalStack() + { + // We need to pin the tag to 4.4.0 because after that version LocalStack has become a paid service for commercial use. + localStack = new LocalStackBuilder("localstack/localstack:4.4.0").Build(); + await localStack.StartAsync(); + } + + [OneTimeTearDown] + public static async Task StopLocalStack() + { + if (localStack != null) + { + await localStack.DisposeAsync(); + } + } + + public async Task CreateBucket() + { + var settings = new S3BodyStorageSettings + { + MinCompressionSize = 64, + ServiceUrl = localStack.GetConnectionString(), + Region = "us-east-1", + Credentials = new S3StaticCredentials { AccessKeyId = "test", SecretAccessKey = "test" }, + BucketName = $"bodies-{Guid.NewGuid():n}" + }; + + using (var client = AuditS3ClientFactory.Create(settings)) + { + await client.PutBucketAsync(settings.BucketName); + } + return new AuditS3BodyStorage(settings); + } + + static readonly DateTime Hour10 = new(2026, 10, 20, 10, 0, 0, DateTimeKind.Utc); + static readonly DateTime Hour11 = Hour10.AddHours(1); + static readonly DateTime NextDay = Hour10.Date.AddDays(1); + + [Test] + public async Task Round_trips_a_small_uncompressed_body() + { + var store = await CreateBucket(); + var bodyId = Guid.NewGuid().ToString(); + var body = Encoding.UTF8.GetBytes("hello world"); + + await store.WriteBody(bodyId, Hour10, body, "text/plain"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + } + + Assert.Multiple(() => + { + Assert.That(result.ContentType, Is.EqualTo("text/plain")); + Assert.That(result.BodySize, Is.EqualTo(body.Length)); + }); + } + + [Test] + public async Task Round_trips_a_large_body_over_the_compression_threshold() + { + var store = await CreateBucket(); + var bodyId = Guid.NewGuid().ToString(); + var body = Encoding.UTF8.GetBytes(new string('a', 100_000)); + + await store.WriteBody(bodyId, Hour10, body, "application/json"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + } + + Assert.That(result.BodySize, Is.EqualTo(body.Length)); + } + + [Test] + public async Task Returns_null_for_a_missing_body() + { + var store = await CreateBucket(); + Assert.That(await store.ReadBody(Guid.NewGuid().ToString(), Hour10), Is.Null); + } + + [Test] + public async Task Read_for_another_hour_finds_nothing() + { + var store = await CreateBucket(); + await store.WriteBody("a", Hour10, Encoding.UTF8.GetBytes("payload"), "text/plain"); + + Assert.That(await store.ReadBody("a", Hour11), Is.Null); + } + + [Test] + public async Task Delete_for_hour_removes_only_that_hour() + { + var store = await CreateBucket(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10, body, "text/plain"); + await store.WriteBody("b", Hour10, body, "text/plain"); + await store.WriteBody("a", Hour11, body, "text/plain"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("a", Hour10), Is.Null); + Assert.That(await store.ReadBody("b", Hour10), Is.Null); + Assert.That(await store.ReadBody("a", Hour11), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_for_hour_removes_more_bodies_than_fit_in_one_batch() + { + var store = await CreateBucket(); + var body = Encoding.UTF8.GetBytes("payload"); + var ids = Enumerable.Range(0, 300).Select(_ => Guid.NewGuid().ToString()).ToArray(); + await Task.WhenAll(ids.Select(id => store.WriteBody(id, Hour10, body, "text/plain"))); + await store.WriteBody("other", Hour11, body, "text/plain"); + + await store.DeleteBodiesForHour(Hour10); + + Assert.Multiple(async () => + { + foreach (var id in ids) + { + Assert.That(await store.ReadBody(id, Hour10), Is.Null, id); + } + + Assert.That(await store.ReadBody("other", Hour11), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_for_day_removes_every_hour_of_that_day_only() + { + var store = await CreateBucket(); + var body = Encoding.UTF8.GetBytes("payload"); + await store.WriteBody("a", Hour10.Date, body, "text/plain"); + await store.WriteBody("b", Hour10.Date.AddHours(23), body, "text/plain"); + await store.WriteBody("c", NextDay, body, "text/plain"); + await store.WriteBody("d", Hour10.Date.AddHours(-1), body, "text/plain"); + + await store.DeleteBodiesForDay(Hour10); + + Assert.Multiple(async () => + { + Assert.That(await store.ReadBody("a", Hour10.Date), Is.Null); + Assert.That(await store.ReadBody("b", Hour10.Date.AddHours(23)), Is.Null); + Assert.That(await store.ReadBody("c", NextDay), Is.Not.Null); + Assert.That(await store.ReadBody("d", Hour10.Date.AddHours(-1)), Is.Not.Null); + }); + } + + [Test] + public async Task Delete_succeeds_when_nothing_is_stored() + { + var store = await CreateBucket(); + + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForHour(Hour10)); + Assert.DoesNotThrowAsync(() => store.DeleteBodiesForDay(Hour10)); + } + + [Test] + [Parallelizable(ParallelScope.None)] + public async Task Rewriting_an_existing_body_keeps_the_first_write() + { + var store = await CreateBucket(); + var bodyId = Guid.NewGuid().ToString(); + var original = Encoding.UTF8.GetBytes("original"); + + await store.WriteBody(bodyId, Hour10, original, "text/plain"); + await store.WriteBody(bodyId, Hour10, Encoding.UTF8.GetBytes("different"), "text/plain"); + + var result = await store.ReadBody(bodyId, Hour10); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(original), "bodies are immutable, so the first write wins"); + } + } + + static byte[] ReadAll(Stream stream) + { + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + return buffer.ToArray(); + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs new file mode 100644 index 0000000000..f2c74ae986 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/BodyStorageSettingsParserTests.cs @@ -0,0 +1,183 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + + [TestFixture] + class BodyStorageSettingsParserTests + { + const int MaxBodySizeToStore = 1234; + + static Dictionary Settings(params (string Key, string Value)[] values) + { + var settings = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var (key, value) in values) + { + settings[key] = value; + } + + return settings; + } + + static BodyStorageSettings Parse(params (string Key, string Value)[] values) => + BodyStorageSettingsParser.Parse(Settings(values), MaxBodySizeToStore); + + [Test] + public void No_storage_type_means_no_external_storage() => + Assert.That(Parse(), Is.Null); + + [Test] + public void An_unknown_storage_type_is_rejected() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "Tape"))); + + [Test] + public void File_system_settings_use_the_defaults() + { + var settings = (FileSystemBodyStorageSettings)Parse( + ("MessageBody/StorageType", "filesystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.StoragePath, Is.EqualTo("/bodies")); + Assert.That(settings.DataSpaceRemainingThreshold, Is.EqualTo(FileSystemBodyStorageSettings.DefaultDataSpaceRemainingThreshold)); + Assert.That(settings.MinCompressionSize, Is.EqualTo(BodyStorageSettings.DefaultMinCompressionSize)); + Assert.That(settings.MaxBodySizeToStore, Is.EqualTo(MaxBodySizeToStore)); + } + } + + [Test] + public void The_file_system_path_is_required() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "FileSystem"))); + + [TestCase("-1")] + [TestCase("101")] + [TestCase("lots")] + public void An_invalid_file_system_threshold_is_rejected(string threshold) => + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "FileSystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies"), + ("MessageBody/FileSystem/DataSpaceRemainingThreshold", threshold))); + + [Test] + public void The_compression_threshold_can_be_set() + { + var settings = Parse( + ("MessageBody/StorageType", "FileSystem"), + ("MessageBody/FileSystem/StoragePath", "/bodies"), + ("MessageBody/MinCompressionSize", "10")); + + Assert.That(settings.MinCompressionSize, Is.EqualTo(10)); + } + + [Test] + public void An_invalid_max_body_size_falls_back_to_the_default() + { + var settings = BodyStorageSettingsParser.Parse( + Settings(("MessageBody/StorageType", "FileSystem"), ("MessageBody/FileSystem/StoragePath", "/bodies")), + 0); + + Assert.That(settings.MaxBodySizeToStore, Is.EqualTo(BodyStorageSettings.DefaultMaxBodySizeToStore)); + } + + [Test] + public void Azure_shared_key_uses_the_audit_container_by_default() + { + var settings = (AzureBlobBodyStorageSettings)Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ConnectionString", "UseDevelopmentStorage=true")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.Authentication, Is.TypeOf()); + Assert.That(settings.ContainerName, Is.EqualTo("audit-bodies")); + } + } + + [Test] + public void Azure_managed_identity_reads_its_options() + { + var settings = (AzureBlobBodyStorageSettings)Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ServiceUri", "https://account.blob.core.windows.net"), + ("MessageBody/Azure/ManagedIdentityClientId", "client"), + ("MessageBody/Azure/ContainerName", "custom")); + + var authentication = (AzureBlobManagedIdentityAuthentication)settings.Authentication; + + using (Assert.EnterMultipleScope()) + { + Assert.That(authentication.ServiceUri, Is.EqualTo(new Uri("https://account.blob.core.windows.net"))); + Assert.That(authentication.ClientId, Is.EqualTo("client")); + Assert.That(settings.ContainerName, Is.EqualTo("custom")); + } + } + + [Test] + public void Azure_requires_exactly_one_authentication_mode() + { + Assert.Throws(() => Parse(("MessageBody/StorageType", "AzureBlob"))); + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ConnectionString", "UseDevelopmentStorage=true"), + ("MessageBody/Azure/ServiceUri", "https://account.blob.core.windows.net"))); + } + + [Test] + public void Azure_rejects_a_relative_service_uri() => + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "AzureBlob"), + ("MessageBody/Azure/ServiceUri", "account"))); + + [Test] + public void S3_uses_the_audit_prefix_by_default() + { + var settings = (S3BodyStorageSettings)Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(settings.BucketName, Is.EqualTo("bucket")); + Assert.That(settings.KeyPrefix, Is.EqualTo("audit-bodies/")); + Assert.That(settings.Credentials, Is.Null); + } + } + + [Test] + public void S3_requires_a_bucket() => + Assert.Throws(() => Parse(("MessageBody/StorageType", "S3"))); + + [Test] + public void S3_credentials_must_come_as_a_pair() + { + Assert.Throws(() => Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket"), + ("MessageBody/S3/AccessKeyId", "key"))); + + var settings = (S3BodyStorageSettings)Parse( + ("MessageBody/StorageType", "S3"), + ("MessageBody/S3/BucketName", "bucket"), + ("MessageBody/S3/AccessKeyId", "key"), + ("MessageBody/S3/SecretAccessKey", "secret")); + + Assert.That(settings.Credentials.AccessKeyId, Is.EqualTo("key")); + } + + [Test] + public void Every_key_is_read_from_the_configuration() + { + Assert.That(new TestConfiguration().ConfigurationKeys, Is.SupersetOf(BodyStorageSettingsParser.Keys)); + } + + sealed class TestConfiguration : EFPersistenceConfigurationBase + { + public override string Name => "Test"; + + protected override IPersistence Create(EFPersisterSettings settings) => throw new NotSupportedException(); + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs index a25ca35b7e..9aa788bfa9 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFBodyTests.cs @@ -1,11 +1,15 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Text; using System.Threading.Tasks; + using Microsoft.EntityFrameworkCore; using NUnit.Framework; using ServiceControl.Audit.Infrastructure; + using ServiceControl.Audit.Persistence.EFCore.Entities; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; class EFBodyTests : EFPersistenceTestFixture { @@ -44,46 +48,100 @@ public async Task Finds_a_body_by_the_unique_message_id_alone() } [Test] - public async Task A_text_body_over_the_limit_is_not_served_but_stays_searchable() + public async Task A_small_text_body_is_kept_inline() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + + Assert.That(Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories), Is.Empty); + } + + [Test] + public async Task A_text_body_over_the_limit_is_served_in_full_and_stays_searchable() { var text = "findme " + new string('x', MaxBodySizeToStore); await Ingest(MakeMessage(), Encoding.UTF8.GetBytes(text)); - var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + var bodyId = await BodyId(); + AuditBodyId.TryParse(bodyId, out var rowCreatedOn, out _); + var body = await MessagesViewStore.GetMessageBody(bodyId, TestContext.CurrentContext.CancellationToken); var found = await MessagesViewStore.QueryMessages("findme", new PagingInfo(), new SortInfo("time_sent", "desc")); using (Assert.EnterMultipleScope()) { - Assert.That(body.Found, Is.True); - Assert.That(body.HasContent, Is.False, "the body is over MaxBodySizeToStore, which answers 204"); + var files = Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories); + Assert.That(files, Has.Length.EqualTo(1)); + Assert.That(Path.GetFileName(Path.GetDirectoryName(files[0])), Is.EqualTo(rowCreatedOn.ToString("yyyyMMddHH")), "bodies are grouped in the hour directory of the row"); + Assert.That(await Read(body), Is.EqualTo(Encoding.UTF8.GetBytes(text))); + Assert.That(body.ContentType, Is.EqualTo("application/json")); + Assert.That(body.ContentLength, Is.EqualTo(text.Length)); + Assert.That(body.Version.HasValue, Is.True); Assert.That(found.Results, Has.Count.EqualTo(1)); } } [Test] - public async Task A_binary_body_is_not_stored() + public async Task A_binary_body_is_served_with_its_exact_bytes() { - await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + byte[] bytes = [0, 1, 2, 255, 254]; + await Ingest(MakeMessage(contentType: "application/octet-stream"), bytes); var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); - Assert.That(body.Found, Is.False); + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Found, Is.True); + Assert.That(body.HasContent, Is.True); + Assert.That(body.ContentType, Is.EqualTo("application/octet-stream")); + Assert.That(body.ContentLength, Is.EqualTo(bytes.Length)); + Assert.That(await Read(body), Is.EqualTo(bytes)); + } } [Test] - public async Task A_body_that_is_not_utf8_is_not_stored() + public async Task A_body_that_is_not_utf8_is_served_with_its_exact_bytes() { - await Ingest(MakeMessage(contentType: null), [0xC3, 0x28]); + byte[] bytes = [0xC3, 0x28]; + await Ingest(MakeMessage(contentType: null), bytes); var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); - Assert.That(body.Found, Is.False); + Assert.That(await Read(body), Is.EqualTo(bytes)); + } + + [Test] + public async Task A_text_body_with_a_nul_is_served_with_its_exact_bytes() + { + byte[] bytes = "a\0b"u8.ToArray(); + await Ingest(MakeMessage(), bytes); + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + Assert.That(await Read(body), Is.EqualTo(bytes)); + } + + [Test] + public async Task A_missing_external_body_answers_no_content() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + foreach (var file in Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories)) + { + File.Delete(file); + } + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Found, Is.True); + Assert.That(body.HasContent, Is.False); + } } [Test] - public async Task A_binary_body_over_the_limit_answers_that_it_was_too_large() + public async Task A_row_written_before_external_storage_that_was_too_large_answers_no_content() { - await Ingest(MakeMessage(contentType: "application/octet-stream"), new byte[MaxBodySizeToStore + 1]); + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + await SetBodyState(BodyState.TooLarge); var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); @@ -94,6 +152,35 @@ public async Task A_binary_body_over_the_limit_answers_that_it_was_too_large() } } + [Test] + public async Task A_row_written_before_external_storage_that_was_not_text_is_not_found() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("text")); + await SetBodyState(BodyState.NotText); + + var body = await MessagesViewStore.GetMessageBody(await BodyId(), TestContext.CurrentContext.CancellationToken); + + Assert.That(body.Found, Is.False); + } + + Task SetBodyState(BodyState state) => + WithDbContext((dbContext, token) => dbContext.AuditMessages.ExecuteUpdateAsync(setters => setters.SetProperty(message => message.BodyState, state), token)); + + static async Task Read(MessageBodyView body) + { + if (body.StreamContent == null) + { + return Encoding.UTF8.GetBytes(body.StringContent); + } + + await using (body.StreamContent) + { + using var buffer = new MemoryStream(); + await body.StreamContent.CopyToAsync(buffer); + return buffer.ToArray(); + } + } + [Test] public async Task A_message_without_a_body_has_no_body_url() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFNoBodyStorageTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFNoBodyStorageTests.cs new file mode 100644 index 0000000000..3184c36d58 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFNoBodyStorageTests.cs @@ -0,0 +1,58 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.IO; + using System.Linq; + using System.Text; + using System.Threading.Tasks; + using NUnit.Framework; + using ServiceControl.Audit.Infrastructure; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + + class EFNoBodyStorageTests : EFPersistenceTestFixture + { + public override Task Setup() + { + SetSettings = settings => + { + settings.MaxBodySizeToStore = MaxBodySizeToStore; + settings.AuditRetentionPeriod = TimeSpan.FromDays(1); + settings.PersisterSpecificSettings.Remove(BodyStorageSettingsParser.StorageTypeKey); + settings.PersisterSpecificSettings.Remove(BodyStorageSettingsParser.FileSystemStoragePathKey); + }; + return base.Setup(); + } + + [Test] + public async Task An_oversized_body_is_dropped_but_stays_searchable() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes("findme " + new string('x', MaxBodySizeToStore))); + + var bodyUrl = (await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc"))).Results.Single().BodyUrl; + var body = await MessagesViewStore.GetMessageBody(bodyUrl.Split('/')[2], TestContext.CurrentContext.CancellationToken); + var found = await MessagesViewStore.QueryMessages("findme", new PagingInfo(), new SortInfo("time_sent", "desc")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(body.Found, Is.True); + Assert.That(body.HasContent, Is.False); + Assert.That(found.Results, Has.Count.EqualTo(1)); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories), Is.Empty); + } + } + + [Test] + public async Task Retention_sweeps_without_a_body_store() + { + await Ingest(MakeMessage(), Encoding.UTF8.GetBytes(new string('x', MaxBodySizeToStore + 1))); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + await configuration.CreateRetention().SweepNow(); + + var remaining = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + Assert.That(remaining.Results, Is.Empty); + } + + const int MaxBodySizeToStore = 1000; + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs index be78469c28..17de4759f9 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFPersistenceTestsConfiguration.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Threading; using System.Threading.Tasks; @@ -9,7 +10,9 @@ namespace ServiceControl.Audit.Persistence.Tests using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Time.Testing; using NServiceBus.CustomChecks; + using NUnit.Framework; using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; using ServiceControl.Audit.Persistence.EFCore.Implementation; using UnitOfWork; @@ -31,6 +34,8 @@ abstract class EFPersistenceTestsConfiguration : IPersistenceTestsConfiguration public string ConnectionString { get; private set; } + public string BodyStoragePath { get; } = Directory.CreateTempSubdirectory("sc_audit_bodies_").FullName; + public string Schema { get; } = $"sc_test_{Guid.NewGuid():N}"; // PostgreSQL keeps only microseconds, so a finer seed would not round trip. @@ -44,6 +49,8 @@ public async Task Configure(Action setSettings) var settings = new PersistenceSettings(TimeSpan.FromDays(1), true, 100000); settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.ConnectionStringKey] = ConnectionString; settings.PersisterSpecificSettings[EFPersistenceConfigurationBase.SchemaKey] = Schema; + settings.PersisterSpecificSettings[BodyStorageSettingsParser.StorageTypeKey] = nameof(BodyStorageType.FileSystem); + settings.PersisterSpecificSettings[BodyStorageSettingsParser.FileSystemStoragePathKey] = BodyStoragePath; setSettings(settings); @@ -62,13 +69,7 @@ public async Task Configure(Action setSettings) hostBuilder.Services.Remove(retention); } - foreach (var type in configuration.GetType().Assembly.DefinedTypes.Concat(typeof(EFPersistenceConfigurationBase).Assembly.DefinedTypes)) - { - if (!type.IsAbstract && type.IsAssignableTo(typeof(ICustomCheck))) - { - hostBuilder.Services.AddTransient(typeof(ICustomCheck), type); - } - } + AddCustomChecks(hostBuilder.Services, configuration.GetType()); host = hostBuilder.Build(); await host.StartAsync(); @@ -80,6 +81,23 @@ public async Task Configure(Action setSettings) BodyStorage = new MessagesViewBodyStorage(MessagesViewStore); } + internal static void AddCustomChecks(IServiceCollection services, Type configurationType) + { + foreach (var type in configurationType.Assembly.DefinedTypes.Concat(typeof(EFPersistenceConfigurationBase).Assembly.DefinedTypes)) + { + if (!type.IsAbstract && type.IsAssignableTo(typeof(ICustomCheck)) && type != typeof(AuditFileSystemBodyStorageCustomCheck)) + { + services.AddTransient(typeof(ICustomCheck), type); + } + } + + // Registered on the endpoint in production, which these tests do not build. + if (services.Any(descriptor => descriptor.ServiceType == typeof(FileSystemBodyStorageSettings))) + { + services.AddTransient(); + } + } + public virtual Task CompleteDBOperation() => Task.CompletedTask; protected virtual IAuditMessagesViewDataStore DecorateMessagesViewStore(IAuditMessagesViewDataStore store) => store; @@ -96,9 +114,21 @@ public async Task Cleanup() { await DropSchema(ConnectionString, Schema); } + + try + { + Directory.Delete(BodyStoragePath, recursive: true); + } + catch (DirectoryNotFoundException) + { + } + catch (IOException ex) + { + await TestContext.Out.WriteLineAsync($"WARNING: Could not delete body storage '{BodyStoragePath}': {ex.Message}"); + } } - internal AuditRetention CreateRetention() => ActivatorUtilities.CreateInstance(ServiceProvider); + internal AuditRetention CreateRetention(params object[] overrides) => ActivatorUtilities.CreateInstance(ServiceProvider, overrides); public abstract Task GetConnectionString(); diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs index fa342f4a46..28e3f996f7 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionCustomCheckTests.cs @@ -111,6 +111,8 @@ public Task> ListExpired(AuditDbContext dbContext, DateT public Task DropExpired(AuditDbContext dbContext, DateTime start, int batchSize, CancellationToken cancellationToken = default) => Task.FromResult(new DropResult(RowsDeleted: 0, Completed: true, Deferred: true)); + public Task DeleteBodies(IAuditBodyStoragePersistence bodyStorage, DateTime expired, CancellationToken cancellationToken = default) => Task.CompletedTask; + public Task ProvisionedUntil(AuditDbContext dbContext, CancellationToken cancellationToken = default) => Task.FromResult(null); } diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs index 3ba9131298..eb2841d8ee 100644 --- a/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/EFRetentionTests.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Audit.Persistence.Tests { using System; + using System.IO; using System.Linq; using System.Threading.Tasks; using Microsoft.Extensions.DependencyInjection; @@ -37,6 +38,60 @@ public async Task Removes_rows_older_than_the_retention_period() Assert.That(remaining.Results.Select(message => message.MessageId), Is.EqualTo(new[] { live.Headers[Headers.MessageId] })); } + [Test] + public async Task Removes_the_external_bodies_with_the_rows() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories), Has.Length.EqualTo(1)); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + await configuration.CreateRetention().SweepNow(); + + var remaining = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + + using (Assert.EnterMultipleScope()) + { + Assert.That(remaining.Results, Is.Empty); + Assert.That(Directory.GetFiles(configuration.BodyStoragePath, "*", SearchOption.AllDirectories), Is.Empty); + } + } + + [Test] + public async Task Deletes_the_bodies_of_the_expired_rows() + { + var message = MakeMessage(contentType: "application/octet-stream"); + await Ingest(message, [1, 2, 3]); + var bodyId = (await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc"))).Results.Single().BodyUrl.Split('/')[2]; + AuditBodyId.TryParse(bodyId, out var createdOn, out _); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + var storage = new RecordingBodyStorage(); + await configuration.CreateRetention(storage).SweepNow(); + + Assert.That(storage.DeletedHours.Concat(storage.DeletedDays), Has.Some.Matches(deleted => deleted == createdOn || deleted == createdOn.Date), + "the hour (SQL Server) or day (PostgreSQL) of the expired rows"); + } + + [Test] + public async Task Keeps_the_rows_when_deleting_their_bodies_fails() + { + await Ingest(MakeMessage(contentType: "application/octet-stream"), [1, 2, 3]); + + configuration.TimeProvider.Advance(TimeSpan.FromDays(3)); + var storage = new RecordingBodyStorage { DeleteFailure = new IOException("storage is down") }; + + Assert.ThrowsAsync(() => configuration.CreateRetention(storage).SweepNow()); + + var remaining = await MessagesViewStore.GetMessages(true, new PagingInfo(), new SortInfo("time_sent", "desc")); + var body = await MessagesViewStore.GetMessageBody(remaining.Results.Single().BodyUrl.Split('/')[2], TestContext.CurrentContext.CancellationToken); + + using (Assert.EnterMultipleScope()) + { + Assert.That(remaining.Results, Has.Count.EqualTo(1)); + Assert.That(body.HasContent, Is.True, "the body is still in storage"); + } + } + [Test] public async Task A_list_version_moves_when_a_sweep_and_an_arrival_leave_the_total_unchanged() { diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs new file mode 100644 index 0000000000..da441ab61a --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/MessageBodyClassifierTests.cs @@ -0,0 +1,163 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Text; + using NServiceBus; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Entities; + using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; + + [TestFixture] + class MessageBodyClassifierTests + { + const int Max = 100; + + static Dictionary Json => new() { [Headers.ContentType] = "application/json" }; + + [TestCase(true)] + [TestCase(false)] + public void An_empty_body_has_no_body(bool hasExternalStorage) + { + var (state, text) = MessageBodyClassifier.Classify(Json, ReadOnlyMemory.Empty, Max, hasExternalStorage); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.None)); + Assert.That(text, Is.Null); + } + } + + [TestCase(true)] + [TestCase(false)] + public void Small_text_stays_inline(bool hasExternalStorage) + { + var (state, text) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes("{}"), Max, hasExternalStorage); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.Stored)); + Assert.That(text, Is.EqualTo("{}")); + } + } + + [TestCase(true)] + [TestCase(false)] + public void Text_at_the_limit_stays_inline(bool hasExternalStorage) + { + var (state, _) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes(new string('a', Max)), Max, hasExternalStorage); + + Assert.That(state, Is.EqualTo(BodyState.Stored)); + } + + [Test] + public void Text_over_the_limit_goes_external_and_keeps_a_prefix() + { + var (state, text) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes(new string('a', Max + 1)), Max, hasExternalStorage: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.EqualTo(new string('a', Max))); + } + } + + [Test] + public void The_prefix_does_not_split_a_multi_byte_character() + { + // 99 ASCII bytes then a 3 byte character that straddles the limit + var body = Encoding.UTF8.GetBytes(new string('a', Max - 1) + "€"); + + var (state, text) = MessageBodyClassifier.Classify(Json, body, Max, hasExternalStorage: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.EqualTo(new string('a', Max - 1))); + } + } + + [TestCase("application/octet-stream")] + [TestCase("application/json; binary")] + public void Binary_content_goes_external_without_a_prefix(string contentType) + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary { [Headers.ContentType] = contentType }, new byte[] { 1, 2, 3 }, Max, hasExternalStorage: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void Encoded_content_goes_external() + { + var headers = new Dictionary { ["Content-Encoding"] = "gzip" }; + + var (state, _) = MessageBodyClassifier.Classify(headers, Encoding.UTF8.GetBytes("text"), Max, hasExternalStorage: true); + + Assert.That(state, Is.EqualTo(BodyState.External)); + } + + [Test] + public void A_body_that_is_not_utf8_goes_external() + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary(), new byte[] { 0xC3, 0x28 }, Max, hasExternalStorage: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void A_body_with_a_nul_goes_external() + { + var (state, text) = MessageBodyClassifier.Classify(Json, "a\0b"u8.ToArray(), Max, hasExternalStorage: true); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.External)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void Without_external_storage_text_over_the_limit_is_dropped_with_a_prefix() + { + var (state, text) = MessageBodyClassifier.Classify(Json, Encoding.UTF8.GetBytes(new string('a', Max + 1)), Max, hasExternalStorage: false); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.TooLarge)); + Assert.That(text, Is.EqualTo(new string('a', Max))); + } + } + + [Test] + public void Without_external_storage_binary_content_is_not_text() + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary { [Headers.ContentType] = "application/octet-stream" }, new byte[] { 1, 2, 3 }, Max, hasExternalStorage: false); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.NotText)); + Assert.That(text, Is.Null); + } + } + + [Test] + public void Without_external_storage_binary_content_over_the_limit_is_too_large() + { + var (state, text) = MessageBodyClassifier.Classify(new Dictionary { [Headers.ContentType] = "application/octet-stream" }, new byte[Max + 1], Max, hasExternalStorage: false); + + using (Assert.EnterMultipleScope()) + { + Assert.That(state, Is.EqualTo(BodyState.TooLarge)); + Assert.That(text, Is.Null); + } + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/OptionalBodyStorageRegistrationTests.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/OptionalBodyStorageRegistrationTests.cs new file mode 100644 index 0000000000..ecd37f16c8 --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/OptionalBodyStorageRegistrationTests.cs @@ -0,0 +1,61 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Linq; + using Microsoft.Extensions.DependencyInjection; + using NServiceBus.CustomChecks; + using NUnit.Framework; + using ServiceControl.Audit.Persistence.EFCore.Abstractions; + using ServiceControl.Audit.Persistence.EFCore.Implementation; + using ServiceControl.Audit.Persistence.EFCore.Implementation.BodyStorage; + using ServiceControl.Audit.Persistence.EFCore.Implementation.UnitOfWork; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + using ServiceControl.Audit.Persistence.UnitOfWork; + + [TestFixture] + class OptionalBodyStorageRegistrationTests + { + [TestCase(false)] + [TestCase(true)] + public void File_system_check_is_registered_once_only_when_configured(bool configured) + { + var services = new ServiceCollection(); + if (configured) + { + services.AddSingleton(new FileSystemBodyStorageSettings { StoragePath = "unused" }); + } + + EFPersistenceTestsConfiguration.AddCustomChecks(services, typeof(OptionalBodyStorageRegistrationTests)); + + Assert.That(services.Count(descriptor => descriptor.ServiceType == typeof(ICustomCheck) + && descriptor.ImplementationType == typeof(AuditFileSystemBodyStorageCustomCheck)), Is.EqualTo(configured ? 1 : 0)); + } + + [Test] + public void Consumers_resolve_without_a_body_store() + { + var settings = new EFPersisterSettings { ConnectionString = "unused", AuditRetentionPeriod = TimeSpan.FromDays(1) }; + var services = new ServiceCollection(); + services.AddSingleton(settings); + services.AddSingleton(TimeProvider.System); + services.AddSingleton(_ => null!); + services.AddSingleton(_ => null!); + services.AddSingleton(_ => null!); + services.AddSingleton(_ => null!); + services.AddSingleton(); + services.AddLogging(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + + using var provider = services.BuildServiceProvider(); + + Assert.DoesNotThrow(() => + { + provider.GetRequiredService(); + provider.GetRequiredService(); + provider.GetRequiredService(); + }); + } + } +} diff --git a/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs b/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs new file mode 100644 index 0000000000..c01501a26e --- /dev/null +++ b/src/ServiceControl.Audit.Persistence.Tests/EFCore/RecordingBodyStorage.cs @@ -0,0 +1,67 @@ +namespace ServiceControl.Audit.Persistence.Tests +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Threading; + using System.Threading.Tasks; + using ServiceControl.Audit.Persistence.EFCore.Infrastructure; + + class RecordingBodyStorage : IAuditBodyStoragePersistence + { + public List<(string BodyId, DateTime CreatedOn, byte[] Body, string ContentType)> Written { get; } = []; + + public List DeletedHours { get; } = []; + + public List DeletedDays { get; } = []; + + public Exception WriteFailure { get; set; } + + public Exception DeleteFailure { get; set; } + + public TaskCompletionSource PendingWrite { get; set; } + + public Task WriteBody(string bodyId, DateTime createdOn, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) + { + if (PendingWrite != null) + { + return PendingWrite.Task; + } + + if (WriteFailure != null) + { + return Task.FromException(WriteFailure); + } + + Written.Add((bodyId, createdOn, body.ToArray(), contentType)); + return Task.CompletedTask; + } + + public Task ReadBody(string bodyId, DateTime createdOn, CancellationToken cancellationToken = default) => + Task.FromResult(null); + + public Task DeleteBodiesForHour(DateTime hour, CancellationToken cancellationToken = default) + { + if (DeleteFailure != null) + { + return Task.FromException(DeleteFailure); + } + + DeletedHours.Add(hour); + return Task.CompletedTask; + } + + public Task DeleteBodiesForDay(DateTime day, CancellationToken cancellationToken = default) + { + if (DeleteFailure != null) + { + return Task.FromException(DeleteFailure); + } + + DeletedDays.Add(day); + return Task.CompletedTask; + } + + public string[] WrittenIds => [.. Written.Select(write => write.BodyId)]; + } +} diff --git a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs index af61ccd47a..3e38d8a7a2 100644 --- a/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs +++ b/src/ServiceControl.Audit.UnitTests/Infrastructure/When_instance_is_setup.cs @@ -48,6 +48,7 @@ public async Task Should_provision_queues() var settings = new Settings("FakeTransport", "FakePersistence") { InstanceName = instanceInputQueueName, + IngestAuditMessages = true, ForwardAuditMessages = true, AssemblyLoadContextResolver = static _ => AssemblyLoadContext.Default }; diff --git a/src/ServiceControl.Audit/App.config b/src/ServiceControl.Audit/App.config index 5ea329b21c..18f35bd6c5 100644 --- a/src/ServiceControl.Audit/App.config +++ b/src/ServiceControl.Audit/App.config @@ -23,6 +23,8 @@ These settings are only here so that we can debug ServiceControl while developin + + diff --git a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs index 0d58e3e76d..4201675811 100644 --- a/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs +++ b/src/ServiceControl.Persistence.EFCore/Abstractions/EFPersisterSettings.cs @@ -1,5 +1,6 @@ namespace ServiceControl.Persistence.EFCore.Abstractions; + public abstract class EFPersisterSettings : PersistenceSettings { public static readonly TimeSpan MigrationCommandTimeout = TimeSpan.FromMinutes(40); diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs index f07ab437a7..af739bc38d 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/AzureBlobBodyStoragePersistence.cs @@ -21,6 +21,12 @@ public AzureBlobBodyStoragePersistence(AzureBlobBodyStorageSettings settings) minBodySizeForCompression = settings.MinCompressionSize; } + internal AzureBlobBodyStoragePersistence(BlobContainerClient container, int minBodySizeForCompression) + { + this.container = container; + this.minBodySizeForCompression = minBodySizeForCompression; + } + public async Task WriteBody(string bodyId, ReadOnlyMemory body, string contentType, CancellationToken cancellationToken = default) { var blob = container.GetBlobClient(bodyId); @@ -45,7 +51,7 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con { await blob.UploadAsync(data, options, cancellationToken); } - catch (RequestFailedException ex) when (ex.Status is (int)HttpStatusCode.Conflict or (int)HttpStatusCode.PreconditionFailed) + catch (RequestFailedException ex) when (ex.ErrorCode == BlobErrorCode.BlobAlreadyExists || ex.ErrorCode == BlobErrorCode.ConditionNotMet) { // A blob for this immutable body already exists. } diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs index 2766e08e3d..a1f1db701f 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStorageInstaller.cs @@ -7,6 +7,7 @@ public class FileSystemBodyStorageInstaller(FileSystemBodyStorageSettings settin { public Task Provision(CancellationToken cancellationToken = default) { + cancellationToken.ThrowIfCancellationRequested(); Directory.CreateDirectory(settings.StoragePath); return Task.CompletedTask; diff --git a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs index a9ed66165b..ba4a043e2b 100644 --- a/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.EFCore/Implementation/BodyStorage/FileSystemBodyStoragePersistence.cs @@ -123,7 +123,7 @@ public async Task WriteBody(string bodyId, ReadOnlyMemory body, string con BodySize = bodySize }); } - catch (FileNotFoundException) + catch (Exception ex) when (ex is FileNotFoundException or DirectoryNotFoundException) { fileStream?.Dispose(); return Task.FromResult(null); diff --git a/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStoragePersistenceTests.cs b/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStoragePersistenceTests.cs new file mode 100644 index 0000000000..51b7bf5944 --- /dev/null +++ b/src/ServiceControl.Persistence.Tests/EFCore/FileSystemBodyStoragePersistenceTests.cs @@ -0,0 +1,111 @@ +namespace ServiceControl.Persistence.Tests; + +using System; +using System.IO; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using ServiceControl.Persistence.EFCore.Abstractions; +using ServiceControl.Persistence.EFCore.Implementation.BodyStorage; + +[TestFixture] +class FileSystemBodyStoragePersistenceTests +{ + string tempDir; + FileSystemBodyStoragePersistence store; + + [SetUp] + public void SetUp() + { + tempDir = Directory.CreateTempSubdirectory("sc-body-tests-").FullName; + store = new FileSystemBodyStoragePersistence(new FileSystemBodyStorageSettings { StoragePath = tempDir, MinCompressionSize = 64 }); + } + + [TearDown] + public void TearDown() => Directory.Delete(tempDir, recursive: true); + + [TestCase(11)] + [TestCase(100_000)] + public async Task Round_trips_compressed_and_uncompressed_bodies(int bodyLength) + { + var body = Encoding.UTF8.GetBytes(new string('a', bodyLength)); + await store.WriteBody("body-1", body, "application/json"); + + var result = await store.ReadBody("body-1"); + + Assert.That(result, Is.Not.Null); + using (result.Stream) + { + Assert.Multiple(() => + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(body)); + Assert.That(result.ContentType, Is.EqualTo("application/json")); + Assert.That(result.BodySize, Is.EqualTo(bodyLength)); + Assert.That(result.Stream.CanSeek, Is.False, "a seekable stream would advertise the header bytes in its Length"); + }); + } + } + + [Test] + public async Task Rewriting_a_body_keeps_the_first_write() + { + await store.WriteBody("body-1", Encoding.UTF8.GetBytes("original"), "text/plain"); + await store.WriteBody("body-1", Encoding.UTF8.GetBytes("different"), "text/plain"); + + var result = await store.ReadBody("body-1"); + + using (result.Stream) + { + Assert.That(ReadAll(result.Stream), Is.EqualTo(Encoding.UTF8.GetBytes("original"))); + } + } + + [Test] + public async Task Missing_body_reads_as_null_and_deletes_quietly() + { + Assert.That(await store.ReadBody("missing"), Is.Null); + Assert.DoesNotThrowAsync(() => store.DeleteBodyIfExists("missing")); + } + + [Test] + public async Task Provisioning_creates_the_storage_directory() + { + var path = Path.Combine(tempDir, "nested", "bodies"); + var installer = new FileSystemBodyStorageInstaller(new FileSystemBodyStorageSettings { StoragePath = path }); + + await installer.Provision(); + await installer.Provision(); + + Assert.That(Directory.Exists(path), Is.True); + } + + [Test] + public void Provisioning_honors_cancellation() + { + var path = Path.Combine(tempDir, "cancelled"); + var installer = new FileSystemBodyStorageInstaller(new FileSystemBodyStorageSettings { StoragePath = path }); + using var cancellation = new CancellationTokenSource(); + cancellation.Cancel(); + + Assert.ThrowsAsync(() => installer.Provision(cancellation.Token)); + Assert.That(Directory.Exists(path), Is.False); + } + + [Test] + public async Task Reading_when_the_storage_directory_is_gone_finds_nothing() + { + Directory.Delete(tempDir, recursive: true); + + Assert.That(await store.ReadBody("body-1"), Is.Null); + + Directory.CreateDirectory(tempDir); + } + + static byte[] ReadAll(Stream stream) + { + using var buffer = new MemoryStream(); + stream.CopyTo(buffer); + return buffer.ToArray(); + } +} diff --git a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs index e03d5c130b..142cf5398a 100644 --- a/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs +++ b/src/ServiceControl.Persistence.Tests/EFCore/InMemoryBodyStoragePersistence.cs @@ -1,6 +1,7 @@ namespace ServiceControl.Persistence.Tests; using System; +using System.Linq; using System.Collections.Generic; using System.IO; using System.Threading; diff --git a/tools/testing-tool/README.md b/tools/testing-tool/README.md index 2ecacf82f7..45b0992af1 100644 --- a/tools/testing-tool/README.md +++ b/tools/testing-tool/README.md @@ -12,7 +12,7 @@ The tool runs an NServiceBus endpoint (`TestingTool.Load`) that sends messages through a handler which fails based on the active scenario. Failed messages are routed to the `error` queue for ServiceControl to ingest. Each scenario throws a real exception type from its own method, so ServiceControl's default grouping (exception type + first stack frame) -puts each failure shape in its own group, and immediate retries are disabled so groups stay clean. Five scenarios are built in: +puts each failure shape in its own group, and immediate retries are disabled so groups stay clean. Six scenarios are built in: | Scenario | Category | Failure shape | |---|---|---| @@ -21,6 +21,7 @@ puts each failure shape in its own group, and immediate retries are disabled so | `poison-message` | Poison | 15% deterministic always-fail messages — retry storm | | `deserialization-failure` | Deserialization | 100% fail with `SerializationException` (bad deployment) | | `background-noise` | Noise | ~3% always-on baseline — rotates through four exception types (four small groups) | +| `large-messages` | Body | Starts at 3 msg/s (rate is configurable), each body about 256 KiB of JSON text (the wire envelope is slightly larger), above the 100 KB external body storage threshold. 20% deterministic failures with `InvalidDataException`; the other 80% succeed and are audited | Scenarios are controlled from the web UI, or via: - `GET /api/scenarios` — list scenarios with live status (category, rate, error counts) @@ -111,7 +112,7 @@ tools/testing-tool/ TestingTool/ # ASP.NET Core host — API endpoints, web UI, NServiceBus endpoint, # scenario runner, bypass writer, recoverability/search jobs, # OTel setup, ServiceControl REST client - TestingTool.Scenarios/ # IScenario contract + the five scenario implementations + TestingTool.Scenarios/ # IScenario contract + the six scenario implementations TestingTool.Contracts/ # shared DTOs (scenarios, status, jobs, bypass) TestingTool.AppHost/ # Aspire AppHost — orchestrates platform, tool, and observability stack obs/ # observability config: OTel Collector, Prometheus, Grafana provisioning @@ -189,6 +190,30 @@ separator) or `--name:value` (colon separator): | `--tag` | *(none — uses the current build's image)* | any image tag, e.g. `pr-1234` or `6.3.1` | Override the ServiceControl container image tag (useful for testing PR-based prereleases) | | `--error-ingestion-scale-unit` | `0` | non-negative integer | Number of additional error-ingestion-only scale-out instances to spin up alongside the primary error instance (each runs with `--error-ingestion-only`) | | `--audit-instances` | `1` | non-negative integer | Number of ServiceControl.Audit instances to add. They take the same `--persistence` backend, report custom checks to the primary and export OTel to the collector | +| `--azure-blob-body-storage` | *(none — uses file-system body storage)* | `""` | Store message bodies in Azure Blob Storage instead of the container file system. Applies only with `--persistence:SqlServer` or `--persistence:PostgreSql`. Quote the value because connection strings contain semicolons. The error instance and its scale-out units share the `error-bodies` container, and each audit instance gets its own `audit-bodies-` container | +| `--s3-body-storage` | *(none — uses file-system body storage)* | `` | Store message bodies in an S3 bucket instead of the container file system. Applies only with `--persistence:SqlServer` or `--persistence:PostgreSql`, and cannot be combined with `--azure-blob-body-storage`. Setup creates the bucket if it is missing and the credentials permit it. The error instance and its scale-out units share the `error-bodies/` key prefix, and each audit instance gets its own `audit-bodies-/` prefix | +| `--s3-region` | SDK default | region name, e.g. `eu-west-1` | Region for `--s3-body-storage`. Requires `--s3-body-storage` | +| `--s3-service-url` | AWS | absolute `http` or `https` URL | S3-compatible endpoint, e.g. MinIO or LocalStack. The ServiceControl containers connect to it, so it must be reachable from inside a container. Requires `--s3-body-storage` | +| `--s3-access-key-id`, `--s3-secret-access-key` | *(none — default AWS credential chain)* | key pair | Static credentials for `--s3-body-storage`. Set both or neither. Without them, the AWS default credential chain runs inside each container; the host's credentials are not forwarded. Static keys appear in the container environment | + +Pass static credentials explicitly, or omit both key options when the containers can use the AWS default credential chain: + +```bash +aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj -- \ + --persistence PostgreSql --s3-body-storage my-bodies --s3-region eu-west-1 \ + --s3-access-key-id "$AWS_ACCESS_KEY_ID" --s3-secret-access-key "$AWS_SECRET_ACCESS_KEY" +``` + +For MinIO or LocalStack, use an endpoint reachable from the containers. This example assumes `host.docker.internal` resolves to the host: + +```bash +aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj -- \ + --persistence SqlServer --s3-body-storage bodies --s3-region us-east-1 \ + --s3-service-url http://host.docker.internal:9000 \ + --s3-access-key-id minioadmin --s3-secret-access-key minioadmin +``` + +The static credential options do not support session tokens. For temporary AWS credentials, configure the default credential chain inside the containers instead. The Aspire dashboard provides allocated ports for each service. The testing tool automatically connects to ServiceControl via the platform's transport and REST API URL, and sends its OTLP diff --git a/tools/testing-tool/TestingTool.AppHost/AppHost.cs b/tools/testing-tool/TestingTool.AppHost/AppHost.cs index 7c94c661b4..eb2c3c0225 100644 --- a/tools/testing-tool/TestingTool.AppHost/AppHost.cs +++ b/tools/testing-tool/TestingTool.AppHost/AppHost.cs @@ -4,6 +4,12 @@ var options = CliOptions.Parse(args); var persistenceType = options.GetValue("persistence", PersistenceType.RavenDb); var imageTag = options.GetValueOrDefault("tag"); +var azureBlobBodyStorage = options.GetValueOrDefault("azure-blob-body-storage"); +if (azureBlobBodyStorage is not null && string.IsNullOrWhiteSpace(azureBlobBodyStorage)) +{ + throw new ArgumentException("--azure-blob-body-storage requires a non-blank Azure Storage connection string."); +} +var s3BodyStorage = S3BodyStorageOptions.Parse(options, azureBlobBodyStorage is not null); Console.WriteLine($"Using persistence type: {persistenceType}"); var builder = DistributedApplication.CreateBuilder(args); @@ -28,7 +34,7 @@ .WithEnvironment("SERVICECONTROL_ALLOWMESSAGEEDITING", "true") .WithEnvironment("SERVICECONTROL_DISABLEEXTERNALINTEGRATIONSPUBLISHING", "true") .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", observability.Collector.GetEndpoint("otlp-grpc")) - .WithPersistenceType(persistenceType) + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage) .WithRunMode(PlatformRunMode.SetupAndRun); for (int i = 0; i < options.GetValue("error-ingestion-scale-unit", 0); i++) { @@ -46,7 +52,7 @@ .WithEnvironment("SERVICECONTROL_ERRORINGESTIONBATCHTIMEOUT", "00:00:00.100") //.WaitFor(primaryErrorInstance) - .WithPersistenceType(persistenceType) + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage) .WithRunMode(PlatformRunMode.Run); } @@ -70,7 +76,7 @@ .WithAuditQueueName(i == 0 ? "audit" : "audit" + i) .WithRunMode(PlatformRunMode.SetupAndRun) .WithIngestionTuning() - .WithPersistenceType(persistenceType); + .WithPersistenceType(persistenceType, azureBlobBodyStorage, s3BodyStorage); } // --- Testing tool --- diff --git a/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs b/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs index 740db8ce37..973bc47eb8 100644 --- a/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs +++ b/tools/testing-tool/TestingTool.AppHost/HostBuilderExtensions.cs @@ -44,7 +44,7 @@ public static IResourceBuilder AddPostgresPersist } public static IResourceBuilder WithPersistenceType( - this IResourceBuilder error, PersistenceType type) + this IResourceBuilder error, PersistenceType type, string? azureBlobConnectionString = null, S3BodyStorageOptions? s3 = null) { if (type == PersistenceType.RavenDb) { @@ -59,11 +59,21 @@ public static IResourceBuilder WithPersiste _ => throw new ArgumentOutOfRangeException(nameof(type), type, null) }; + // The primary and its scale-out units share one database and body storage location. + error = azureBlobConnectionString is not null + ? error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "AzureBlob") + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_AZURE_CONNECTIONSTRING", azureBlobConnectionString) + : s3 is not null + ? error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "S3") + .WithS3Environment("SERVICECONTROL_MESSAGEBODY_S3_", s3) + : error + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem") + .WithEnvironment("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlBodyStorage"); + return error .WaitFor(db) - //file storage for now - .WithEnvironment("SERVICECONTROL_MESSAGEBODY_STORAGETYPE", "FileSystem") - .WithEnvironment("SERVICECONTROL_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlBodyStorage") .WithEnvironment("SERVICECONTROL_PERSISTENCETYPE", PersistenceTypeName(type)) .WithEnvironment("SERVICECONTROL_DATABASE_CONNECTIONSTRING", db); } @@ -71,9 +81,7 @@ public static IResourceBuilder WithPersiste /// - /// Points a ServiceControl.Audit instance at the same SQL Server or PostgreSQL container the - /// primary uses. The keys are namespaced to the audit instance, and there is no body storage: - /// the audit EF persisters store bodies inline in the audit row, capped by MaxBodySizeToStore. + /// Points an Audit instance at the primary's database container with the configured body storage. /// /// /// The setting root namespace is "ServiceControl.Audit", and the settings reader rewrites both @@ -83,7 +91,7 @@ public static IResourceBuilder WithPersiste /// so the namespaced form wins. /// public static IResourceBuilder WithPersistenceType( - this IResourceBuilder audit, PersistenceType type) + this IResourceBuilder audit, PersistenceType type, string? azureBlobConnectionString = null, S3BodyStorageOptions? s3 = null) { if (type == PersistenceType.RavenDb) { @@ -97,6 +105,21 @@ public static IResourceBuilder WithPersiste _ => throw new ArgumentOutOfRangeException(nameof(type), type, null) }; + // Audit instances need separate body storage locations; see docs/message-body-storage.md. + audit = azureBlobConnectionString is not null + ? audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "AzureBlob") + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_AZURE_CONNECTIONSTRING", azureBlobConnectionString) + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_AZURE_CONTAINERNAME", "audit-bodies-" + audit.Resource.Name) + : s3 is not null + ? audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "S3") + .WithS3Environment("SERVICECONTROL_AUDIT_MESSAGEBODY_S3_", s3) + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_S3_KEYPREFIX", "audit-bodies-" + audit.Resource.Name + "/") + : audit + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_STORAGETYPE", "FileSystem") + .WithEnvironment("SERVICECONTROL_AUDIT_MESSAGEBODY_FILESYSTEM_STORAGEPATH", "/tmp/ServiceControlAuditBodyStorage"); + return audit .WaitFor(db) .WithEnvironment("SERVICECONTROL_AUDIT_PERSISTENCETYPE", PersistenceTypeName(type)) @@ -143,6 +166,30 @@ public static IResourceBuilder AddTransport( } + static IResourceBuilder WithS3Environment(this IResourceBuilder builder, string prefix, S3BodyStorageOptions s3) + where T : IResourceWithEnvironment + { + builder = builder.WithEnvironment(prefix + "BUCKETNAME", s3.Bucket); + if (s3.Region is not null) + { + builder = builder.WithEnvironment(prefix + "REGION", s3.Region); + } + + if (s3.ServiceUrl is not null) + { + builder = builder.WithEnvironment(prefix + "SERVICEURL", s3.ServiceUrl); + } + + if (s3.AccessKeyId is not null && s3.SecretAccessKey is not null) + { + builder = builder + .WithEnvironment(prefix + "ACCESSKEYID", s3.AccessKeyId) + .WithEnvironment(prefix + "SECRETACCESSKEY", s3.SecretAccessKey); + } + + return builder; + } + static string PersistenceTypeName(PersistenceType persistence) => persistence switch { PersistenceType.SqlServer => "SQLServer", diff --git a/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs b/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs new file mode 100644 index 0000000000..5a0a1ec48a --- /dev/null +++ b/tools/testing-tool/TestingTool.AppHost/S3BodyStorageOptions.cs @@ -0,0 +1,54 @@ +namespace TestingTool.AppHost; + +/// +/// S3 body storage settings from the --s3-* command-line options. Without keys, the container uses the AWS default credential chain. +/// +public record S3BodyStorageOptions(string Bucket, string? Region, string? ServiceUrl, string? AccessKeyId, string? SecretAccessKey) +{ + /// null when --s3-body-storage is not set. + public static S3BodyStorageOptions? Parse(CliOptions options, bool azureBlobSelected) + { + var bucket = Read(options, "s3-body-storage"); + var region = Read(options, "s3-region"); + var serviceUrl = Read(options, "s3-service-url"); + var accessKeyId = Read(options, "s3-access-key-id"); + var secretAccessKey = Read(options, "s3-secret-access-key"); + + if (bucket is null) + { + if (region is not null || serviceUrl is not null || accessKeyId is not null || secretAccessKey is not null) + { + throw new ArgumentException("--s3-region, --s3-service-url, --s3-access-key-id, and --s3-secret-access-key require --s3-body-storage."); + } + + return null; + } + + if (azureBlobSelected) + { + throw new ArgumentException("--azure-blob-body-storage and --s3-body-storage are mutually exclusive."); + } + + if ((accessKeyId is null) != (secretAccessKey is null)) + { + throw new ArgumentException("--s3-access-key-id and --s3-secret-access-key must be set together."); + } + + if (serviceUrl is not null && !(Uri.TryCreate(serviceUrl, UriKind.Absolute, out var uri) && (uri.Scheme == Uri.UriSchemeHttp || uri.Scheme == Uri.UriSchemeHttps))) + { + throw new ArgumentException("--s3-service-url must be an absolute http or https URL."); + } + + return new S3BodyStorageOptions(bucket, region, serviceUrl, accessKeyId, secretAccessKey); + } + + static string? Read(CliOptions options, string key) + { + if (!options.TryGetValue(key, out var value)) + { + return null; + } + + return string.IsNullOrWhiteSpace(value) ? throw new ArgumentException($"--{key} requires a non-blank value.") : value; + } +} diff --git a/tools/testing-tool/TestingTool.Scenarios/IScenario.cs b/tools/testing-tool/TestingTool.Scenarios/IScenario.cs index eb314ac683..3edf42c67c 100644 --- a/tools/testing-tool/TestingTool.Scenarios/IScenario.cs +++ b/tools/testing-tool/TestingTool.Scenarios/IScenario.cs @@ -21,6 +21,9 @@ public interface IScenario /// Default recommended emission rate in messages/second. double DefaultRate { get; } + /// Fixed message body size in bytes, or null for the default 3–6 KB range. + int? BodySizeBytes { get; } + /// Activity source used to emit OpenTelemetry traces for this scenario's work. ActivitySource ActivitySource { get; } diff --git a/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs b/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs new file mode 100644 index 0000000000..3e42797b5c --- /dev/null +++ b/tools/testing-tool/TestingTool.Scenarios/LargeMessagesScenario.cs @@ -0,0 +1,28 @@ +using System.Runtime.CompilerServices; + +namespace TestingTool.Scenarios; + +/// +/// Large message scenario: every body is about 256 KiB, above the default 100 KB external body storage +/// threshold. A deterministic subset of messages (by hash) fails, so both the Audit and Error ingestion +/// paths store large bodies. +/// +public sealed class LargeMessagesScenario(string shardId) : ScenarioBase(shardId) +{ + public override string Name => "large-messages"; + public override string Description => "Publishes about 256 KiB message bodies and fails 20% of them. Exercises external body storage on the Audit and Error paths."; + public override string Category => "Body"; + public override double DefaultRate => 3; + public override int? BodySizeBytes => 256 * 1024; + + private const double FailureRatio = 0.20; + + public override bool ShouldFail(string messageId) => Hash(messageId) < FailureRatio; + + public override Exception CreateException() => + Capture(ProcessLargePayload, "large:payload-too-large"); + + [MethodImpl(MethodImplOptions.NoInlining)] + private static void ProcessLargePayload() => + throw new InvalidDataException("The large message payload could not be processed."); +} diff --git a/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs b/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs index 8a9576bcee..f38bc38767 100644 --- a/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs +++ b/tools/testing-tool/TestingTool.Scenarios/MessageTextGenerator.cs @@ -62,10 +62,16 @@ public static class MessageTextGenerator /// mixed with filler prose. The result is a JSON object with /// several string fields so it resembles a realistic integration message body, and the /// embedded terms are guaranteed to be present in . + /// When is set the body grows to at least that many UTF-8 bytes instead. /// - public static string GenerateBody(long sequence) + public static string GenerateBody(long sequence, int? targetBytes = null) { - var targetSize = Random.Shared.Next(MinSize, MaxSize + 1); + if (targetBytes is { } requested) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(requested, nameof(targetBytes)); + } + + var targetSize = targetBytes ?? Random.Shared.Next(MinSize, MaxSize + 1); // Choose a random subset of searchable terms to embed in this body. Embedding several // per message means every term is covered frequently across a run, so searches drawn diff --git a/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs b/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs index 28bd70b2f9..3b05566ce5 100644 --- a/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs +++ b/tools/testing-tool/TestingTool.Scenarios/ScenarioBase.cs @@ -20,6 +20,7 @@ protected ScenarioBase(string shardId) public abstract string Description { get; } public abstract string Category { get; } public virtual double DefaultRate => 10; + public virtual int? BodySizeBytes => null; public ActivitySource ActivitySource { get; } public virtual TimeSpan? Cooldown => null; diff --git a/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs b/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs index 9a15884619..ea051f8120 100644 --- a/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs +++ b/tools/testing-tool/TestingTool/DirectErrorQueueWriter.cs @@ -197,9 +197,9 @@ private async Task WriteLoop(BypassRun run, int workerIndex, CancellationToken c { while (!ct.IsCancellationRequested && run.TryClaim(out var seq)) { - // Plausible JSON body (~3–6 KB) seeded with searchable terms so the bypass path + // Plausible JSON body (scenario-sized, default ~3–6 KB) seeded with searchable terms so the bypass path // also feeds ServiceControl's full-text search index with real content. - var textBody = MessageTextGenerator.GenerateBody(seq); + var textBody = MessageTextGenerator.GenerateBody(seq, scenario.BodySizeBytes); var message = new LoadMessage { Sequence = seq, TextBody = textBody }; diff --git a/tools/testing-tool/TestingTool/Program.cs b/tools/testing-tool/TestingTool/Program.cs index 1f93e62640..6ea727445d 100644 --- a/tools/testing-tool/TestingTool/Program.cs +++ b/tools/testing-tool/TestingTool/Program.cs @@ -31,6 +31,7 @@ builder.Services.AddSingleton(_ => new PoisonMessageScenario(shardId)); builder.Services.AddSingleton(_ => new DeserializationScenario(shardId)); builder.Services.AddSingleton(_ => new RandomBackgroundNoiseScenario(shardId)); +builder.Services.AddSingleton(_ => new LargeMessagesScenario(shardId)); builder.Services.AddSingleton(); builder.Services.AddSingleton(); diff --git a/tools/testing-tool/TestingTool/ScenarioRunner.cs b/tools/testing-tool/TestingTool/ScenarioRunner.cs index 76b6d62a34..a7879903bc 100644 --- a/tools/testing-tool/TestingTool/ScenarioRunner.cs +++ b/tools/testing-tool/TestingTool/ScenarioRunner.cs @@ -174,9 +174,8 @@ private async Task SendOneAsync(ScenarioRuntime runtime, long seq, CancellationT { var scenario = runtime.Scenario; - // Plausible JSON body (~3–6 KB) seeded with searchable terms so ServiceControl's - // full-text search index has real content to exercise. - var textBody = MessageTextGenerator.GenerateBody(seq); + // Searchable terms ensure full-text queries have indexed content to match. + var textBody = MessageTextGenerator.GenerateBody(seq, scenario.BodySizeBytes); // Deterministic message id shared with the handler: FailingMessageHandler calls // scenario.ShouldFail(context.MessageId), so setting the id here ensures the