diff --git a/CHANGELOG.md b/CHANGELOG.md index fa9bf084c4..f0b89377aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** Known `[integrations.*]` tables, `[tester_cookie]`, and `[tinybird]` now reject unknown fields, including in disabled integration blocks. Integration parse errors name the offending key. `ts config validate` and `ts config push` also reject unknown integration IDs and list the supported IDs; runtime startup warns and ignores unknown IDs for compatibility with stored configs. Before deploying the new binary, use the new CLI to run `ts config validate` on the current config, correct the reported fields and IDs, then push the cleaned config while the old binary is still running. On Fastly, wait for the config-store propagation window to elapse before activating the new binary; a successful push does not mean every point of presence (POP) has the new config. Stored unknown fields can otherwise prevent startup. Intentionally open payload maps remain supported. Lockr uses `app_id`; remove unsupported `organization_id` and `project_id` fields. + - **Breaking:** `x-ts-version` now reports the deployed git version — the release tag, else the branch, else the first 6 characters of the commit — compiled in from the build-time `TRUSTED_SERVER__GIT_VERSION` (set by the deploy pipeline) or local git, and is sent by every adapter, including on the Fastly `GET /health` probe. The Fastly service version it previously carried is now `x-ts-fastly-version`; update dashboards, monitors, and scripts that read `x-ts-version` as the Fastly version number. Builds with neither the override nor git omit the header. - **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave. diff --git a/crates/trusted-server-cli/tests/config_env_overlay.rs b/crates/trusted-server-cli/tests/config_env_overlay.rs index f2f8478ec5..8fe60645da 100644 --- a/crates/trusted-server-cli/tests/config_env_overlay.rs +++ b/crates/trusted-server-cli/tests/config_env_overlay.rs @@ -468,3 +468,55 @@ fn pushed_envelope_sha_is_stable_for_one_context_key_allowlist() { ); } } + +#[test] +fn config_validate_and_push_reject_unknown_integration_ids_and_fields() { + let cases = [ + ("prebdi", "enabled", true, "prebdi"), + ("gpt", "rewrite_scripts", false, "rewrite_scripts"), + ("gpt", "enable", false, "enable"), + ("osano", "made_up_key", false, "made_up_key"), + ]; + for (id, field, field_value, expected) in cases { + for command in ["validate", "push"] { + let project = migrated_project(); + let mut document = fs::read_to_string(&project.config_path) + .expect("should read valid config") + .parse::() + .expect("should parse valid config"); + document["integrations"][id] = toml_edit::table(); + document["integrations"][id][field] = value(field_value); + fs::write(&project.config_path, document.to_string()) + .expect("should write invalid integration config"); + let mut process = Command::new(env!("CARGO_BIN_EXE_ts")); + process + .args(["config", command, "--manifest"]) + .arg(&project.manifest_path) + .arg("--app-config") + .arg(&project.config_path) + .current_dir(project.directory.path()); + if command == "push" { + process.args(["--adapter", "axum", "--yes", "--no-diff"]); + } + + let output = process.output().expect("should run config command"); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + !output.status.success(), + "should reject {id}.{field} during {command}: {stderr}" + ); + assert!( + stderr.contains(expected), + "should name {expected} during {command}: {stderr}" + ); + assert!( + !project + .directory + .path() + .join(".edgezero/local-config-trusted_server_config.json") + .exists(), + "should reject invalid configuration before writing a local blob" + ); + } + } +} diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index f8a439c6ee..0446065510 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -35,7 +35,6 @@ use crate::integrations::{ use crate::settings::{AssetOriginAuth, IntegrationConfig, Settings}; const DEPLOY_VALIDATION_FIELD: &str = "trusted_server"; -#[cfg(test)] const DEPLOY_VALIDATED_INTEGRATION_IDS: &[&str] = &[ #[cfg(feature = "test-utils")] crate::test_support::async_html_finalization::ID, @@ -249,6 +248,16 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { /// Returns [`TrustedServerError`] when non-secret configuration or a secret key /// reference is invalid. pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report> { + let unknown_ids = unknown_integration_ids(settings); + if !unknown_ids.is_empty() { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "Unknown integration IDs: {}. Known integration IDs: {}", + unknown_ids.join(", "), + DEPLOY_VALIDATED_INTEGRATION_IDS.join(", ") + ), + })); + } validate_secret_key_references(settings)?; validate_non_secret_deploy_placeholders(settings)?; validate_js_asset_proxy_config(settings)?; @@ -272,6 +281,11 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report Result<(), Report> { + for id in unknown_integration_ids(settings) { + log::warn!( + "Unknown integration ID '{id}' is ignored; check the integration name or remove its configuration" + ); + } settings.reject_placeholder_secrets()?; validate_js_asset_proxy_config(settings)?; settings.validate_admin_handler_passwords()?; @@ -281,6 +295,17 @@ pub fn validate_settings_for_runtime( Ok(()) } +fn unknown_integration_ids(settings: &Settings) -> Vec<&str> { + let mut ids = settings + .integrations + .keys() + .map(String::as_str) + .filter(|id| !DEPLOY_VALIDATED_INTEGRATION_IDS.contains(id)) + .collect::>(); + ids.sort_unstable(); + ids +} + fn validate_js_asset_proxy_config(settings: &Settings) -> Result<(), Report> { let Some(raw_config) = settings.integrations.get(JS_ASSET_PROXY_INTEGRATION_ID) else { return Ok(()); @@ -1504,6 +1529,176 @@ password = "production-admin-password-32-bytes" .expect("test config with typed Prebid bundle build table should validate"); } + #[test] + fn deploy_validation_rejects_unknown_integration_id() { + let mut settings = valid_settings(); + settings + .integrations + .insert("prebdi".to_owned(), serde_json::json!({ "enabled": false })); + + let error = validate_settings_for_deploy(&settings) + .expect_err("should reject an unknown integration ID even when disabled"); + let message = error.to_string(); + assert!( + message.contains("prebdi"), + "should name the unknown ID: {message}" + ); + for id in DEPLOY_VALIDATED_INTEGRATION_IDS { + assert!(message.contains(id), "should list known ID {id}: {message}"); + } + } + + #[test] + fn runtime_validation_warns_about_unknown_integration_id() { + let mut settings = valid_settings(); + settings + .integrations + .insert("prebdi".to_owned(), serde_json::json!({ "enabled": true })); + let capture = crate::test_support::logging::WarningCapture::start(); + + validate_settings_for_runtime(&settings) + .expect("should keep starting with an unknown stored integration ID"); + + let warnings = capture.take(); + assert!( + warnings + .iter() + .any(|warning| warning.message.contains("prebdi")), + "should warn about the ignored unknown ID" + ); + } + + #[test] + fn deploy_validation_rejects_unknown_integration_fields() { + let cases = [ + ( + "adserver_mock", + serde_json::json!({ "endpoint": "https://mediator.example.com/mediate" }), + ), + ("aps", serde_json::json!({})), + ("datadome", serde_json::json!({})), + ("didomi", serde_json::json!({})), + ( + "google_tag_manager", + serde_json::json!({ "container_id": "GTM-ABC1234" }), + ), + ("gpt", serde_json::json!({})), + ("gpt_diagnostics", serde_json::json!({})), + ( + "js_asset_proxy", + serde_json::json!({ "assets": [{ "path": "/assets/vendor.js", "origin_url": "https://cdn.example.com/vendor.js" }] }), + ), + ("lockr", serde_json::json!({ "app_id": "example-app" })), + ("nextjs", serde_json::json!({})), + ("osano", serde_json::json!({})), + ( + "permutive", + serde_json::json!({ "organization_id": "example", "workspace_id": "example" }), + ), + ( + "prebid", + serde_json::json!({ "external_bundle_url": "https://cdn.example.com/prebid.js" }), + ), + ("sourcepoint", serde_json::json!({})), + ( + "testlight", + serde_json::json!({ "endpoint": "https://testlight.example.com/auction" }), + ), + ]; + let mut accepted = Vec::new(); + for (id, base) in cases { + for enabled in [true, false] { + let mut config = base.clone(); + config["enabled"] = serde_json::json!(enabled); + let mut settings = valid_settings(); + settings.integrations.insert(id.to_owned(), config.clone()); + validate_settings_for_deploy(&settings) + .unwrap_or_else(|error| panic!("should accept baseline {id}: {error:?}")); + + config["made_up_key"] = serde_json::json!("x"); + settings.integrations.insert(id.to_owned(), config); + match validate_settings_for_deploy(&settings) { + Ok(()) => accepted.push(format!("{id} (enabled = {enabled})")), + Err(error) => { + let message = error.to_string(); + assert!( + message.contains("made_up_key"), + "should name unknown field for {id}: {message}" + ); + assert!( + message.contains(id), + "should name integration {id}: {message}" + ); + } + } + } + } + assert!( + accepted.is_empty(), + "should reject unknown fields in all integrations: {accepted:?}" + ); + } + + #[test] + fn deploy_validation_rejects_unknown_fields_in_minimal_disabled_integrations() { + for id in DEPLOY_VALIDATED_INTEGRATION_IDS { + let mut settings = valid_settings(); + settings + .integrations + .insert((*id).to_owned(), serde_json::json!({ "enabled": false })); + validate_settings_for_deploy(&settings) + .unwrap_or_else(|error| panic!("should accept minimal disabled {id}: {error:?}")); + + settings.integrations.insert( + (*id).to_owned(), + serde_json::json!({ "enabled": false, "made_up_key": true }), + ); + let error = validate_settings_for_deploy(&settings) + .expect_err("should reject unknown fields even with missing enabled-only fields"); + assert!( + error.to_string().contains("made_up_key"), + "should name unknown field for {id}: {error:?}" + ); + } + } + + #[test] + fn deploy_validation_preserves_open_integration_payload_maps_and_aliases() { + let mut settings = valid_settings(); + settings.integrations.insert( + "adserver_mock".to_owned(), + serde_json::json!({ + "enabled": true, + "endpoint": "https://mediator.example.com/mediate", + "context_query_params": { "example_context": "example_query" } + }), + ); + settings.integrations.insert( + "datadome".to_owned(), + serde_json::json!({ + "enabled": true, + "client_side_configuration": { "example_option": { "custom_key": true } } + }), + ); + settings.integrations.insert( + "prebid".to_owned(), + serde_json::json!({ + "enabled": true, + "external_bundle_url": "https://cdn.example.com/prebid.js", + "managed_user_ids": [{ "name": "sharedId", "params": { "example_option": true } }] + }), + ); + settings.integrations.insert( + "lockr".to_owned(), + serde_json::json!({ + "enabled": true, "app_id": "example-app", "rewrite_sdk_host": false + }), + ); + + validate_settings_for_deploy(&settings) + .expect("should accept open payload maps and explicit compatibility fields"); + } + #[test] fn deploy_validation_covers_registered_integration_builders() { let validated_ids: HashSet<&'static str> = diff --git a/crates/trusted-server-core/src/integrations/adserver_mock.rs b/crates/trusted-server-core/src/integrations/adserver_mock.rs index 640574e031..d64597fb7c 100644 --- a/crates/trusted-server-core/src/integrations/adserver_mock.rs +++ b/crates/trusted-server-core/src/integrations/adserver_mock.rs @@ -38,6 +38,7 @@ use crate::settings::{IntegrationConfig, Settings}; /// Configuration for mock ad server integration. #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct AdServerMockConfig { /// Whether this integration is enabled #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/didomi.rs b/crates/trusted-server-core/src/integrations/didomi.rs index a92c643aa3..7e9c65e74a 100644 --- a/crates/trusted-server-core/src/integrations/didomi.rs +++ b/crates/trusted-server-core/src/integrations/didomi.rs @@ -23,6 +23,7 @@ const DIDOMI_DEFAULT_PREFIX: &str = "/integrations/didomi/consent"; /// Configuration for the Didomi consent notice reverse proxy. #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct DidomiIntegrationConfig { /// Whether the integration is enabled. #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/google_tag_manager.rs b/crates/trusted-server-core/src/integrations/google_tag_manager.rs index 74a21a64fe..41850b2953 100644 --- a/crates/trusted-server-core/src/integrations/google_tag_manager.rs +++ b/crates/trusted-server-core/src/integrations/google_tag_manager.rs @@ -238,6 +238,7 @@ pub(crate) fn rewrite_gtm_rsc_span(content: &str, record_text: bool) -> String { } #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct GoogleTagManagerConfig { #[serde(default = "default_enabled")] pub enabled: bool, diff --git a/crates/trusted-server-core/src/integrations/gpt.rs b/crates/trusted-server-core/src/integrations/gpt.rs index 3292cc5b3a..919c44ae96 100644 --- a/crates/trusted-server-core/src/integrations/gpt.rs +++ b/crates/trusted-server-core/src/integrations/gpt.rs @@ -63,6 +63,7 @@ const ROUTE_PREFIX: &str = "/integrations/gpt"; /// Configuration for the Google Publisher Tags integration. #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct GptConfig { /// Enable/disable the integration. #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs index 6b301bf935..3deb8f36f6 100644 --- a/crates/trusted-server-core/src/integrations/js_asset_proxy.rs +++ b/crates/trusted-server-core/src/integrations/js_asset_proxy.rs @@ -37,6 +37,7 @@ const ERROR_ORIGIN_STATUS: &str = "js-asset-origin-status"; /// Configuration for the JavaScript asset proxy integration. #[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] pub struct JsAssetProxyConfig { /// Enables or disables the integration. #[serde(default)] diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index 002182bd13..aa42f69370 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -32,6 +32,7 @@ const LOCKR_INTEGRATION_ID: &str = "lockr"; /// Configuration for Lockr integration. #[derive(Debug, Deserialize, Validate)] +#[serde(deny_unknown_fields)] pub struct LockrConfig { /// Enable/disable the integration #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/nextjs/mod.rs b/crates/trusted-server-core/src/integrations/nextjs/mod.rs index ea5fb08ad4..afd0ad399d 100644 --- a/crates/trusted-server-core/src/integrations/nextjs/mod.rs +++ b/crates/trusted-server-core/src/integrations/nextjs/mod.rs @@ -24,6 +24,7 @@ use rsc_stream::NextJsRscStreamProcessorFactory; use script_rewriter::NextJsNextDataRewriter; #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct NextJsIntegrationConfig { #[serde(default = "default_enabled")] pub enabled: bool, diff --git a/crates/trusted-server-core/src/integrations/permutive.rs b/crates/trusted-server-core/src/integrations/permutive.rs index b59f4e3820..0e2b0bbc59 100644 --- a/crates/trusted-server-core/src/integrations/permutive.rs +++ b/crates/trusted-server-core/src/integrations/permutive.rs @@ -28,6 +28,7 @@ const PERMUTIVE_INTEGRATION_ID: &str = "permutive"; /// Configuration for Permutive integration. #[derive(Debug, Deserialize, Validate)] +#[serde(deny_unknown_fields)] pub struct PermutiveConfig { /// Enable/disable the integration #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/sourcepoint.rs b/crates/trusted-server-core/src/integrations/sourcepoint.rs index aa88ac08a9..d9b18a6f3d 100644 --- a/crates/trusted-server-core/src/integrations/sourcepoint.rs +++ b/crates/trusted-server-core/src/integrations/sourcepoint.rs @@ -171,6 +171,7 @@ static SP_MESSAGE_ORIGIN_GUARD_PATTERN: LazyLock = LazyLock::new(|| { /// Configuration for the Sourcepoint first-party proxy. #[derive(Debug, Clone, Deserialize, Validate)] +#[serde(deny_unknown_fields)] pub struct SourcepointConfig { /// Whether the integration is enabled. #[serde(default = "default_enabled")] diff --git a/crates/trusted-server-core/src/integrations/testlight.rs b/crates/trusted-server-core/src/integrations/testlight.rs index 9e68492bd3..b2b7f7f6c2 100644 --- a/crates/trusted-server-core/src/integrations/testlight.rs +++ b/crates/trusted-server-core/src/integrations/testlight.rs @@ -24,6 +24,7 @@ use crate::tsjs; const TESTLIGHT_INTEGRATION_ID: &str = "testlight"; #[derive(Debug, Deserialize, Validate)] +#[serde(deny_unknown_fields)] pub struct TestlightConfig { #[serde(default = "default_enabled")] pub enabled: bool, diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index c93585d367..8cee198387 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -298,6 +298,19 @@ pub trait IntegrationConfig: DeserializeOwned + Validate { } impl IntegrationSettings { + fn parse_error(integration_id: &str, error: serde_json::Error) -> Report { + let detail = error.to_string(); + let mut message = + format!("Integration '{integration_id}' configuration could not be parsed"); + // Field diagnostics identify schema keys. Other serde errors can echo + // configuration values, so retain them only in the report's cause chain. + if detail.starts_with("unknown field ") || detail.starts_with("missing field ") { + message.push_str(": "); + message.push_str(&detail); + } + Report::new(error).change_context(TrustedServerError::Configuration { message }) + } + /// Inserts a configuration value for an integration. /// /// # Errors @@ -368,21 +381,13 @@ impl IntegrationSettings { }; if Self::is_explicitly_disabled(raw) { - T::validate_disabled_schema(raw).change_context(TrustedServerError::Configuration { - message: format!( - "Integration '{integration_id}' configuration could not be parsed" - ), - })?; + T::validate_disabled_schema(raw) + .map_err(|error| Self::parse_error(integration_id, error))?; return Ok(None); } - let config: T = serde_json::from_value(raw.clone()).change_context( - TrustedServerError::Configuration { - message: format!( - "Integration '{integration_id}' configuration could not be parsed" - ), - }, - )?; + let config: T = serde_json::from_value(raw.clone()) + .map_err(|error| Self::parse_error(integration_id, error))?; // Field validation runs only for integrations that resolve to enabled. // An integration whose `enabled` flag is omitted falls back to its @@ -2046,6 +2051,7 @@ impl Proxy { /// Direct Tinybird Events API telemetry configuration. #[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] pub struct TinybirdSettings { /// Master enablement for auction telemetry ingestion. #[serde(default)] @@ -2952,6 +2958,7 @@ pub enum AuctionDebugCommentFormat { /// Tester-cookie endpoint configuration. #[derive(Debug, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] pub struct TesterCookieConfig { /// Enable tester-cookie endpoints that set and clear `ts-tester`. #[serde(default)] @@ -4072,7 +4079,7 @@ mod tests { use super::*; use crate::auction::build_orchestrator; use crate::integrations::{ - IntegrationRegistry, gpt::GptConfig, nextjs::NextJsIntegrationConfig, + IntegrationRegistry, gpt::GptConfig, lockr::LockrConfig, nextjs::NextJsIntegrationConfig, prebid::PrebidIntegrationConfig, }; use crate::redacted::Redacted; @@ -5932,6 +5939,63 @@ origin_host_header_overide = "www.example.com""#, } } + #[test] + fn integration_parse_errors_hide_invalid_values() { + for enabled in [true, false] { + let mut integrations = IntegrationSettings::default(); + integrations.insert( + "gpt".to_string(), + json!({ + "enabled": enabled, + "gam_attribution_enabled": "fictional-private-value", + }), + ); + + let error = integrations + .get_typed::("gpt") + .expect_err("should reject an invalid boolean for enabled and disabled configs"); + let message = error.to_string(); + assert!( + message.contains("Integration 'gpt' configuration could not be parsed"), + "should retain integration context" + ); + assert!( + !message.contains("fictional-private-value"), + "should not echo invalid values in the outer error message" + ); + } + } + + #[test] + fn integration_parse_errors_name_missing_fields() { + let mut integrations = IntegrationSettings::default(); + integrations.insert("lockr".to_string(), json!({ "enabled": true })); + + let error = integrations + .get_typed::("lockr") + .expect_err("should reject missing required integration fields"); + assert!( + error.to_string().contains("missing field"), + "should retain missing-field diagnostics" + ); + } + + #[test] + fn settings_reject_unknown_tester_cookie_and_tinybird_fields() { + for section in ["tester_cookie", "tinybird"] { + let source = format!( + "{}\n[{section}]\nmade_up_key = true\n", + crate_test_settings_str() + ); + let error = Settings::from_toml(&source) + .expect_err("should reject unknown keys in strict settings sections"); + assert!( + format!("{error:?}").contains("made_up_key"), + "should name offending {section} key: {error:?}" + ); + } + } + #[test] fn prepare_runtime_rejects_invalid_handler_regex() { let toml_str = crate_test_settings_str().replace(r#"path = "^/secure""#, r#"path = "(""#); diff --git a/crates/trusted-server-core/src/test_support/async_html_finalization.rs b/crates/trusted-server-core/src/test_support/async_html_finalization.rs index 9e488c4eff..82cc898ebc 100644 --- a/crates/trusted-server-core/src/test_support/async_html_finalization.rs +++ b/crates/trusted-server-core/src/test_support/async_html_finalization.rs @@ -29,6 +29,7 @@ const ORIGINAL: &[u8] = b"EXAMPLE_UNSIGNED"; const REPLACEMENT: &[u8] = b"EXAMPLE_FINALIZED"; #[derive(Clone, Deserialize, Validate)] +#[serde(deny_unknown_fields)] pub(crate) struct FixtureConfig { enabled: bool, #[validate(url)] diff --git a/crates/trusted-server-integration-tests/tests/html_eof_runtime.rs b/crates/trusted-server-integration-tests/tests/html_eof_runtime.rs index 2dd89e0683..1ade6af84f 100644 --- a/crates/trusted-server-integration-tests/tests/html_eof_runtime.rs +++ b/crates/trusted-server-integration-tests/tests/html_eof_runtime.rs @@ -173,6 +173,8 @@ impl<'a> Harness<'a> { "build", "-p", "trusted-server-cli", + "--features", + "trusted-server-core/test-utils", "--bin", "ts", "--locked", diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index d325b6a768..260c91bfa8 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -178,9 +178,27 @@ openssl rand -base64 32 ### Strict Key Validation -Trusted Server rejects unknown TOML keys in runtime configuration. Before pushing -or upgrading config, remove stale fields and typos; otherwise config loading can -fail and the service will return its startup-error response. +`ts config validate` and `ts config push` reject unknown integration IDs and list +the supported IDs. Unknown fields in known integration tables are rejected, +including when `enabled = false`. The error names the integration and offending +field. `[tester_cookie]` and `[tinybird]` also reject unknown fields. Payload maps +such as Prebid managed-user-ID `params` and Adserver Mock `context_query_params` +remain open for application-defined keys. + +Runtime startup rejects unknown fields in known tables. For compatibility with +previously stored configurations, an unknown integration ID only logs a warning +and its table is ignored at runtime. + +This warning is emitted each time runtime settings are loaded. On Fastly without +reusable sandboxes, that means every request; remove unknown IDs before the next +config push to avoid repeated warnings. + +Before upgrading, use the new CLI to run `ts config validate` against the current +config, correct the reported settings, and push the cleaned config while the old +binary is still running. On Fastly, wait for the config-store propagation window +to elapse before activating the new binary; a successful push does not mean every +point of presence (POP) has the new config. Otherwise a stored unknown field can +prevent startup and cause the service to return its startup-error response. ## Configuration Files diff --git a/docs/guide/error-reference.md b/docs/guide/error-reference.md index c2ddcebbd9..07d170861e 100644 --- a/docs/guide/error-reference.md +++ b/docs/guide/error-reference.md @@ -378,7 +378,7 @@ curl https://myorg.edge.permutive.app/workspace-123-web.js **Error Message:** ``` -No route matched for /integrations/custom/endpoint +No route matched for /assets/vendor.js ``` **Cause:** Integration not enabled or route not registered @@ -388,8 +388,12 @@ No route matched for /integrations/custom/endpoint 1. Enable integration: ```toml -[integrations.custom] +[integrations.js_asset_proxy] enabled = true + +[[integrations.js_asset_proxy.assets]] +path = "/assets/vendor.js" +origin_url = "https://cdn.example.com/vendor.js" ``` 2. Verify integration is compiled in (check build logs) diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index ef9e57b0f3..6972759976 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -43,27 +43,32 @@ Add Lockr configuration to `trusted-server.toml`: ```toml [integrations.lockr] enabled = true -api_endpoint = "https://api.lockr.io" -organization_id = "your-org-id" -project_id = "your-project-id" +app_id = "example-app" ``` ### Configuration Options -| Field | Type | Required | Description | -| ----------------- | ------- | -------- | --------------------------------------------- | -| `enabled` | boolean | No | Enable/disable integration (default: `false`) | -| `api_endpoint` | string | Yes | Lockr API endpoint URL | -| `organization_id` | string | Yes | Your Lockr organization ID | -| `project_id` | string | Yes | Your Lockr project ID | +| Field | Type | Required | Description | +| ------------------- | ------- | -------- | -------------------------------------------------------------------- | +| `enabled` | boolean | No | Enable/disable integration (default: `true` when the section exists) | +| `app_id` | string | Yes | Non-empty Lockr application ID | +| `api_endpoint` | string | No | API base URL; defaults to the integration's API host | +| `sdk_url` | string | No | SDK URL; defaults to the integration's trust-server SDK | +| `cache_ttl_seconds` | integer | No | SDK cache TTL, 60–86400 seconds (default: `3600`) | +| `rewrite_sdk` | boolean | No | Rewrite SDK URLs in HTML (default: `true`) | +| `origin_override` | string | No | Override the Origin header sent to the API | +| `rewrite_sdk_host` | boolean | No | Deprecated compatibility field; host routing is handled by the SDK | + +Unknown fields, including the unsupported `organization_id` and `project_id`, +are rejected. Remove them before upgrading and configure `app_id` instead. ### Environment Variables +The CLI applies overrides to matching fields already present in the config: + ```bash TRUSTED_SERVER__INTEGRATIONS__LOCKR__ENABLED=true -TRUSTED_SERVER__INTEGRATIONS__LOCKR__API_ENDPOINT=https://api.lockr.io -TRUSTED_SERVER__INTEGRATIONS__LOCKR__ORGANIZATION_ID=your-org-id -TRUSTED_SERVER__INTEGRATIONS__LOCKR__PROJECT_ID=your-project-id +TRUSTED_SERVER__INTEGRATIONS__LOCKR__APP_ID=example-app ``` ## Features @@ -211,7 +216,7 @@ curl https://edge.example.com/ **Solutions**: -- Verify `organization_id` and `project_id` are correct +- Verify `app_id` is correct - Check Lockr API endpoint is reachable - Ensure GDPR consent is granted - Review Lockr API credentials diff --git a/docs/superpowers/plans/2026-10-27-strict-integration-validation.md b/docs/superpowers/plans/2026-10-27-strict-integration-validation.md new file mode 100644 index 0000000000..00fec40c15 --- /dev/null +++ b/docs/superpowers/plans/2026-10-27-strict-integration-validation.md @@ -0,0 +1,55 @@ +# Strict integration validation implementation plan + +**Goal:** Refuse silently ignored configuration typos while retaining runtime +startup for unknown integration IDs. + +**Architecture:** Use the existing complete ID inventory in deploy/runtime +validation and serde strictness for known table fields. Preserve open payload +maps, disabled-block semantics, and the current configuration schema. + +**Tech stack:** Rust, serde, error-stack, validator, TOML, existing warning capture. + +## 1. Regression evidence + +Files: `crates/trusted-server-core/src/config.rs`, +`crates/trusted-server-core/src/settings.rs`, CLI config tests where appropriate. + +- [x] Add deploy rejection tests for unknown IDs and enabled/disabled unknown fields. +- [x] Assert the ordinary CLI-facing error text names the offending key. +- [x] Add runtime warning capture and successful validation for unknown IDs. +- [x] Add strict tester-cookie/Tinybird regressions and open-map preservation tests. +- [x] Run focused tests against unchanged production code and retain failures. + +Run: `cargo +1.98.1 test -p trusted-server-core --lib --locked -- config::tests` +and focused settings tests. New regressions should fail for the expected cause. + +## 2. Minimal implementation + +Files: `config.rs`, `settings.rs`, and config structs in integration modules +`adserver_mock`, `didomi`, `google_tag_manager`, `gpt`, `js_asset_proxy`, `lockr`, +`nextjs/mod`, `permutive`, `sourcepoint`, and `testlight`. + +- [x] Promote the known-ID inventory; reject unknown map keys during deploy validation. +- [x] Warn for unknown map keys during runtime validation without rejecting startup. +- [x] Add `#[serde(deny_unknown_fields)]` to the twelve identified config structs. +- [x] Include serde errors in enabled and disabled `get_typed` outer contexts. +- [x] Rerun regressions and native core suite; preserve minimal disabled blocks, + aliases, nested schemas, and open parameter maps. + +## 3. Documentation and entry points + +Files: `docs/guide/configuration.md`, `docs/guide/integrations/lockr.md`, +`docs/guide/error-reference.md`, `CHANGELOG.md`. + +- [x] Explain strict known-field validation and runtime unknown-ID warnings. +- [x] Replace unsupported Lockr fields and custom-integration example. +- [x] Document validate, clean up, push, deploy rollout order and startup risk. +- [x] Exercise CLI config validation with fictional invalid configuration fixtures. + +## 4. Verification and review + +- [x] Run formatting and target-matched checks using repository-pinned tools. +- [x] Run applicable CI gates from `AGENTS.md`; record blockers explicitly. +- [x] Request independent local review of the combined changes and resolve findings. +- [x] Inspect final diff and preserve unrelated checkout changes. +- [x] Report tested base/content, passing evidence, blocked gates and retained worktree. diff --git a/docs/superpowers/specs/2026-10-27-strict-integration-validation-design.md b/docs/superpowers/specs/2026-10-27-strict-integration-validation-design.md new file mode 100644 index 0000000000..ba5e49788b --- /dev/null +++ b/docs/superpowers/specs/2026-10-27-strict-integration-validation-design.md @@ -0,0 +1,71 @@ +# Strict integration configuration validation + +## Problem and scope + +Integration settings are stored as an open JSON map. Deploy validation reads only +known IDs, and ten integration schemas silently discard unknown fields. A typo +such as `[integrations.prebdi]` or `enable = false` can therefore pass validation +without producing the intended behavior. + +This change hardens the current `[integrations.*]` schema. It does not introduce +the separate module-selection configuration redesign. + +## Approved behavior + +- Config validate and push reject unknown integration IDs, naming them and + listing the supported IDs from the complete validation inventory. +- Runtime validation warns about unknown IDs and continues, preserving startup + for previously stored unused tables. +- All known integration config structs reject unknown fields, including disabled + blocks. Minimal disabled blocks continue to omit enabled-only required fields. +- `tester_cookie` and `tinybird` reject unknown fields as well. +- Integration parse errors include the offending field in the outer error text + consumed by CLI validation. +- Intentionally open payload maps and explicit compatibility aliases remain open + or accepted. No new dependency is required. + +## Implementation boundaries + +Promote the existing complete integration-ID inventory in `config.rs` to production +use. Validate map keys at deploy validation and warn at runtime validation. Keep +feature-gated fixture IDs aligned with their existing validation paths. Generic +registered builders alone are insufficient because Prebid and APS use separate +registration paths. + +Add serde strictness only to the ten integration config structs that lack it, +and the two additional settings structs. Include unknown/missing-field details in +both branches of `IntegrationSettings::get_typed`; retain other serde error details +only in the report's cause chain so the outer message does not echo invalid values. + +## Compatibility and alternatives + +Known tables with unknown fields will fail runtime startup with the new binary. +Before deploying, run the new CLI's `ts config validate`, correct the reported +settings, and push the cleaned config while the old binary is still running. On +Fastly, wait for the config-store propagation window to elapse before activating +the new binary; a successful push does not mean every point of presence (POP) has +the new config. Removing previously ignored keys is safe for the old binary. +Unknown IDs only warn at runtime. + +A warn-first release for unknown fields would avoid this startup break but needs +additional parsing machinery. The approved choice is the issue's smaller strict +schema fix with documented migration. The future module schema is outside scope. + +## Acceptance and preservation + +| Scenario | Required result | +| ------------------------------------------------ | --------------------------------------------- | +| Unknown integration ID during deploy validation | Error names ID and known IDs | +| Unknown integration ID at runtime | Warning names ID; startup validation succeeds | +| Unknown field in enabled or disabled known block | Error names integration and field | +| Minimal explicitly disabled block | Accepted without enabled-only required fields | +| Open payload parameter maps | Arbitrary payload keys remain accepted | +| Deprecated explicit aliases | Remain accepted | +| Unknown tester-cookie or Tinybird key | Deserialization fails | +| CLI config validate/push | Errors expose offending integration key | + +Regression tests must distinguish the original behavior before implementation. +Run focused native core and CLI checks, adapter tests and target-matched lint, +then the repository's required CI gates. Record unavailable tools or mismatched +versions separately from passing evidence. Correct the configuration guide, +Lockr example, error reference, and changelog upgrade instructions.