Repository navigation
Automated Sync from main to stable - #40
Open
rhods-devops-app[bot] wants to merge 275 commits into
Open
rhods-devops-app[bot] wants to merge 275 commits into
rhods-devops-app[bot] wants to merge 275 commits into
Conversation
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Evaluate issue acceptance using trusted default-branch code and the PR author's repository permissions. Report the native workflow job result, protect enforcement files, and align PR creation and review guidance. Closes NVIDIA#4339 Signed-off-by: Evan Lezar <elezar@nvidia.com>
The VM runtime scripts sourced pins.env optionally and duplicated the LIBKRUN_REF and UMOCI_VERSION defaults, so every bump had to update the same value in several files. Require pins.env, drop the duplicated defaults, and fail on an invalid LIBKRUN_REF instead of silently building upstream HEAD. Environment overrides still take precedence through the defaults in pins.env. Signed-off-by: Florent Benoit <fbenoit@redhat.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
…RKDIR (NVIDIA#3931) * test(oci): share OCI image checks across Docker and Podman Add an oci-image feature testsuite that runs against installed artifacts. It covers custom WORKDIR placement, image content and ownership, workspace writes from the main process and exec, file transfer including directory-merge upload, OCI user identity, the managed /sandbox fallback, and rejection of an unwritable WORKDIR. CI runs the suite on the Docker rootful, Podman rootful, and Podman rootless tmachine guests. Replace the Docker-only custom_image e2e with the shared suite. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): require WorkspaceValidationFailed and cover image USER fallback The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed reason, so unrelated provisioning failures cannot pass it. The numeric USER scenario supplies a policy without a process section, so the image USER is the only source of the sandbox identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(podman): leave shared OCI identity checks to the oci-image suite The oci-image feature suite covers the sandbox identity seen by the main process and sandbox exec on Docker and Podman. The Podman e2e keeps the checks only it can make: the pinned image ID and the isolated workload/supervisor container pair. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): accept existing workspace rejection diagnostics Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): fix cleanup ordering and restore coverage parity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): remove migrated legacy workspace helpers and checks Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed The supervisor exits with the reserved status when the sandbox rejects the image working directory. Docker maps that exit during readiness and monitoring and records it before removing the supervisor container, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): require WorkspaceValidationFailed for unwritable WORKDIR Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(oci): keep exit 78 reserved and report fixed workspace diagnostics The supervisor no longer passes on a workload's own exit status 78, so drivers cannot misreport it as a workspace rejection after a runtime restart. The rejection is recognized once at agent start and carried as a typed error. Docker reports the same fixed condition message as Podman and logs the supervisor output instead, and its monitor removes the supervisor container when a stop overlaps diagnostics collection. The OCI image suite registers images for cleanup only after they build. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure handling and OCI image tests Report a recorded Docker workspace rejection once, where supervisor readiness fails, and match its fixed message exactly. Read the Podman supervisor exit status once per inspection. Choose the supervisor exit status in one function and derive the workspace error type. The OCI image suite shares step helpers, checks that a custom workspace does not also create /sandbox, and its playbook waits for a connected gateway and collects failure diagnostics through the shared role. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): replace podman_oci_identity with shared and driver suite checks The oci-image feature suite now checks, on Docker and Podman, that the image user has no capabilities, has no_new_privs set, and cannot read the sandbox control credentials, and that a custom WORKDIR is not a mount. The checks that need container inspection (pinned image ID, network=none, supervisor user and capabilities) move to a small test in the Podman driver suite. Remove the legacy podman_oci_identity e2e test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): keep podman_oci_identity for Podman-only isolation checks Restore the Podman e2e test removed in the previous commit and drop only the checks the shared oci-image suite now runs on Docker and Podman: workload capabilities, no_new_privs, credential readability, and the workspace and /sandbox mount checks. The pinned image ID and the workload/supervisor isolation checks stay, as does the Podman driver suite unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure reporting and OCI image suite Carry the Docker control supervisor's failure reason in a typed error instead of matching status text, build the fixed workspace message where the wait result is classified, and drop the fake Docker server test. Inline image cleanup ordering in the suite, read the container engine once, share sandbox naming, and stop granting a literal /sandbox in the image identity policy. Add e2e:oci-image mise tasks and use them from the ARM64 Docker workflow and TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): move shared container checks into the OCI image suite Inspect the containers the driver created through the gateway's engine in the shared suite: the workload runs the inspected image ID as the OCI identity without a network or supervisor credential mounts, and the supervisor is a separate non-root container. Keep only the Podman-specific supervisor identity and capability checks in podman_oci_identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): check the workspace rejection reason from sandbox JSON Read the WorkspaceValidationFailed condition reason from sandbox get --output json instead of matching CLI output, and restore the Docker driver's original workspace failure helpers. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): capture fixture image identity before sandbox creation Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com>
…le connection (NVIDIA#4452) * fix(kubernetes): restart kube read timeout when each request is written (fixes NVIDIA#4451) Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(kubernetes): use a per-request tower timeout instead of a custom kube client Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…errors (NVIDIA#4326) Signed-off-by: Adrien Langou <alangou@nvidia.com>
…IDIA#4342) Signed-off-by: Adrien Langou <alangou@nvidia.com>
…VIDIA#4422) * test(podman): run the e2e binaries that pass in the tmachine guest Remove ten binaries from podmanE2eFollowUpBinaries so the rootless tmachine e2e-podman archive runs them: credential_gating, forward_proxy_graphql_l7, forward_proxy_jsonrpc_l7, forward_proxy_l7_bypass, host_gateway_alias, landlock, no_proxy, podman_oci_identity, provider_refresh_handles, and websocket_conformance. All ten passed in two consecutive probe runs. Update the exclusion comments and TESTING.md to match what the remaining entries still need. Contributes to NVIDIA#3712. Signed-off-by: politerealism <burdcat17@gmail.com> * test(podman): explain why each remaining e2e binary is excluded Give every remaining podmanE2eFollowUpBinaries entry its own reason, and mark the five that the wrapper-based Podman lane still runs (driver_config_volume, local_driver_token_restart, podman_corporate_proxy, podman_gateway_start, provider_token_exchange) so it is clear they are covered there until tmachine provides what they need. Contributes to NVIDIA#3712. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com>
…IA#2441) * refactor(prover): remove legacy unused reachability constraints Remove can_write_to_endpoint() and the solver variables and encoding methods that exist solely to support it. No active query calls this method; confirmed zero callers across the monorepo. Crate is not published externally, so no public API concern. Removed as root cause: - can_write_to_endpoint(): zero callers in the codebase Removed as direct consumers of can_write_to_endpoint(): - binary_can_write: only read inside can_write_to_endpoint() - credential_has_write: only read inside can_write_to_endpoint() Removed as directly dead (no query ever read them): - credential_has_destructive: flagged by compiler, silenced with #[allow(dead_code)] rather than removed - filesystem_readable: flagged by compiler, silenced with #[allow(dead_code)] rather than removed Removed as now-empty encoding methods: - encode_credentials(): exclusively populated credential_has_write and credential_has_destructive - encode_filesystem(): exclusively populated filesystem_readable Also removed two #[allow(dead_code)] allowances and the corresponding HashMap::new() initialisers and build() call sites. Active query behavior is unchanged. All existing prover tests pass without modification and cover all four finding categories end-to-end. No new tests added as they would duplicate existing coverage exactly. Note: removing encode_credentials() leaves write_actions_for_scopes() and destructive_actions_for_scopes() in credentials.rs with no callers. Left as a follow-up outside the scope of this change. Closes NVIDIA#2412 Signed-off-by: Evie Howard <evhoward@redhat.com> * refactor(prover): remove dead write-capability helpers from binary registry BinaryProtocol::can_write(), BinaryCapability::can_write(), and write_mechanisms() lost their only callers when can_write_to_endpoint() and its supporting model fields were removed. Delete the now-dead methods. write_mechanisms() had no callers prior to this series; the two can_write() methods were made dead by the preceding model.rs cleanup. Signed-off-by: Evie Howard <evhoward@redhat.com> * docs(prover): remove stale write-bypass mention from module doc comment The module doc incorrectly listed write-bypass violation detection as a prover capability. The prover only checks reachability queries; write-bypass detection was never implemented. Signed-off-by: Evie Howard <evhoward@redhat.com> --------- Signed-off-by: Evie Howard <evhoward@redhat.com>
* fix(cli): backpressure sandbox uploads Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): preserve upload cancellation and errors Keep cancellation active through upload setup and retry delays, and terminate cancelled SSH probes. Preserve protocol failures through output completion and report captured diagnostics on early extraction failure. Remove redundant upload validation and extend the affected regressions. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(cli): carry upload credit over SSH streams Replace the framed upload protocol with SSH's own streams. The client sends one JSON plan line, the raw archive and EOF. The supervisor's stdout carries only cumulative totals of bytes written to extraction stdin, extraction output goes to stderr, and the exit status reports the result. Extraction starts inline through the ordinary exec path. This removes the frame codec, SHA-256 digests, the CLI sha2 dependency and the supervisor upload state machine while keeping the 1 MiB credit window, the 4 MiB input guard, cancellation and error reporting. A failed archive now stops SSH before closing stdin, so EOF never marks a truncated archive complete. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): isolate upload SSH connections and pin credit cadence - Apply the forward multiplexing overrides to the upload ssh command, so a user ControlMaster cannot carry an upload over another session's master connection. - Kill ssh before closing its stdin when a transfer fails locally, and include ssh stderr in transport errors. - Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI chunks fit the credit left after an ack and that the window stays below the supervisor's pending-input guard. - Log the real reason when the supervisor rejects an upload. - Test that a single-packet upload reports its final total. Signed-off-by: Shiju <shiju@nvidia.com> * test(cli): assert upload ssh opens the subsystem without a TTY Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com>
* ci(security): temporarily make release Codex scans advisory Signed-off-by: Adrien Langou <alangou@nvidia.com> * ci(security): remove Codex from release scans Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* docs: document workspace and provider labels (fixes NVIDIA#4250) Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(providers): support CLI labels and selector filtering Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): update schema inventory and scoped test environment Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
…2632) Signed-off-by: Evie Howard <evhoward@redhat.com>
…IDIA#4488) * ci(compatibility): review PR breaks and acknowledge release risks Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(ci): use matching compatibility review tooling Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): scope Rust SDK review to public exports Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): account for Go struct field additions Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): restrict egress and fix report retries Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(ci): skip maintainer approval in merge queue Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * refactor(ci): remove maintainer queue-state lookup Signed-off-by: Jim Meyer <jimeyer@nvidia.com> --------- Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(otel): enable gzip and https OTLP export Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(otel): reject untrusted OTLP collector certificates Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com>
… and labels (NVIDIA#4497) * feat(kubernetes): add operator allowlist for sandbox pod annotations Add allowed_pod_annotations to the Kubernetes driver config. Sandbox template annotations must match an exact key or a <domain>/* entry, and the default empty list rejects all of them. Keys in the openshell.ai and agents.x-k8s.io domains are always rejected. Expose the setting through the server.allowedPodAnnotations Helm value. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): cover pod annotation rejection on sandbox create Assert the exact rejected annotation keys, exercise create_sandbox_inner with a disallowed annotation, and point the rejection error at the Helm gatewayConfig path. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): drop Helm path from pod annotation rejection error The rejection error reaches API callers, not operators, so name only the gateway config key. Spell the Helm path in the runtimes docs with the quoted table key to avoid an invalid nested gatewayConfig rendering. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): condense pod annotation allowlist tests Fold the allowlist matching and validation cases into table-driven tests, reuse the json_struct helper for templates, and document why the openshell.ai and agents.x-k8s.io domains are reserved. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(kubernetes): add operator allowlist for sandbox pod labels Sandbox template labels were copied onto workload pods unfiltered. Labels select pods for NetworkPolicy rules and Services, so a caller could place a sandbox under rules meant for other workloads or set OpenShell-owned labels. Add allowed_pod_labels, sharing the allowed_pod_annotations entry format, validation, and reserved openshell.ai / agents.x-k8s.io domains. Empty by default, which rejects every template label. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(rfc): propose sandbox service exposure architecture Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): link service exposure review Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): shorten service exposure proposal Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): remove service exposure architecture diagrams Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): share application ingress across supervisor tenants Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): describe direct and reverse tunnel proxy connections Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(rfc): propose supervisor service ingress and optional proxy Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): describe proxy as discovery reference implementation Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): emphasize supervisor app listener and ingress extensibility Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(rfc): clarify driver-independent service ingress Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
* test(server): add a local PostgreSQL test runner The openshell-server tests that need a real PostgreSQL server are ignored by default and had no shared way to run. The mutation-replay test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every contributor had to provision a database by hand, and the advisory-lock tests that follow in this series need the same setup. Add mise run test:rust:postgres. It runs every ignored postgres_* test in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE selects one) on the image pinned by the Kubernetes e2e fixture and removes it on exit. Tests run one at a time because advisory locks are database-wide. The runner always points the legacy replay variable at the selected database, so an inherited URL cannot send that test to a different server. test:postgres-runner checks that selection with a fake cargo and needs no database or container engine. CI does not run the PostgreSQL tests; TESTING.md documents the task. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): hold the mutation guard while deleting a provider DeleteProvider checked that no sandbox referenced the provider and then deleted the record without holding the sandbox mutation guard. Sandbox create and provider attach take that guard while they write provider references, so one of them could add a reference after the attached sandbox check passed, and the delete then removed a provider that a sandbox spec still named. Take the guard before the attached sandbox check, as provider create and update already do, so the check and the delete run against a stable set of sandbox references, and reject an empty name after authorization but before the guard. A new test holds the guard, attaches the provider while the delete waits, and asserts that the delete fails with FailedPrecondition and leaves the provider in place. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): release staged credentials when a refresh fails early A provider credential refresh stages the minted values under new credential handles before it commits the provider update, and deletes those handles when validation or persistence fails. Two earlier returns skipped that cleanup. The minted expiry was converted to a protobuf timestamp only after staging, so an expiry outside the timestamp range failed the refresh and left the staged values in the credential driver. A failure to acquire the sandbox mutation guard returned the same way. Convert the expiry before staging anything, so a bad value fails before any handle exists, and delete the staged handles when the guard cannot be acquired. A new test refreshes a stored credential with an expiry of i64::MAX and asserts that the call fails, the provider keeps its original handles, and the credential driver holds the same number of values as before. The guard failure path has no test here, because the SQLite guard used by unit tests cannot fail. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool The cross-object mutation guard took its advisory-lock session from the 10-connection data pool and closed it on every release, so each guarded mutation opened a new database connection, and lock holders competed with their own critical sections for data connections. A lock wait that hit the 10 second lock_timeout surfaced as INTERNAL. Add a lazy, dedicated 4-connection lock pool and a new persistence::mutation_lock module for the lock key, mode and set types. Store::acquire_distributed_mutation_guard now takes a lock set and a deadline, and each lock statement sets lock_timeout to the time left before that deadline, so a cancelled acquisition never waits past it on the server. Returned connections are scrubbed with pg_advisory_unlock_all() and reused, and each return is bounded by 5 seconds so a stalled session cannot keep a pool permit. A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a new LockTimeout error, and so does a wait for a lock connection during which every one was checked out at some point, or that started with less than a second left, too little to open one. The API returns it as UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry delay, because the mutation lock was not acquired, so the request's guarded writes did not run. Only the guard acquisition classifies 55P03; any other 55P03 stays a database error. A lock connection that PostgreSQL does not open by the deadline despite at least a second to do so (refused, out of connection slots, starting up, or not answering, all of which SQLx retries silently) is a database error, not a lock timeout. Mutations still take the legacy global key exclusively, so old and new replicas keep excluding each other during a rolling upgrade. Each PostgreSQL-backed gateway pod can now open up to 14 database connections (10 data plus 4 lock) instead of 10, and the chart's autoscaling.maxReplicas comment says what its default of 4 needs. The SSH host identity lock stays on a data connection, because sandbox creation takes it while holding its mutation guard and a second lock-pool connection there could exhaust the pool. The provider credential refresh lock stays there too, so a refresh still progresses while every lock connection is in use. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): partition mutation locks by workspace and sandbox Every cross-object mutation, and every lifecycle and supervisor update, took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory lock. Unrelated sandboxes and providers therefore serialized across all replicas, and one slow critical section delayed every other mutation in the fleet. Replace that lock with hierarchical intention locks on a global key, one key per workspace and one key per sandbox, each held shared or exclusive. Global settings and policy writers and platform-scope profile writers hold the global key exclusively. Provider and workspace-scoped profile writers, including DeleteProvider and credential refresh, hold their workspace key exclusively. Sandbox writers and supervisor reports hold their sandbox key exclusively under shared global and workspace keys. Lifecycle, driver-watch and reconcile paths take only process-local keys and keep relying on compare-and-swap across replicas. Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each guard takes its local keys and then its advisory locks in ascending order under one 10 second deadline, and no path nests guards. The settings mutex is removed, since the global and sandbox keys cover it. Sandbox settings are keyed by sandbox name, which a sandbox key no longer covers once another sandbox reuses the name, so a settings update returns NOT_FOUND unless its sandbox still holds the name after the read, and ABORTED if the settings row was recreated before the write. The global key keeps its legacy value, so older replicas, which hold it exclusively for every mutation, still exclude new ones during a rolling upgrade. Guards now also give up after 10 seconds on SQLite, and provider refresh reports a guard timeout as UNAVAILABLE instead of INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND. Startup endpoint-status reconciliation still holds the global key for its whole scan. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): reconcile endpoint status on startup one sandbox at a time Startup endpoint-status reconciliation held the global mutation key exclusively for its whole scan. On PostgreSQL that blocked every guarded mutation on every replica until the starting gateway had walked all sandboxes, and the scan first waited behind every in-flight sandbox mutation. It also paged with an offset, so sandboxes deleted mid-scan shifted rows between pages, and a write that raced the scan (a Conflict from a lifecycle update on another replica, or a sandbox deleted after it was listed) failed gateway startup. Walk the sandboxes with keyset paging instead, and reset each candidate under its own sandbox-scoped guard, up to four at a time to match the lock pool. A candidate with a fresh shared owner is skipped before any guard is taken. Under the guard the reconciliation re-checks the owner, re-reads the sandbox, skips it when it is gone or no longer carries endpoint status, and writes against the version it just read. A Conflict or a database error at write time, which is how a row deleted after the re-read surfaces, is retried with a fresh read, at most five attempts per sandbox. A lock timeout on the guard is retried the same way, so a holder of the sandbox, its workspace or the global key that releases within the retry window cannot stop a replica from starting. No lock-timeout retry starts once the scan has run for three lock timeouts, so a scan stuck behind a long holder fails within about 40 seconds, inside the chart's 60-second startup probe, and logs the sandbox it gave up on. Store::list_all_messages loses its only caller and is removed. Reconciliation stays fail-closed: a holder that outlasts the retry window, an owner lookup failure, or a sandbox that keeps changing still fails startup, so stale endpoint success is never served. User-visible change: a restarting gateway no longer stalls mutations of unrelated sandboxes and providers on its peers while it reconciles. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): skip the endpoint-status lock when a session was replaced When a supervisor session ends, the gateway resets that sandbox's tool server endpoint evidence unless a replacement session already owns endpoint observation. The replacement check ran only after the reset had read the sandbox and taken its sandbox mutation guard, so a reset whose supervisor had already reconnected still queued behind other mutations of that sandbox, and on PostgreSQL it used a lock-pool connection, only to find it had nothing to do. Check for a replacement first: a live local session, or a fresh owner record on a peer. When one exists, return without reading the sandbox or taking the guard. Otherwise take the guard and repeat the same check under it, because a replacement's pre-acknowledgement reset runs under the same sandbox key and resetting after it would wipe the replacement's fresh evidence. Both checks share one helper, so they cannot drift apart. When no replacement exists, the reset costs one extra owner-row read. This helps most when a replica notices a dead stream late, after the supervisor has already reconnected elsewhere (a network partition or a keepalive expiry). Deferring to a replacement leaves the old session's evidence to the replacement's pre-acknowledgement reset, which can time out on its bounded mutation lock wait. That reset's failure path removed the replacement and released its owner record but scheduled no invalidation, so the old session's success evidence could outlive both sessions. An old session that ended after its replacement registered already deferred this way, and the unguarded check now makes every disconnect that finds the replacement defer too. When the reset fails while the replacement is still current, also schedule the retried invalidation that follows a session end. That failure path keeps the lifecycle demotion from abandon_session_setup, but runs it in the spawned task before the retry and only while this session is still current, so a timed-out reset returns promptly. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): cover staged credential cleanup on a lock timeout A provider credential refresh deletes its staged handles when it cannot acquire the provider mutation guard, but that path had no test: the SQLite guard used by unit tests could not fail. Mutation locks now acquire under a deadline that tests can shorten, so the timeout path can be exercised directly. The new test shortens the lock deadline to 50 ms, holds the workspace mutation guard, and refreshes a provider that already stores an AWS access key. The refresh returns UNAVAILABLE with the MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle, which still resolves to the old key, and the credential driver holds the same number of values as before the refresh. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock wait and timeout metrics A handler mutation guard can now wait up to 10 seconds for its keys and then fail with UNAVAILABLE, but nothing showed operators how long guards waited or how often they gave up, short of reading warning logs on each replica. Export openshell_server_mutation_lock_wait_seconds, a histogram of the time one guard took to acquire all of its keys (local registry plus PostgreSQL advisory locks), recorded on success, and openshell_server_mutation_lock_timeouts_total, a counter of guard acquisitions that timed out. Both carry a bounded scope label: global, workspace or sandbox. The platform scope, whose workspace is empty, counts as global. The histogram uses the same 1 ms to 15 s buckets as the relay and peer histograms, and the timeout counter starts at 0 for every scope, so rate() works on an idle replica. The timeout warning now also logs the scope. A guard that fails for another reason, such as a lock connection PostgreSQL did not open, is not counted as a timeout and logs a "mutation lock acquisition failed" warning instead. Lifecycle, driver-watch and reconcile paths take only process-local locks and never time out, so they are not recorded. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): add PostgreSQL mutation lock integration tests The scoped mutation locks were only unit-tested on SQLite, where the database layer is a no-op. Add ignored postgres_* tests that run the advisory-lock layer against a real PostgreSQL through mise run test:rust:postgres, each in its own disposable schema (TestSchema). persistence/mutation_lock_pg_tests.rs uses two stores on one database as two replicas. It covers disjoint sandbox scopes holding at once, a workspace key blocking only sandboxes in that workspace, the global key blocking every scope, and exclusion in both directions against a raw holder of the legacy global key, whose 55P03 stays a database error outside a guard acquisition. It checks that timed-out and cancelled acquisitions release the keys they already took, that a released lock connection returns to the pool without residual locks, that a release stalled on the network closes its session and the pool recovers, and that the lock pool never grows past its size. It also checks that a full lock pool times out as lock contention, while a lock connection that PostgreSQL never answers fails as a database error. compute/mutation_guard.rs runs the random scope mix across two PostgreSQL runtimes and checks that a by-id guard waits for a workspace writer on the other replica. The provider refresh lock test moves onto the runner too, and checks that a refresh proceeds while every lock connection is held. A bench_postgres_* capacity check measures a paced burst of 500 reconnects 12 ms apart into one replica and fails when the p99 lock wait reaches a fifth of the lock timeout. Machine load moves those waits, so mise run test:rust:postgres skips it and mise run test:rust:postgres:bench runs it alone, through a new OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner. The tests add test-only helpers: the lock pool's size and idle count and the backend pid of a held lock session. Nothing changes outside tests and the test runner. Part of NVIDIA#3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * ci: run the PostgreSQL-backed server tests Branch Checks now runs the postgres_* tests through the same runner developers use, on an x86_64 Linux runner with Docker. They cover lock scope exclusion, interop with replicas still on the old global key, cancellation, lock-pool bounds and connection failures, which no other CI lane checks directly. The load-sensitive bench_postgres_* capacity checks stay local. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * docs: document scoped mutation locks and lock-pool sizing In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): check mutation lock ordering in debug builds The mutation lock ordering rules were prose only, so a nested guard on a different key passed every test and could only deadlock under contention. Debug builds now register each held guard with the Tokio task (or the thread, outside a task) that acquired it, and panic before waiting when a local key, a blocking lifecycle gate or the SSH identity lock would break rules 1, 4 or 5. Startup endpoint-status reconciliation and the tests that hold a guard while driving guarded code run under lock_order::branch(), which marks deliberate concurrency within one task. Release builds compile the check out. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): do not hold gateway shutdown for the endpoint-status retry When a replacement supervisor session failed its endpoint-status reset, the spawned cleanup task kept the session lifetime guard through retry_endpoint_status_after_supervisor_disconnect. That retry only ends once the reset is durable, so a gateway shutdown during a database or lock outage waited the full 10 s and reported ownership cleanup as incomplete. The guard now covers only the lifecycle demotion, as in finish_supervisor_session, and a SQLite test checks that shutdown completes while the retry is still blocked. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * refactor(server): make the platform profile lock scope explicit Platform-scope provider profile writers (import, update, delete) pass an empty workspace on purpose and need X(global), because they validate sandboxes in every workspace. MutationScope::profiles() now names that escalation and picks Global for the platform scope, instead of relying on MutationScope::Workspace("") to mean it. An empty workspace name now trips a debug assertion, and release builds keep X(global) as a fail-safe that over-locks rather than under-locks. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): amortize local mutation lock table sweeps LocalMutationLocks swept its whole table on every key lookup, so a burst of N concurrent acquisitions on one replica cost O(N^2) under a blocking mutex. The table is now swept only when a new key grows it past the larger of 64 and twice the live entries left by the last sweep, which makes acquisition amortized O(1). Released entries, including those of cancelled waiters, stay bounded at about twice the peak live set. The lifecycle gate registry keeps its per-call sweep. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the database pool ceiling configurable Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5 for on-disk SQLite. One pool is shared by every database-backed RPC, so that number is also the gateway's ceiling on concurrent database work. Once every connection is checked out, callers queue on acquire and sqlx logs "time to acquire exceeded slow threshold"; sandbox creates then time out and are retried, which adds load rather than shedding it. The right ceiling is deployment-specific — it depends on how many gateway replicas share the database and what max_connections the server itself allows — so it cannot be a single number baked into the binary. Expose it on the same config surface as the rest of the gateway's settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the TOML key database_max_connections, resolved in that precedence order. Helm deployments set the key through gatewayConfig. Omitting it keeps each backend's previous value, so existing deployments do not move. Values below 1 are rejected rather than silently replaced by the default: a zero pool would block every acquire, and quietly ignoring a typo would reproduce the ceiling the operator is trying to lift. An in-memory SQLite database stays pinned to one connection, since the database lives inside that connection. Refs: NVIDIA#2561 Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com> (cherry picked from commit 33517b3) [emacchi: dropped the server.dbMaxConnections Helm value; operators set the key through gatewayConfig] Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the mutation lock pool size configurable The mutation lock pool was fixed at 4 connections per replica. That number rests on one reconnect-burst bench rather than scale testing, so a gateway that serves many more sandboxes needs a way to raise it without a rebuild. Add --db-lock-max-connections, OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections gateway.toml key next to the data pool knob, resolved in the same order; Helm deployments set it through gatewayConfig. The default stays 4, zero is rejected, SQLite ignores it with a warning, and the gateway logs both pool sizes at startup. PostgreSQL now needs at least 2 data connections, because the SSH identity lock keeps one while it queries the pool. The connection sizing docs, the HA guide formula and the debug skill now count data plus lock connections. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock errors and lock connections in use A mutation lock acquisition that failed without timing out, such as a lock connection PostgreSQL never opened or a failed lock statement, only logged a warning, and readiness stays healthy because it pings the data pool. Count those failures in openshell_server_mutation_lock_errors_total{scope}, kept apart from timeouts because they point at PostgreSQL capacity or connectivity rather than contention. Export openshell_server_mutation_lock_connections_in_use as a gauge slot held by each checked-out lock connection, next to openshell_server_mutation_lock_connections_capacity, which reads the configured lock pool size; both exist only with PostgreSQL. The metrics catalog, the HA guide's signals and PromQL examples, and the debug skill cover the new series. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): time mutation lock holds and test a stalled holder Nothing bounds how long a mutation guard is held, so a holder stuck in a compute driver, credential backend or middleware call keeps its scope locked while waiters time out. Each guard now records its hold in openshell_server_mutation_lock_hold_seconds{scope} when it is released, and logs a warning when the hold outlasted the lock wait timeout, so operators can find the replica and scope behind those timeouts. A PostgreSQL test with a stalled holder pins the current contract: waiters on its sandbox, workspace and global scopes fail by their deadline, other scopes proceed, the holder keeps its key, and a full lock pool on one replica leaves the other working. The metrics catalog, the HA guide and the debug skill describe the new signals, which observe holds without bounding them. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
) * fix(compute): allow stopping errored sandboxes for recovery Signed-off-by: Shiju <shiju@nvidia.com> * test(compute): focus sandbox error recovery coverage Remove duplicate cancellation and idempotency checks covered by existing lifecycle tests. Keep recovery guidance focused on the stop/start commands and timeout exception. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com>
…#2449) The core IPv6 forwarding fix landed on main via NVIDIA#2552; this carries the docs example forward into the current Port Forwarding section of docs/how-it-works/sandboxes/overview.mdx after the page move in NVIDIA#3705. Remaining scope is documentation only. Signed-off-by: Andrew White <andrewh@cdw.com> Co-authored-by: andrewwhitecdw <andrewwhitecdw@users.noreply.github.com>
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
…VIDIA#4106) * refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…A#4534) * ci: use lifecycle:frozen as the sole stale issue exemption Signed-off-by: Evan Lezar <elezar@nvidia.com> * ci: support state:frozen as a stale issue exemption Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com>
* Add triage duty tooling Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): surface dependabot prs early Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): keep local state untracked Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): clarify queue continuation order Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): remove playbook whitespace Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(triage): address duty tooling review feedback Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * docs(triage): minimize changes to original duty skills Signed-off-by: Jim Meyer <jimeyer@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Jim Meyer <jimeyer@nvidia.com> Co-authored-by: Jim Meyer <jimeyer@nvidia.com>
The seccomp notification probes install their listener on a launcher thread and receive it over a channel. When the install fails, the launcher drops the sender and exits, so the broker only sees a closed channel and returns a generic "launcher disappeared" error. The thread is never joined, so the kernel's errno is discarded. That collapses several distinct failures into one opaque string: a notification struct size mismatch, a failed PR_SET_NO_NEW_PRIVS, and any non-EINVAL SECCOMP_SET_MODE_FILTER rejection all look identical to a pre-5.19 kernel refusing WAIT_KILLABLE_RECV, which the install path already retries without the flag. Join the launcher on the recv() failure path and propagate its error unchanged, so both the ErrorKind and raw_os_error survive to the qualification report. Apply this to all three probes. Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
… docker tag to v9.8-1793088151 (opendatahub-io#134) Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com> Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com> Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com> Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com> Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
…A#4962) * chore(ci): extend issue stale window to 28 days Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * chore(ci): exempt roadmap issues from stale handling Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
…4832) * docs(security): use GitHub private vulnerability reporting Signed-off-by: Adrien Langou <alangou@nvidia.com> * docs(security): clarify report quality and advisory workflows Signed-off-by: Adrien Langou <alangou@nvidia.com> * docs(security): require structured evidence-backed reports Signed-off-by: Adrien Langou <alangou@nvidia.com> * docs(security): remove CVE coordination wording Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com>
Signed-off-by: Adrien Langou <alangou@nvidia.com>
) * fix(deps): update setuptools to 84.1.0 Update the locked setuptools package and the build backend minimum to the patched release for GHSA-xmj3-9gf7-h52w. Signed-off-by: Adrien Langou <alangou@nvidia.com> * fix(deps): update vitest to 4.1.12 Update Vitest and V8 coverage together to resolve GHSA-qxg6-89mg-rqv8 and GHSA-xp26-3454-mjgp. Preserve unrelated locked dependencies. Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com>
…A#4976) * fix(ci): allow accepted protobuf breaks for one prerelease Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(ci): cover all v0.1.6 prerelease candidates Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(ci): document full rollback of temporary protobuf exception Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(ci): complete protobuf test copyright header Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
… clusters (NVIDIA#4842) * feat(gateway): add hashed service-routing hostnames for multi-gateway clusters Closes NVIDIA#4840 Add an optional service_routing_gateway_name gateway setting (TOML, CLI, env, Helm) that disambiguates exposed service URLs when multiple gateways share a cluster, ingress, and wildcard certificate. When configured, exposed-service hostnames become a deterministic 16-character lowercase base32 SHA-256 label of (gateway name, workspace, sandbox, service) under the same base domain, keeping every hostname exactly one label deep for wildcard-certificate compatibility. Unset gateways preserve the existing readable hostname format byte-for-byte, and configured gateways still accept legacy hosts during migration. Hashed hosts resolve through an in-memory route index hydrated from persisted ServiceEndpoint records at startup, refreshed on miss for multi-replica deployments, maintained across expose, delete, and sandbox cleanup, and enforced for collisions before endpoint persistence. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(lint): satisfy clippy doc lints on main-landed sandbox_runtime code The shared-supervisor landing (9a85667) predates the last Branch Checks run on main, so its doc-comment style was never gated. Split the three overlong first doc paragraphs so workspace clippy -D warnings passes; behavior is unchanged. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(gateway): address hashed-hostname review feedback Three changes from PR review: Identity model: replace service_routing_gateway_name with an explicit service_hostname_format mode (readable, the default, or hashed). In hashed mode the discriminator hashed into exposed-service hostnames is the existing gateway installation name ([openshell.gateway].name, operator- or platform-assigned, defaulting to the Helm release name), so no second per-gateway name is introduced. The name must be a DNS label and unique among gateways sharing the service base domain — an operational contract held by the deploying platform, documented in the configuration reference. gateway_jwt.gateway_id remains separate and untouched; renaming the installation now changes every minted URL. Request-path amplification: miss-triggered service route index rebuilds are debounced to one attempt per five seconds. Unknown hash labels reach the rebuild path before authentication, and without the debounce each request carrying a random well-formed label triggered a full all-workspaces endpoint scan. Startup hydration and explicit rebuilds bypass the debounce. OCSF regression: cross-origin rejections for hashed service hosts emitted no security event because the logging path used the readable-host parser only. The emitter now resolves hashed hosts through the in-memory route index (never the miss-triggered rebuild path) and still emits for unknown hash-shaped labels, attributed to the raw label. Also updates the debug-openshell-cluster skill for the new gateway TOML fields, per the repository instructions for gateway config changes. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream NVIDIA#3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
…#4977) The "Using Podman" example no longer produces a working gateway on rootless Podman. Replace the single command with a tested procedure that runs the gateway with mTLS, and document the five settings it needs: a writable state directory, a same-path data directory, generated signing keys and TLS certificates, the SELinux type for the Podman socket, and matching host and container ports. Refs NVIDIA#4972 Signed-off-by: politerealism <burdcat17@gmail.com>
…VIDIA#4987) * ci(release): allow release publishers to be dispatched individually Move the TypeScript SDK and wheel publishers into reusable workflows and add workflow_dispatch to them and to snap-publish.yml, following sync-docs.yml. Release still calls each one. A maintainer can now publish one missing component for an existing release without rebuilding or rerunning Release. The snap publisher can upload artifacts from an earlier run, and a manual latest/stable upload requires a release tag so it still rechecks the newest stable release. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(release): allow the Go SDK tag to be dispatched individually Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(release): verify dispatched snap and wheel inputs match the release A dispatched snap upload checks that its artifacts were built for the release tag. A wheel publish first confirms the tag points to the given commit and that each wheel is attached to that release at the given version. artifact-run-id is declared on workflow_call so callers do not rely on an undeclared input. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(release): verify the provenance of dispatched snap artifacts A manual snap upload now requires artifact-run-id to be this repository's publishing workflow for the channel, and stable snaps must match the GitHub Release asset digests. A skipped stable upload now reports a notice. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * Revert "ci(release): verify the provenance of dispatched snap artifacts" This reverts commit 8bcf1e1. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com> Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com> Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
…A#4985) A launch the supervisor pool could not place was refused as AssignmentError::Capacity, so a pool with no attached member, a pool not ready yet, and a placement failure all surfaced as "the shared supervisor has no capacity for another sandbox instance". Add LaunchError::Unplaced carrying the pool's reason, and report a pool with no live member as having no attached replica instead of no replica with room. Capacity refusals of a full host are unchanged. Signed-off-by: divesh <dgude@nvidia.com>
) * fix(supervisor): guard shared supervisor pool configuration Raise the gateway's minimum supervisor pool recovery timeout from 10 to 30 seconds. A failed pool member takes up to about 24 seconds to detect, so a shorter deadline ended workloads another member would have adopted. The Sandbox Protocol range stays 10 to 300 seconds for compatibility. Refuse an [openshell.gateway.supervisor_pool] table during config preflight, and at startup before the compute driver is built, unless the selected driver hosts sandboxes on the shared supervisor pool. A new ComputeDriverFactory::uses_shared_supervision hook reports this; the Kubernetes factory returns true with supervisor_topology = "gateway". Widen the readiness deadline of a shared-startup supervisor test that timed out under full-workspace load. Signed-off-by: divesh <dgude@nvidia.com> * fix(gateway): require driver selection for pool preflight Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(gateway): remove pool preflight note Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: divesh <dgude@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com> Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com> Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: red-hat-konflux <126015336+red-hat-konflux[bot]@users.noreply.github.com> Co-authored-by: red-hat-konflux[bot] <126015336+red-hat-konflux[bot]@users.noreply.github.com>
Signed-off-by: Krzysztof Malczuk <kmalczuk@redhat.com>
…ream-main/2000krysztof Fix/sync upstream main/2000krysztof
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Sync from main to stable
This PR automatically syncs the
mainbranch to thestablebranch by opening a pull request frommainintostable.Sync Summary
Latest commit:
7454d4caMerge remote-tracking branch 'upstream/main'Total commits to sync: 250
Source:
https://github.com/red-hat-data-services/OpenShell.git@mainTarget:
https://github.com/red-hat-data-services/OpenShell.git@stablePR head:
mainCommits to be synced
7454d4caMerge remote-tracking branch 'upstream/main'7d48bd05chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] #50)bf8f013fchore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] opendatahub-io/openshell#132)6b705c69chore(deps): refresh rpm lockfiles (chore(deps): refresh rpm lockfiles opendatahub-io/openshell#133)a17a4295chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1793088151 (chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1793088151 opendatahub-io/openshell#134)f0675834Merge remote-tracking branch 'upstream/main'8b2f529aMerge remote-tracking branch 'upstream/main'8730ce53Merge remote-tracking branch 'upstream/main'f07127e5feat(sandbox): add suspend and resume lifecycle (feat(sandbox): add suspend and resume lifecycle NVIDIA/OpenShell#4765)7b0f8ae0fix(gateway): require authentication or explicit development opt-in (fix(gateway): require authentication or explicit development opt-in NVIDIA/OpenShell#4821)c8d318f8Merge remote-tracking branch 'upstream/main'0d4725b1Merge remote-tracking branch 'upstream/main'17eda52ddocs(windows): route main documentation to the Windows release branch (docs(windows): route main documentation to the Windows release branch NVIDIA/OpenShell#4758)f5d9223efeat(supervisor): let operators require specific middleware bindings (feat(supervisor): let operators require specific middleware bindings NVIDIA/OpenShell#4678)5c533d6fdocs: document local Nix setup and the OpenShell binary cache (docs: document local Nix setup and the OpenShell binary cache NVIDIA/OpenShell#4931)4cf8f7c0Merge remote-tracking branch 'upstream/main'58873a26Merge remote-tracking branch 'upstream/main'd0a163a4docs(rfc): add RFC 0018 for multi-sandbox and HA supervisors (docs(rfc): add RFC 0018 for multi-sandbox and HA supervisors NVIDIA/OpenShell#4377)2ff59bf3feat(agents): allow gator to reach GHSA private forks (feat(agents): allow gator to reach GHSA private forks NVIDIA/OpenShell#4893)435341cdMerge remote-tracking branch 'upstream/main'fc1e06ceMerge remote-tracking branch 'upstream/main'cc14afccfix(ci): exempt Dependabot PRs from issue acceptance (fix(ci): exempt Dependabot PRs from issue acceptance NVIDIA/OpenShell#4928)9c6c2520fix(deps): update tokio and crypto-bigint security patches (fix(deps): update tokio and crypto-bigint security patches NVIDIA/OpenShell#4924)37b73027chore(ci): update rust-cache to include upstream fixes (chore(ci): update rust-cache to include upstream fixes NVIDIA/OpenShell#4748)90d8b46bMerge remote-tracking branch 'upstream/main'72bc3d36ci(release): infer optional workflow tags (ci(release): infer optional workflow tags NVIDIA/OpenShell#4707)92d5bcf4chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] #45)ba13dcd7Merge remote-tracking branch 'upstream/main'2884a8a4chore(deps): refresh rpm lockfiles (chore(deps): refresh rpm lockfiles opendatahub-io/openshell#131)d8f25065chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] opendatahub-io/openshell#130)587dc921Merge remote-tracking branch 'upstream/main'952095cbchore(deps): refresh rpm lockfiles (chore(deps): refresh rpm lockfiles opendatahub-io/openshell#128)5e17fe94chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] opendatahub-io/openshell#127)260e844dMerge remote-tracking branch 'upstream/main'94543428chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] #44)fb89c250chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1792869995 (chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1792869995 #43)baaf0d16chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1792869995 (chore(deps): update registry.access.redhat.com/ubi9/nodejs-24-minimal docker tag to v9.8-1792869995 opendatahub-io/openshell#129)ff4aafd4chore(deps): refresh rpm lockfiles (chore(deps): refresh rpm lockfiles #2)a57902c4Merge remote-tracking branch 'upstream/main'f4a9bd60Merge remote-tracking branch 'upstream/main'e3b4b05fMerge remote-tracking branch 'upstream/main'c4688aa8feat(helm): add certificate hook pod labels and annotations (feat(helm): add certificate hook pod labels and annotations NVIDIA/OpenShell#4895)e0016fa3Merge remote-tracking branch 'upstream/main'0b3b3a3dMerge remote-tracking branch 'upstream/main'f571c8e9docs(agents): ban AI attribution in PR descriptions and issues (docs(agents): ban AI attribution in PR descriptions and issues NVIDIA/OpenShell#4890)16250df2feat(helm): add server.extraEnv for the gateway container (feat(helm): add server.extraEnv for the gateway container NVIDIA/OpenShell#4876)567d3603fix(build): raise file limit for native gateway builds (fix(build): raise file limit for native gateway builds NVIDIA/OpenShell#3122)e0ba0789ci: keep rust caches warm on main and stop saving binary caches on branches (ci: keep rust caches warm on main and stop saving binary caches on branches NVIDIA/OpenShell#4881)b824e91bMerge remote-tracking branch 'upstream/main'1fce5ffaMerge remote-tracking branch 'upstream/main'fd5a55bcchore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] #1)92434dcbtest(windows): repair the Windows MSVC test suite (test(windows): repair the Windows MSVC test suite NVIDIA/OpenShell#4862)72e4eaa8feat(snap): publish prerelease builds to latest/beta channel (feat(snap): publish prerelease builds to latest/beta channel NVIDIA/OpenShell#4870)8244f739Merge remote-tracking branch 'upstream/main'e188cadcMerge remote-tracking branch 'upstream/main'b504c301ci(security): allow DRS version references in zizmor5bfe2fbcdocs(contributing): exempt build and flaky-test fixes from the issue requirement (docs(contributing): exempt build and flaky-test fixes from the issue requirement NVIDIA/OpenShell#4875)bc4c8f00fix(deps): update ctutils to resolve timing advisory (fix(deps): update ctutils to resolve timing advisory NVIDIA/OpenShell#4867)38fc5c26Merge pull request CARRY: fix(skills): move ODH installer guidance out of upstream skill opendatahub-io/openshell#124 from opendatahub-io/fix/skill-sync-conflict/andre-motta6b32278bCARRY: fix(skills): move ODH installer guidance out of upstream skill60e6434achore(deps): refresh rpm lockfiles (chore(deps): refresh rpm lockfiles opendatahub-io/openshell#120)2d7a6f85chore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] opendatahub-io/openshell#122)82a187cdMerge remote-tracking branch 'upstream/main'0e010064Merge pull request CARRY: test(odh): qualify managed workspace SCC identity isolation opendatahub-io/openshell#114 from 2000krysztof/test/odh-managed-workspace-scc-isolation5db572a9Merge remote-tracking branch 'upstream/main'6386a44cchore(deps): refresh rpm lockfiles [SECURITY] (chore(deps): refresh rpm lockfiles [SECURITY] opendatahub-io/openshell#119)eb0bdc9aMerge remote-tracking branch 'upstream/main'9a85667dfeat(kubernetes): host sandboxes on the shared supervisor (feat(kubernetes): host sandboxes on the shared supervisor NVIDIA/OpenShell#4702)07b7a74bfix(server): bound the provisioning-deadline scan's lock wait (fix(server): bound the provisioning-deadline scan's lock wait NVIDIA/OpenShell#4549)eec00ad4fix(podman): Increase the API timeout (Increase the podman driver's API timeout NVIDIA/OpenShell#4820)c018a045Merge remote-tracking branch 'upstream/main'132c5823docs(extensibility): document external compute driver setup (docs(extensibility): document external compute driver setup NVIDIA/OpenShell#4826)78376bd0fix(installer): resolve the gateway config dir like the CLI does (fix(installer): resolve the gateway config dir like the CLI does NVIDIA/OpenShell#4063)ea13dfcerefactor(supervisor): isolate state per sandbox instance (refactor(supervisor): isolate state per sandbox instance NVIDIA/OpenShell#4701)9152b6e4fix(vm): boot a digest-pinned image from the cache without a registry call (fix(vm): boot a digest-pinned image from the cache without a registry call NVIDIA/OpenShell#4582)cb42f5d9fix(snap): use openshell-prover instead of openshell.prover (fix(snap): use openshell-prover instead of openshell.prover NVIDIA/OpenShell#4835)fac82327test(testkit): add the testkit library for end-to-end suites (test(testkit): add the testkit library for end-to-end suites NVIDIA/OpenShell#4724)fe2eadf3Merge remote-tracking branch 'upstream/main'c2830aa0Merge remote-tracking branch 'upstream/main'7bbf1bfaMerge remote-tracking branch 'upstream/main'39d95b42chore(e2e): remove obsolete OpenShift sandbox SCC grant (chore(e2e): remove obsolete OpenShift sandbox SCC grant NVIDIA/OpenShell#4823)86e520a7Merge remote-tracking branch 'upstream/main'1a3330f6Merge remote-tracking branch 'upstream/main'0a516523chore(server): remove unused server macros crate (chore(server): remove unused server macros crate NVIDIA/OpenShell#4818)725d5c6fMerge remote-tracking branch 'upstream/main'029fda68refactor(cli): use Rust SDK transport (refactor(cli): use Rust SDK transport NVIDIA/OpenShell#4817)029ff74bMerge remote-tracking branch 'upstream/main'93508667Merge remote-tracking branch 'upstream/main'7bfc460cMerge remote-tracking branch 'upstream/main'1d880c9afix(examples): repair multi-agent notepad launch (fix(examples): repair multi-agent notepad launch NVIDIA/OpenShell#4807)20cb306dperf(docker): slow steady-state supervisor healthcheck (perf(docker): slow steady-state supervisor healthcheck NVIDIA/OpenShell#4802)4b7486e7test(auth): share fixtures for gateway audit event tests (test(auth): share fixtures for gateway audit event tests NVIDIA/OpenShell#4778)1846b2f0Merge pull request CARRY: fix(odh): restore upstream skill edits and SPDX header check opendatahub-io/openshell#118 from opendatahub-io/fix/sync-skill-license-headers/andre-motta33149c18CARRY: fix(odh): restore upstream skill edits and SPDX header check229382e5docs(sandbox): align workload image requirements with current runtime (docs(sandbox): align workload image requirements with current runtime NVIDIA/OpenShell#3967)5b429171Merge remote-tracking branch 'upstream/main'19ae2b7etest(cli): retry forward tests when ssh loses the port after the check (test(cli): retry forward tests when ssh loses the port after the check NVIDIA/OpenShell#4799)a4c27f9crefactor(auth): share audit helpers for gateway denials and rejections (refactor(auth): share audit helpers for gateway denials and rejections NVIDIA/OpenShell#4777)2fd802effeat(skills): add release-captain skill for the weekly release cadence (feat(skills): add release-captain skill for the weekly release cadence NVIDIA/OpenShell#4435)e7dc051bchore(sdk): update Vite to 8.1.6 (chore(sdk): update Vite to 8.1.6 NVIDIA/OpenShell#4792)Merging
GitHub automerge is enabled for this pull request once required checks pass.