Repository navigation
Conversation
InfoSec incident "Intrepid Falcon" (eMudhra intermediate CA compromise). The JDK 17 image used by the DRA steps is Ubuntu 20.04, which has no fixed ca-certificates package, and those steps run as uid 1000 so a runtime mitigation cannot apply. Distrust the two affected roots in the image at build time (which also rebuilds the Java keystore), failing the build if either remains in the keystore. Bump the image to version 4 and point the DRA create and S3 upload steps at it. The image must be built and pushed before this merges. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Pinging @elastic/ml-core (Team:ML) |
Version 4 already exists in the registry (pushed in 2023, never recorded in the repo, and still trusting the compromised CAs), so the patched image was pushed as version 5 and version 4 was left untouched. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Ubuntu 22.04's ca-certificates package (20260816ubuntu0.22.04.1 and later) carries the fix for the eMudhra intermediate CA compromise, whereas Ubuntu 20.04 has no fixed package. Rebuild the image on 22.04 and drop the build-time distrust workaround, keeping a build-time check that fails if either affected root is still in the Java keystore. Bump the image to version 6 and point the DRA create and S3 upload steps at it. Version 5 (Ubuntu 20.04 with the roots distrusted) remains in the registry as a fallback. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An earlier edit bumped the ml-linux-dependency-builder entry instead of the JDK 17 image entry. Restore the dependency builder to version 4 and record version 6 for ml_cpp_linux_x86_64_jdk17, matching build_linux_jdk.sh and the DRA pipelines. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
edsavage
changed the title
prwhelan
approved these changes
Oct 21, 2026
| # ca-certificates package (20260816ubuntu0.22.04.1 and later) removes the | ||
| # affected roots, so no workaround is needed here, but fail the build if an | ||
| # image built from a stale base still has either root in the Java keystore. | ||
| RUN ! keytool -list -cacerts -storepass changeit | grep -i -E 'comodo_aaa_services_root|emsign_ecc_root_ca_-_g3' |
Member
There was a problem hiding this comment.
do we also want to check if /etc/ssl/certs are fixed as well?
Contributor
Author
There was a problem hiding this comment.
Yes, I think that's a good idea, thanks! I'll add the check.
Extend the build-time guard to fail if either compromised root is in /etc/ssl/certs, matched by SHA-256 fingerprint, as well as the Java keystore. The guard also fails if the certificate store can't be read, so it can't pass vacuously. Tested against the published version 6 (passes), a stale Ubuntu 20.04 base (fails) and an image with no usable store (errors). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
jan-elastic
reviewed
Oct 22, 2026
| version being recorded here. It was left untouched. Check the registry for the | ||
| next free tag before building a new version. | ||
| - Version 5 was Ubuntu 20.04 with the eMudhra roots distrusted at build time. | ||
| Version 6 moved to Ubuntu 22.04, whose ca-certificates package carries the fix. |
Contributor
There was a problem hiding this comment.
Nit: - (bullet) missing
jan-elastic
reviewed
Oct 22, 2026
| RUN fps=$(for f in /etc/ssl/certs/*.pem; do openssl x509 -noout -fingerprint -sha256 -in "$f"; done) && \ | ||
| [ "$(echo "$fps" | wc -l)" -gt 50 ] && \ | ||
| ! echo "$fps" | grep -q -E 'D7:A7:A0:FB:5D:7E:27:31:D7:71:E9:48:4E:BC:DE:F7:1D:5F:0C:3E:0A:29:48:78:2B:C8:3E:E0:EA:69:9E:F4|86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B' && \ | ||
| ! keytool -list -cacerts -storepass changeit | grep -q -i -E 'comodo_aaa_services_root|emsign_ecc_root_ca_-_g3' |
Contributor
There was a problem hiding this comment.
What happens if keytool isn't on the path or cacerts is missing, or something else goes wrong?
If I understand correctly, it will return an error, an then ! keytool ... passes.
Contributor
There was a problem hiding this comment.
Suggested fix:
RUN bad_fps='D7:A7:A0:FB:5D:7E:27:31:D7:71:E9:48:4E:BC:DE:F7:1D:5F:0C:3E:0A:29:48:78:2B:C8:3E:E0:EA:69:9E:F4|86:A1:EC:BA:08:9C:4A:8D:3B:BE:27:34:C6:12:BA:34:1D:81:3E:04:3C:F9:E8:A8:62:CD:5C:57:A3:6B:BE:6B' && \
fps=$(for f in /etc/ssl/certs/*.pem; do openssl x509 -noout -fingerprint -sha256 -in "$f"; done) && \
[ "$(echo "$fps" | wc -l)" -gt 50 ] && \
! echo "$fps" | grep -q -E "$bad_fps" && \
java_fps=$(keytool -list -cacerts -storepass changeit | grep 'SHA-256') && \
[ "$(echo "$java_fps" | wc -l)" -gt 50 ] && \
! echo "$java_fps" | grep -q -E "$bad_fps"
jan-elastic
requested changes
Oct 22, 2026
jan-elastic
left a comment
jan-elastic
left a comment
Contributor
There was a problem hiding this comment.
Generally LGTM, see comments
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is a companion to the wider ranging #3251
The JDK 17 image used by the DRA steps (
create_dra,upload_dra_to_s3) was found by the CA trust checks in #3251 to trust the compromised CAs, and the steps run as uid 1000 (imageUID), so the runtime mitigation from that PR cannot apply.Instead we move from Ubuntu 20.04 (no fixed
ca-certificates) to Ubuntu 22.04, whoseca-certificates(20260816ubuntu0.22.04.1 and later) removes the affected roots, including from the Java keystore. With this approach no workaround is needed.The image version is bumped to
6and is built and pushed to the docker registry.TODO: Backports?