[edsavage]

Summary

Follow-up to #3251 (review suggestion): bake the eMudhra intermediate CA compromise mitigation (InfoSec incident "Intrepid Falcon", https://emudhra.we-have-a-problem.com/) into the Orka macOS image instead of applying it at runtime on every job. That removes the runtime authorization workaround on macOS, covers the checkout clone, and handles stores the runtime script doesn't touch on macOS (JDK cacerts, Homebrew, certifi). Temporary: remove once the base image and Homebrew carry the fix.

• .ci/orka/distrust_emudhra_cas.sh (matches the two roots by SHA-256 fingerprint):
  • early, before install.sh: deny the roots in the Keychain (security add-trusted-cert -d -r deny, with the temporary com.apple.trust-settings.admin right if a headless session is refused) and remove them from /etc/ssl/cert.pem using only awk (the Orka 3 base may not have python3 yet: DEV-2062 : Orka 3 Migration #3026 added brew install python@3.12 to install.sh). install.sh and third_party_deps.sh download the Homebrew installer, Zulu JDK, CMake, Boost and PyTorch over TLS (the last two end up in shipped binaries), so this runs first.
  • late, after the installs: brew upgrade ca-certificates (best effort), then remove the roots from Homebrew's bundle, every certifi and pip-vendored cacert.pem in the known site-packages locations, and every JDK's cacerts.
  • verify: fail the build unless the bundles and JDK keystores are clean, the Keychain no longer trusts the roots, and /usr/bin/curl rejects both of the incident's test hosts.
• orka3-macos-14-arm.pkr.hcl: copy the script and run it early before install.sh, then late and verify after third_party_deps.sh. The image is renamed to ml-macos-14-arm-004: building an existing name succeeds but pushing it to the registry fails, so each new image needs a new name (the image CI uses today is -003).
• third_party_deps.sh: PIP_CERT=/etc/ssl/cert.pem for the pip install (pip's vendored certifi still has the roots, and this runs after early).
• README.md: update for the Orka 3 definition (it described the old Orka 2 file, which I kept listed: it built ml-macos-14-arm-003.orkasi, the image CI uses today).

Testing

The functions were exercised on copies (nothing on the dev machine was modified), under macOS bash 3.2:

• PEM filter on a copy of /etc/ssl/cert.pem: 128 -> 126 certs, both roots gone, header lines removed too, idempotent. curl --cacert with the filtered copy rejects both test hosts (exit 60) and still works for GitHub, PyPI and Maven Central.
• PEM filter on a copy of Homebrew's bundle (192 -> 190). Discovery found 10 certifi and pip-vendored bundles across the Pythons on the machine, every one of which contained at least one of the roots.
• JDK keystores (copies of 17, 21, 22 and 24): exactly the two roots removed (for example 106 -> 104 entries). This caught a real bug: aliases look like comodoaaaca [jdk] and contain a space.
• verify on an unmitigated machine reports every store and exits non-zero. Its Keychain check uses security verify-cert -p basic, which I confirmed returns 0 for trusted roots and non-zero for untrusted certificates.

Not tested here (needs the Packer build): the Keychain denial and authorizationdb handling (same commands as the runtime script in #3251, which pass on the Orka agent in CI), brew update/upgrade, running as root on a fresh VM, and the Packer run itself (packer validate -syntax-only passes).

To validate and roll out

1. Build the image (packer build orka3-macos-14-arm.pkr.hcl); verify fails the build if any store is still unmitigated. If the Packer SSH session refuses an action (DEV-2062 : Orka 3 Migration #3026 hit similar headless-session quirks with launchctl), early stops the build with the error rather than continuing.
2. Use the new image in the pipelines. Not in this PR: DEV-2062 : Orka 3 Migration #3026 did not change the pipelines, which still use ml-macos-14-arm-003.orkasi (built by the Orka 2 definition and migrated to Orka 3 by MacStadium, not rebuilt: "we should not have to rebuild the image on the new environment as they are transitioning them"). An image built from this definition is named ml-macos-14-arm-004; I don't yet know the exact image value (with or without the .orkasi suffix the migrated image has) to put in build_macos.json.py for it. main, 9.5 and 9.4 all use -003; 8.19 uses a separate macOS 12 image that is not covered here.
3. Once the branches use it, remove the macOS branch of the runtime script from [ML] Check CI images for, and mitigate, trust of the compromised eMudhra CAs #3251.

Notes

• The README's "Versioning" section keeps the sequence-number convention and now says why: building an existing name succeeds but the registry push fails.
• The README's description of the source image's configuration (Xcode command line tools, auto-login) predates Orka 3 and I have not verified it for generic-14-sonoma-arm; the Orka 3 definition adds the passwordless sudo entry itself.
• packer fmt -check already fails on this file upstream (alignment in the locals and source blocks); I did not reformat lines I didn't change.

🤖 Generated with Claude Code