[sputt]

With --experimental_remote_repo_contents_cache, a module extension or
repository rule that reads a Label makes Bazel copy that whole repo to
disk. The python extension and the pythons_hub repo read
runtimes_manifest.txt this way, so every build materializes
rules_python.

This replaces runtimes_manifest.txt and
runtimes_manifest_workspace.bzl with a single runtimes_manifest.bzl
(MANIFEST_TEXT) loaded directly by both Bzlmod and WORKSPACE mode, and
updates sort_manifest.py to sort runtimes_manifest.bzl in place.

Before: DEBUG: Materializing remote repo @@rules_python+ on every
build. After: no materialization; the same 741 @pythons_hub targets
and an identical versions.bzl. Tested with Bazel 10.0.0rc1 and a
remote cache, querying @pythons_hub//:all from a fresh output base
after a warm-up run.

Tests: the mock module_ctx can't read real files, which is why the
extension tests had a fake default manifest. Since the manifest is now
loaded rather than read, they use the real one, and the fake is removed.
The archive-flavor test passes its own manifest through a new
testing-only _default_runtime_manifest argument.

[rickeylev]

Hm, interesting, and a bit confusing.

(1) How can the rules_python bzlmod extension be processed if rules_python isn't available?

(2) Does this effectively mean bzlmod extensions can't read any files without incurring a download of an entire repository?

[sputt]

(1) Bazel has a virtual file system that will download the remote files on demand when loading bzl files. All the starlark can work without materializing the whole thing.
(2) Yes absolutely true today. You will see a "materializing" warning when this happens. I think we can fix this in Bazel bazelbuild/bazel#31923

This is a decent change on its own merits, but we can close in favor of a ctx.read(label) change later after my Bazel PR merges if you prefer.

[rickeylev]

I brought this up in #bzlmod on bazel slack: https://bazelbuild.slack.com/archives/C014RARENH0/p1792553506777879

It turns out bazel will prefetch all .bzl files. So renaming it to runtimes_manifest.bzl (even if its just plain text) would be a hacky way to avoid the later fetch.

Does changing read(path(label)) to just read(label) avoid the problem of materializing the entire rule_python repo? From that conversation, it sounded like the issue is path() currently forces full materialization. read(label) should incur fetching just the single file from the remote cache instead. If so, lets go with that

If not, then:

(1) Whats the minimum version of bazel to get the above behavior? Basically we're going to drop bazel 7 support when bazel 10 comes out, so I'd rather not add it just for bazel 7.

(2) Some variant of this PR SGTM. In that case, the main change in this PR to make is removing the dead code and renaming as appropriate -- runtimes_manifest.txt essentially becomes unused, and runtimes_manifest_workspace.bzl becomes a misnomer.

But either way, there's no avoiding the fetch of the content: either bazel prefetches it due to the .bzl extension, or it gets fetched later when read() is called.

[rickeylev]

does read(label) (no path()) avoid the problem

I checked, it doesn't. There's a pending PR in bazel to allow that, but not in current versions. Ah well. Yeah, lets move it all to the .bzl file as a string for now, then.

[sputt]

Min version: the remote repo contents cache and .bzl prefetch (bazelbuild/bazel#29082) are in 9.0.0, backported to 8.8.0. Nothing here is specific to Bazel 7. It’s a plain load(), so it works on any version.

Fetching: agreed, the bytes get fetched either way. The difference is one .bzl file versus copying all of @rules_python+ to disk.

Gap: #4280’s mctx.read(Label("//python/private:artifacts_manifest.json")) in the config extension still materializes the repo for every consumer. The release pipeline injects that file, so the fix touches the generator too. We can send it as a separate PR.

Upstream: my bazelbuild/bazel#31923 duplicated bazelbuild/bazel#31493, which is still open. Even after it ships, the runtimes manifest stays a .bzl because WORKSPACE needs it. artifacts_manifest could go back to read(label) once the minimum Bazel version has the fix.

[rickeylev]

@sputt feel free to send a PR to address artifacts_manifest.json in a similar manner as done in this PR. i.e. move content into .bzl file with embedded json string