[simonchrz]

Sharded repodata (CEP 16) fell through to the uncached passthrough, so clients that use it, such as pixi, got neither cooldown filtering nor an offline copy.

• Without cooldown, repodata_shards.msgpack.zst and the shards are served through the metadata cache.

• With cooldown, the proxy rewrites the index. Clients cache shards by the hash in the index and never fetch a hash twice, so filtered shards can't keep the upstream hash. In the rewritten index:

  • each hash is XORed with a mask derived from channel, subdir and the current hour;
  • shards_base_url points at cooldown-shards/{hour}/.

  That route undoes the mask, fetches the upstream shard (through the cache) and drops records still in cooldown. Clients fetch their shards again once an hour, and versions leaving cooldown show up within that hour. The plain shard paths answer 404 while cooldown is on.

• Fallback: if the index puts packages or shards somewhere the proxy doesn't route, or has an unknown version, the proxy answers 404. rattler then falls back to repodata.json, which is filtered as before.

Supported layouts are shards next to the index (conda-forge) and ./shards/ (rattler-index).

This adds vmihailenco/msgpack/v5 as a dependency; klauspost/compress moves from indirect to direct.

Checked with pixi 0.83 against conda-forge through a local proxy. With a 30 day cooldown it locked numpy 2.5.3 and fetched only cooldown-shards/... paths, since 2.5.4 is from 10 October. Without cooldown it locked 2.5.4, and with cache_metadata the indexes and 43 shards were cached.

Fixes #529

[andrew]

shardsLayout resolves base_url and shards_base_url against the upstream index URL, but the client resolves them against the proxy's URL, so two kinds of layout pass the check without going through the proxy:

• An absolute URL back to the upstream subdir (https://conda.anaconda.org/conda-forge/noarch/) is accepted. Without cooldown the index is served verbatim and the client fetches shards and packages straight from upstream, uncached. With cooldown shards_base_url is rewritten but base_url isn't, so package downloads still go around the proxy.
• A root-relative path is accepted too. prefix.dev publishes base_url: "/conda-forge/noarch/", which matches the subdir upstream but resolves to http://host/conda-forge/noarch/ on the proxy, outside the /conda/ mount, and 404s. prefix.dev only avoids this today because its shards_base_url is on shards.prefix.dev and fails the check first.

Please have resolve reject any reference with a scheme, a host, or a leading /, so only empty or relative paths are accepted, and add both cases to TestCondaShardIndexUnsupportedLayout.

Smaller: the header comment at conda_shards.go:21-37 mostly repeats the commit message. Cut it down to why the hashes are masked (clients cache shards by hash, so filtered shards need different hashes that rotate hourly); the 404 fallback is already described on the handlers.

[simonchrz]

Done in eb5ff1a: resolve now rejects a scheme, a host or a leading /, and TestCondaShardIndexUnsupportedLayout covers the upstream URL (with and without scheme) and root-relative paths for both fields. The header comment is cut down to why the hashes are masked.