[tamird]

The source hashing pass owns the source records and only needs each name and its digest after hashing. Consume the records in parallel, borrow each one to compute its seed and digest, then move the name into the result. This removes a string clone per source and lets each source’s subincludes drop when its hash finishes.

I replayed a synthetic graph with 82,500 source files, 68,000 rules and 9,500 generated files. The source names total 1,897,500 UTF-8 bytes. The unmodified v67.0.1 release and this change produced identical hash and edge files in all seven pairs. The first pair warmed the fixture. The six timed pairs include the slowest candidate run (8.265 s). Differences in median elapsed time, process CPU and peak RSS stayed within the observed variation.

The 192 Rust tests, 21 CLI tests and format check passed with bazel test --nocache_test_results //src:rust_tests //src:cli_tests //:rust_format_check.

Codex for Tamir.

Reproduce the input and compare outputs

Build the optimized generator at each revision with bazel build -c opt //:bazel-diff. Save the executables as stock and candidate in a fresh directory. In that directory, save the fixture helper as perf_workload.py and the program below as fixture.py. Fixture helper

from pathlib import Path

from perf_workload import (
    ATTRIBUTE_STRING, TARGET_GENERATED_FILE, TARGET_RULE, TARGET_SOURCE_FILE,
    encode_attribute, encode_rule, encode_string_field, encode_target, encode_varint,
    write_replay_bazel,
)

SOURCES, RULES, GENERATED, OWNERS = 82_500, 68_000, 9_500, 8_200


def source(i):
    package, name = f"bench/p{i // 512:03d}", f"s{i:06d}.py"
    return package, name, f"//{package}:{name}"


def records():
    for i in range(SOURCES):
        visibility = ["//visibility:public"]
        if i % 384 == 0:
            visibility.append("//bench:authorized")
        payload = encode_string_field(1, source(i)[2])
        payload += b"".join(encode_string_field(5, value) for value in visibility)
        yield encode_target(TARGET_SOURCE_FILE, payload)
    attrs = (
        encode_attribute("generator_name", ATTRIBUTE_STRING, string_value="generator"),
        encode_attribute("$internal_attr_hash", ATTRIBUTE_STRING, string_value="hash"),
    )
    for i in range(RULES):
        outputs = [f"//bench:g{i:06d}"] if i < OWNERS else []
        if i < GENERATED - OWNERS:
            outputs.append(f"//bench:g{i + OWNERS:06d}")
        inputs = [source(i)[2]]
        if i < SOURCES - RULES:
            inputs.append(source(i + RULES)[2])
        if i:
            inputs.append("//bench:r000000")
        payload = encode_rule(f"//bench:r{i:06d}", "bench_rule", location="",
                              attributes=attrs, rule_inputs=tuple(inputs), rule_outputs=tuple(outputs),
                              starlark_environment_hash="", instantiation_stack=())
        yield encode_target(TARGET_RULE, payload)
    for i in range(GENERATED):
        owner = i if i < OWNERS else i - OWNERS
        payload = encode_string_field(1, f"//bench:g{i:06d}") + encode_string_field(2, f"//bench:r{owner:06d}")
        yield encode_target(TARGET_GENERATED_FILE, payload)


if __name__ == "__main__":
    root = Path.cwd()
    for i in range(SOURCES):
        package, name, _ = source(i)
        path = root / package / name
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(f"VALUE = {i}\n")
    wire = root / "mixed.pb"
    with wire.open("xb") as out:
        for message in records():
            out.write(encode_varint(len(message)))
            out.write(message)
    write_replay_bazel(root / "bazel-replay", wire, root)

python3 fixture.py
for arm in stock candidate; do
  RAYON_NUM_THREADS=8 "./$arm" generate-hashes --workspacePath="$PWD" \
    --bazelPath="$PWD/bazel-replay" --excludeExternalTargets --includeTargetType \
    --depEdgesFile="$arm.edges.json" "$arm.hashes.json"
done
cmp stock.hashes.json candidate.hashes.json
cmp stock.edges.json candidate.edges.json

The SHA-256 digests were 902286b4f78b59693f2e5054c6e13c24edecdf21288c5e54255961a499c0802b for hashes and 20694e2d4903d82628a3bf289c5c5ef9b2b00611de41745ee252cf73a0000adf for edges.