Repository navigation
fix(packaging): exclude tests from the wheel - #979
Conversation
Colocated tests, test helpers, conftest.py, type_tests/ and test fixtures were shipped in the wheel (154 of 465 files). A setuptools build_py hook now drops them, shrinking the wheel from 1.16 MB to 0.68 MB. `make test-wheel` previously ran the shipped tests from site-packages, so it is replaced by a `test_wheel` nox session running py/tests/wheel/: a smoke test that checks version templating, py.typed, absence of test files, that every module imports, entry points, the CLI, and a local Eval. The upload-wheel CI job now runs it so packaging regressions fail on PRs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77ca3d32ea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| @nox.session() | ||
| def test_wheel(session): |
There was a problem hiding this comment.
When a developer runs the documented make test target from a clean checkout, it invokes nox -x without selecting sessions, so this new default session runs without the required preceding make build; _get_braintrust_wheel() then raises because dist/ contains no wheel. Nox documents that all default sessions run when none are specified and that default=False removes an explicitly invoked session from that set (Nox configuration), so declare this build-dependent session with @nox.session(default=False) (or exclude it from make test).
AGENTS.md reference: AGENTS.md:L152-L152
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, fixed in 87b6ef2: test_wheel is now @nox.session(default=False), so make test (bare nox -x) no longer picks it up. make test-wheel still selects it explicitly (verified: 11 passed after make build). Since nox -l lists non-default sessions with - rather than * , scripts/nox-matrix.py already skips it, so I also dropped the now-redundant --exclude-session test_wheel flags from checks.yaml and update-session-weights.yaml.
test_helpers is public and imported by downstream test suites (e.g. zenml-io/kitaru, braintrustdata/docs-training) for with_memory_logger and init_test_logger, so excluding it would break them. Keep shipping it and assert it in the wheel smoke test. Also drop a stale wheel-mode note from CONTRIBUTING.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`make test` runs a bare `nox -x`, which would pick up test_wheel and fail without a built wheel in dist/. Mark it default=False; it still runs when selected explicitly via `make test-wheel`. Non-default sessions are not listed with `* ` by `nox -l`, so nox-matrix.py already skips it and the explicit --exclude-session flags in CI are dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upload-wheel job now runs `make test-wheel`; update the reproduction steps and note how to read test_wheel failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0da52743c6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| [build-system] | ||
| requires = ["setuptools>=82.0.1"] | ||
| build-backend = "setuptools.build_meta" | ||
| backend-path = ["."] |
There was a problem hiding this comment.
Setting backend-path = ["."] declares that the PEP 517 backend itself must be loaded from this directory, but setuptools.build_meta is supplied by the build environment rather than the repository. Standards-compliant frontends therefore reject source and editable installs: for example, pip wheel . --no-build-isolation --no-deps now fails with BackendUnavailable: Cannot find module 'setuptools.build_meta' in ['.../py']. This breaks common pip install workflows from a checkout, Git URL, or sdist; expose the custom hook without declaring external setuptools as an in-tree backend.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 9868792. Reproduced BackendUnavailable with pip wheel . both with and without build isolation (uv doesn't enforce the in-tree check, which is why make build passed). _build_hooks.py is now the in-tree backend: it re-exports setuptools.build_meta (setuptools' documented wrapper pattern) and build-backend = "_build_hooks". Verified: pip wheel . (isolated and --no-build-isolation) and pip wheel <sdist> all succeed with 307 files and no test modules; pip install -e . still resolves tests from src/; make build && make test-wheel passes; editable test_core subset passes.
`backend-path = ["."]` with `build-backend = "setuptools.build_meta"` is rejected by pip (BackendUnavailable), because PEP 517 requires a backend named with backend-path to live in that path, breaking pip installs from a checkout, Git URL, or sdist. uv does not enforce this. Make _build_hooks the backend by re-exporting setuptools.build_meta, per setuptools' documented in-tree backend pattern. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
The wheel ships our colocated tests: 154 of its 465 files are
test_*.py,conftest.py, private test helpers (_test_*.py),type_tests/andfixtures/test-image.png. That puts aconftest.pyinto users' site-packages and roughly doubles the wheel size. Some shipped tests don't even work: one btx test module crashes on import becausebraintrust/btx/spec-ref.txtisn't packaged.Changes
py/_build_hooks.pyadds abuild_pysubclass that dropstest_*,_test_*andconftestmodules. It is registered via[tool.setuptools.cmdclass]withbackend-path = ["."], and included in the sdist throughMANIFEST.inbecause uv builds the wheel from the sdist.type_tests/andfixtures/are excluded both as packages and as package data. Pyproject settings alone can't do this, becauseexclude-package-datadoesn't filter.pymodules. The sdist still contains the tests.braintrust.test_helpersis still shipped. It's public and downstream test suites import it (zenml-io/kitaruandbraintrustdata/docs-trainingusewith_memory_loggerandinit_test_logger). The smoke test checks that it stays importable.make test-wheel. It rannox -s test_core -- --wheel, which executed the tests shipped inside the wheel. It now runs a newtest_wheelsession overpy/tests/wheel/test_wheel.py, from a temp dir against the wheel indist/. The test checks:py.typedis presenttype_tests/orfixtures/are in the wheelbraintrust.*submodule imports, allowing missing optional third-party packagesbraintrust --helpworksEval(..., no_send_logs=True)runs--wheelmode: the noxfile plumbing, the wheel-only VCR skip inconftest.py, the wheel branch intest_version.py, and two dead "scripts/ not available" skips.test_wheelis excluded from the sharded nox matrix (inchecks.yamlandupdate-session-weights.yaml).upload-wheeljob now runsbuild test-wheel, so packaging regressions like fix(packaging): include image fixture in wheel #977 fail on PRs instead of at release time.verify-buildunchanged.Validation
make build && make test-wheel: 11 passed.mainfailstest_does_not_ship_testsandtest_all_modules_import(the btxspec-ref.txtcrash). Its version check also failed, but only because that wheel was built without templating.nox -s test_corefrom source: 1095 passed, 106 skipped, 12 xfailed.🤖 Generated with Claude Code