Repository navigation
test: align contract coverage and isolate integration suites - #130
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
It spans ~40 files and reclassifies many tests out of the default CI suite, a coverage-impacting decision that warrants human review despite the changes being almost entirely test-only.
1 open issue
What changed in this PR
This PR applies the AGENTS.md testing policy across the whole workspace. It removes brittle snapshot assertions (adjustable task defaults, migration-absence checks, duplicate/fake-only tests), replaces implementation-shape assertions with observable-behavior checks, isolates heavyweight tests behind @pytest.mark.integration/@pytest.mark.slow, makes the metrics-backoff and DDP tests deterministic, and fixes the stale motrix-deploy version by reading distribution metadata. It is almost entirely test-only; the single production change replaces a hardcoded __version__ with importlib.metadata.version("motrix-deploy").
Changes:
- Reclassify real-network/subprocess/simulator/ONNX/GPU tests as
integration, keeping lightweight unit cases in the fast suite, and document the selection split inAGENTS.md. - Replace snapshot/implementation-shape assertions with behavior- or relationship-based checks (randomization bounds, drain counts, friction forwarding, ONNX clipping parity, noise distribution via the production kernel).
- Harden flaky/heavyweight tests: deterministic metrics backoff via a controlled clock/condition, FileStore-based DDP rendezvous with guaranteed worker cleanup, CUDA-index test without a real GPU, and resolve the deploy version from metadata.
| File | Description |
|---|---|
| AGENTS.md | Documents the fast/numba/integration pytest selections and integration scope. |
| motrix_deploy/src/motrix_deploy/__init__.py | Reads __version__ from distribution metadata instead of a hardcoded string. |
| motrix_deploy/tests/test_package.py | Updates version test to compare against metadata (now tautological); marks CLI/import tests integration. |
| motrix_deploy/tests/test_config.py | Adds CLI _duration_steps boundary test; drops adjustable-default snapshots. |
| motrix_deploy/tests/test_{policy,artifact}.py | Marks ONNX/CLI inspect tests integration. |
| motrix_rl/tests/test_metrics.py | Deterministic backoff via controlled clock/condition; socketless fixtures; integration markers. |
| motrix_rl/tests/test_runner.py | Marks live-metrics test integration. |
| tests/test_cli_config.py | Resolves provider config type via frameworks.get_config_type; drops default snapshots. |
| tests/test_bench_duration.py | Marks spawn-worker tests integration. |
| motrix_rl_builtin/tests/* | Isolates CUDA/DDP/IPC/spawn tests; strengthens drain-count and trainer-selection assertions. |
| motrix_rl_skrl/tests, motrix_rl_rslrl/tests | CPU-only mock device; ONNX clipping parity via reference; seed set-compare; integration markers. |
| motrix_envs/tests/* | Noise tested through production batch kernel; randomization bounds from config; removes snapshot/shape assertions. |
| motrix_env_core/tests, motrix_env_motrixsim/tests | Backend-inference tests via real base classes; removes duplicate/abstract-surface tests. |
| motrix_deploy_mujoco/*, motrix_deploy_motrixsim/*, motrix_deploy_unitree/*, motrix_deploy_tasks/* | Broad integration marking; behavior-based physics/friction/limit checks replacing monkeypatch call-count snapshots. |
Review effort: Balanced
4786d94 to
a1424e7
Compare

Summary
Validation
Compatibility and limitations