Skip to content

test: align contract coverage and isolate integration suites - #130

Merged
wlgys8 merged 1 commit into
mainfrom
refactor/test-contracts-and-isolation
Oct 17, 2026
Merged

wlgys8 merged 1 commit into
mainfrom
refactor/test-contracts-and-isolation

Conversation

@wlgys8

@wlgys8 wlgys8 commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Apply the AGENTS.md testing policy across environment, deployment and RL suites: remove adjustable task-default snapshots, obsolete migration absence assertions, fake-only and duplicate checks.
  • Replace implementation-shape assertions with observable behavior: configured randomization bounds, repeated simulator opens, physics forwarding and ONNX clipping parity.
  • Reduce heavyweight fixtures: test noise distribution through the production noise kernel; use synthetic motion-library data; test invalid duration at the CLI boundary without repeated full subprocess launches.
  • Make metrics backoff regression deterministic with controlled time and synchronization; guarantee sender cleanup. Isolate DDP rendezvous and clean up both workers on failure; remove unbounded drain loops.
  • Classify real network, subprocess/distributed, simulator, ONNX and GPU tests as integration while retaining lightweight unit cases. Document test selections in AGENTS.md.
  • Fix the stale motrix-deploy runtime version by reading distribution metadata and testing the consistency relationship instead of a hardcoded version.

Validation

  • Repository fast suite: 1676 passed, 5 skipped (not slow and not integration and not numba).
  • Dedicated Numba/slow CPU suite: 35 passed.
  • Integration suite excluding test/test_all_envs.py: 194 passed.
  • Additional targeted environment, deployment, provider, CLI and registry suites passed during implementation.
  • Metrics backoff regression passed 20 consecutive runs.
  • prek run --all-files
  • git diff --check
  • uv lock --check

Compatibility and limitations

  • Training/runtime behavior unchanged except resolving the stale deployment package version from installed distribution metadata.
  • No new project dependencies or lockfile changes.
  • Validation used Python 3.10 with bounded CPU/Numba threads. Torch tests temporarily used existing system NVIDIA library directories through LD_LIBRARY_PATH because the venv lacks some shared libraries; Torch was not reinstalled.
  • Full environment-catalog smoke, actual GPU execution, real DDS hardware and interactive viewers were not validated. Hardware/viewer tests use injected implementations; simulator and ONNX CPU integration tests were executed.
  • Based on latest main after PR feat(rl): expose GPU profiles to source consumers #129 merged; this PR contains only the testing cleanup.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It spans ~40 files and reclassifies many tests out of the default CI suite, a coverage-impacting decision that warrants human review despite the changes being almost entirely test-only.

1 open issue
What changed in this PR

This PR applies the AGENTS.md testing policy across the whole workspace. It removes brittle snapshot assertions (adjustable task defaults, migration-absence checks, duplicate/fake-only tests), replaces implementation-shape assertions with observable-behavior checks, isolates heavyweight tests behind @pytest.mark.integration/@pytest.mark.slow, makes the metrics-backoff and DDP tests deterministic, and fixes the stale motrix-deploy version by reading distribution metadata. It is almost entirely test-only; the single production change replaces a hardcoded __version__ with importlib.metadata.version("motrix-deploy").

Changes:

  • Reclassify real-network/subprocess/simulator/ONNX/GPU tests as integration, keeping lightweight unit cases in the fast suite, and document the selection split in AGENTS.md.
  • Replace snapshot/implementation-shape assertions with behavior- or relationship-based checks (randomization bounds, drain counts, friction forwarding, ONNX clipping parity, noise distribution via the production kernel).
  • Harden flaky/heavyweight tests: deterministic metrics backoff via a controlled clock/condition, FileStore-based DDP rendezvous with guaranteed worker cleanup, CUDA-index test without a real GPU, and resolve the deploy version from metadata.
File Description
AGENTS.md Documents the fast/numba/integration pytest selections and integration scope.
motrix_deploy/​src/​motrix_deploy/​__init__.py Reads __version__ from distribution metadata instead of a hardcoded string.
motrix_deploy/​tests/​test_package.py Updates version test to compare against metadata (now tautological); marks CLI/import tests integration.
motrix_deploy/​tests/​test_config.py Adds CLI _duration_steps boundary test; drops adjustable-default snapshots.
motrix_deploy/​tests/​test_{policy,artifact}.py Marks ONNX/CLI inspect tests integration.
motrix_rl/​tests/​test_metrics.py Deterministic backoff via controlled clock/condition; socketless fixtures; integration markers.
motrix_rl/​tests/​test_runner.py Marks live-metrics test integration.
tests/​test_cli_config.py Resolves provider config type via frameworks.get_config_type; drops default snapshots.
tests/​test_bench_duration.py Marks spawn-worker tests integration.
motrix_rl_builtin/​tests/​* Isolates CUDA/DDP/IPC/spawn tests; strengthens drain-count and trainer-selection assertions.
motrix_rl_skrl/​tests, motrix_rl_rslrl/​tests CPU-only mock device; ONNX clipping parity via reference; seed set-compare; integration markers.
motrix_envs/​tests/​* Noise tested through production batch kernel; randomization bounds from config; removes snapshot/shape assertions.
motrix_env_core/​tests, motrix_env_motrixsim/​tests Backend-inference tests via real base classes; removes duplicate/abstract-surface tests.
motrix_deploy_mujoco/​*, motrix_deploy_motrixsim/​*, motrix_deploy_unitree/​*, motrix_deploy_tasks/​* Broad integration marking; behavior-based physics/friction/limit checks replacing monkeypatch call-count snapshots.

Review effort: Balanced

Comment thread motrix_deploy/tests/test_package.py Outdated
@wlgys8
wlgys8 force-pushed the refactor/test-contracts-and-isolation branch from 4786d94 to a1424e7 Compare October 17, 2026 16:24
@wlgys8
wlgys8 merged commit 2155781 into main Oct 17, 2026
8 checks passed
@wlgys8
wlgys8 deleted the refactor/test-contracts-and-isolation branch October 17, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants