An RL-agent training environment based on ghOSt and Gymnasium

Contents

1. Prerequisites
2. Build
3. Tests
4. Running the environment
5. Scheduler <-> environment protocol (v2)
6. Tests and scheduling metrics
7. Results: scheduler comparison

Prerequisites

• An Ubuntu 20.04 virtual machine;
• The ghOSt kernel, ghOSt-kernel. It can be installed automatically with the kernel installation script; brief instructions are in the comments at the top of the script;
• Bazel: Bazel Installation Guide;
• Apt packages: libnuma-dev libcap-dev libelf-dev libbfd-dev gcc clang-12 llvm zlib1g-dev python-is-python3 (see the original README below for details);
• Pip: Pip installation;
• Clone this repository and switch to the dev branch: git checkout dev;
• Creating a Python venv is recommended: python3 -m venv <path to the directory for the virtual environment>.

A set of commands for installing the userspace part (that is, everything after the kernel is installed successfully) could look like this:

curl -G -L https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64 -o bazelisk
sudo mv bazelisk /usr/local/bin/bazel
chmod +x /usr/local/bin/bazel

sudo apt update
sudo apt install libnuma-dev libcap-dev libelf-dev libbfd-dev gcc clang-12 llvm zlib1g-dev python-is-python3

git clone https://github.com/moevm/ghost-userspace.git && cd ghost-userspace
git checkout dev
echo "5.1.0" > .bazelversion
sudo apt install python3.8-venv
python3 -m venv .venv
source .venv/bin/activate

Build

Building the scheduler: bazel build rl_scheduler_agent. If the build fails, see this issue in the repository.

Building the test workload binaries: bazel build simple_exp single_exp depart_exp yield_exp

Bazel does not install the dependencies of the Python code; install them manually: pip install -r requirements.txt

Tests

Unit and integration tests of the environment

The tests of the training environment do not require most of the components to be installed. The project uses Bazel to run tests, but technically the tests can also be run directly. To run a single test file:

$ bazel test //rl_env/tests:<file name in rl_env/tests without extension> --test_output=all

For example, all tests of the environment class:

$ bazel test //rl_env/tests:scheduler_env_tests --test_output=all

All tests at once:

$ bazel test //rl_env/tests:all_tests --test_output=all --flaky_test_attempts=3

If you run the tests several times without changing the code, the results of successful runs are cached and the tests are not run again. To force the tests to run, add the --nocache_test_results flag to the test command.

End-to-end test

The end-to-end test rl_env/demo.py is outdated: it was meant for an environment that reacts only to the appearance of new tasks.

Running the environment

The scheduler, the environment and a minimal stub in place of the RL agent are started by the rl_env/launch.py script. The script must be run as root. If you use a venv, remember to activate it in the root shell:

$ sudo su
# source .venv/bin/activate # Change the path to match the location of your venv

launch.py accepts parameters of the environment and of the scheduler launch (usually not needed on the first run). To see the available parameters:

# python3 rl_env/launch.py --help

Starting the environment, the stub agent and the scheduler:

# python3 rl_env/launch.py <optional arguments>

Press Ctrl+C to stop all components. The scheduler's logs and metrics are written to rl_env/logs/<timestamp>/ (scheduler.out, scheduler.err, stats.json).

The workload is started separately, in another console:

$ bazel-bin/single_exp # Any other built workload can be used instead

If something goes wrong, enclaves left over from previous scheduler runs can be removed manually:

# ls /sys/fs/ghost
ctl  enclave_1	version
# echo destroy > /sys/fs/ghost/enclave_1/ctl

...or with a single command:

# for i in /sys/fs/ghost/enclave_*/ctl; do echo destroy > $i; done

If old enclaves are not removed, a new scheduler instance will not start.

Scheduler <-> environment protocol (v2)

A single persistent TCP connection: the environment (SchedulerEnv) listens on port 14014; the scheduler connects by itself and reconnects if the connection drops.

• Scheduler -> environment: u32 n_words, then n_words × u64 (network byte order): [request_id, action_expected, callback_type, task metrics (7), metrics of the queued tasks (7 × k)].
• Environment -> scheduler (only for events with action_expected == 1): u64 request_id, u32 place.

Scheduler agents never wait for the environment's answer. A task is queued at the default position immediately. When a decision with its request_id arrives, the agent of the task's CPU moves the task to position place (if it is still in the queue). An answer to another or an outdated request cannot be applied to a different task: decisions are matched by request_id and gtid.

rl_scheduler_agent flags:

• --rl_env_port (14014);
• --rl_decision_ttl (1s): how long to wait for an answer;
• --rl_max_inflight (1024);
• --runnable_timeout (5s, 0 disables it): the kernel watchdog. If a runnable task is not scheduled within this time, the enclave is destroyed and its tasks go to CFS, so a stuck agent cannot hang the workload forever;
• --stats_file: JSON with metrics, written on exit and on SIGUSR2.

Agent logs are written to files (rl_env/logs/<time>/scheduler.{out,err}). Never run the agent with a stdout/stderr=PIPE that nobody reads: after 64 KiB of output, all agent threads block in write() and scheduling stops.

Tests and scheduling metrics

C++ unit tests (runqueue, protocol, communicator). They need neither a ghOSt kernel nor root:

$ bazel test //:rl_scheduler_test --test_output=all

Hang stress test (root). On a hang it saves a snapshot of the thread states, gdb traces and logs. A table for analyzing the snapshots is at the top of the file:

# .venv/bin/python rl_env/tests/stress_test.py --bin_dir <bazel-bin> --cpus 6

Scheduler comparison on the same workload (sched_bench: threads with CPU bursts and sleeps). Compared: native CFS, fifo_per_cpu_agent, fifo_centralized_agent, agent_cfs, agent_sol and rl_scheduler_agent with the heuristic, front and random policies. Metrics: wakeup and start latency (p50/p99/max), turnaround, makespan, throughput, context switches. For RL, additionally: scheduling latency, CPU utilization, the fate of decisions (applied/late/expired/dropped/stale) and RTT. The result is written to summary.md and results.csv:

$ bazel build sched_bench rl_scheduler_agent fifo_per_cpu_agent fifo_centralized_agent agent_cfs agent_sol
# .venv/bin/python experiments/scripts/sched_bench.py --bin_dir <bazel-bin> --cpus 6 --procs 4 --reps 10

Results: scheduler comparison

Experiment

• Machine: VirtualBox VM, 8 vCPUs (AMD Ryzen 9 9900X3D host), 15 GiB RAM, kernel 5.11.0-ghOSt (ghOSt ABI 91).
• ghOSt CPUs: 0-5 (--cpus 6). CPUs 6-7 stay CFS-only for the desktop, the Python environment and the benchmark driver.
• Workload: 4 concurrent sched_bench processes × 4 threads = 16 threads on 6 CPUs. Each thread runs 200 cycles of 500 µs of CPU work (measured as thread CPU time, so preemption does not shorten it) followed by a 1 ms sleep with an absolute deadline.
• Repetitions: 20 per scheduler, all 160 completed (none hung). cfs-native runs the same threads in plain Linux CFS with no ghOSt agent.
• RL rows: rl_scheduler_agent --verbose=0 with an in-process SchedulerEnv driven by a fixed policy: heuristic (the first empty slot, i.e. the back of the queue, as in rl_env/launch.py), front (always place 0) and random. The scheduler asks the environment for a placement on TaskNew, on the first wakeup of a task not yet assigned to a CPU, on TaskYield and on TaskPreempted; every other callback is only reported to the environment.
• Command:

  # .venv/bin/python experiments/scripts/sched_bench.py --bin_dir <bazel-bin> --cpus 6 --procs 4 --reps 20
  

Each cell is mean +/- sample standard deviation over the 20 repetitions (the variance is the square of the standard deviation). Each repetition contributes one value per metric:

• wakeup latency p50 / p99: actual wake-up time minus the intended one after each sleep, i.e. how long a woken thread waited for a CPU, measured by the workload; the mean of the 4 processes' percentiles.
• wakeup max: the largest wakeup latency among the 4 processes.
• start latency p50: time from thread creation until it first runs; the mean over the 4 processes.
• turnaround: time from thread creation until it finishes; the mean over the 4 processes.
• makespan: wall time of the slowest of the 4 processes.
• throughput: iterations per second, summed over the 4 processes.
• involuntary context switches: per process, from /proc/<pid>/task/<tid>/status; the mean over the 4 processes. Voluntary switches are 800 per process for every scheduler (one per sleep) and are not shown.

scheduler	wakeup p50 (µs)	wakeup p99 (µs)	wakeup max (µs)	start p50 (µs)	turnaround mean (ms)	makespan (ms)	throughput (iter/s)	involuntary ctx switches
cfs-native	80.9 +/- 1.8	750.5 +/- 103.9	3125.7 +/- 2037.3	125.9 +/- 27.7	327.6 +/- 2.7	332.9 +/- 4.9	9684.9 +/- 99.5	18.6 +/- 8.6
fifo-per-cpu	96.0 +/- 4.6	4300.0 +/- 3882.4	11108.8 +/- 7555.4	510.2 +/- 179.9	356.1 +/- 27.1	358.6 +/- 27.4	8988.4 +/- 635.9	449.9 +/- 17.3
fifo-centralized	146.5 +/- 4.1	789.6 +/- 179.5	2754.0 +/- 3215.5	265.2 +/- 80.5	351.2 +/- 4.3	354.3 +/- 4.8	9066.5 +/- 116.8	7.2 +/- 1.0
ghost-cfs	103.1 +/- 1.8	3551.3 +/- 2577.4	11171.2 +/- 5464.2	265.9 +/- 228.9	408.4 +/- 11.2	417.9 +/- 12.6	7728.1 +/- 216.3	1240.6 +/- 12.0
sol	136.9 +/- 3.9	899.3 +/- 170.8	1792.0 +/- 682.7	439.8 +/- 133.8	352.7 +/- 2.4	357.1 +/- 2.7	8998.8 +/- 67.7	5.8 +/- 0.8
rl-heuristic	188.3 +/- 6.8	946.2 +/- 103.4	3717.5 +/- 950.8	570.1 +/- 226.0	358.1 +/- 1.9	367.4 +/- 3.0	8745.8 +/- 59.3	629.4 +/- 9.9
rl-front	189.1 +/- 7.3	1017.4 +/- 159.3	5721.3 +/- 8076.7	532.9 +/- 117.0	360.1 +/- 4.6	369.7 +/- 8.3	8694.7 +/- 165.2	626.9 +/- 8.5
rl-random	195.0 +/- 8.0	1018.5 +/- 153.4	4539.2 +/- 1828.8	567.5 +/- 93.1	361.5 +/- 2.4	369.6 +/- 3.3	8684.3 +/- 70.1	622.7 +/- 6.5

Metrics measured inside rl_scheduler_agent (--stats_file) and the fate of RL decisions, per repetition (mean +/- standard deviation over the 20 repetitions):

scheduler	sched. latency p99 (µs)	queue wait p50 (µs)	CPU utilization (%)	env RTT p50 (ms)	requested	applied	late	superseded	undelivered	abandoned	dropped	stale
rl-heuristic	463.2 +/- 90.2	26.8 +/- 0.8	70.0 +/- 0.6	138.7 +/- 12.6	2524.1 +/- 40.2	2.9 +/- 2.3	4.5 +/- 1.4	857.1 +/- 73.8	116.1 +/- 10.4	907.7 +/- 13.6	635.7 +/- 80.2	0 +/- 0
rl-front	488.3 +/- 98.4	26.9 +/- 0.9	70.1 +/- 0.7	132.8 +/- 11.3	2516.2 +/- 33.8	2.7 +/- 2.3	4.8 +/- 1.7	861.0 +/- 52.3	110.0 +/- 13.4	915.9 +/- 17.3	621.8 +/- 61.4	0 +/- 0
rl-random	514.2 +/- 76.7	27.0 +/- 0.7	70.3 +/- 0.8	136.2 +/- 9.2	2500.3 +/- 25.8	2.1 +/- 1.5	4.1 +/- 1.9	861.0 +/- 58.3	107.3 +/- 10.8	915.0 +/- 13.5	610.9 +/- 67.5	0 +/- 0

• sched. latency / queue wait: from the task becoming runnable until it is committed to a CPU (p99 and p50).
• applied: the task was moved to the requested place.
• late: the decision arrived after the task had left the queue.
• superseded: the task had already died or woken again with a newer request.
• undelivered: the decision was received but not yet applied when the agent stopped.
• abandoned: the request was still outstanding at shutdown.
• dropped: too many requests outstanding.
• stale: the answer carried a request id that was never issued.
• No request expired (--rl_decision_ttl 1 s); that column is omitted.

Observations

• The ghOSt schedulers reach 80-94% of native CFS throughput: FIFO centralized 94%, SOL and FIFO per-CPU 93%, the RL scheduler 90% with every policy, ghOSt CFS 80%. ghOSt CFS has the most involuntary context switches.
• Run-to-run stability differs a lot. FIFO per-CPU is the least stable: its throughput varies by 7% between repetitions (standard deviation / mean) and its p99 wakeup latency by 90%, so a single run can land anywhere between 7383 and 9678 iter/s, i.e. from 24% below native CFS to on par with it. ghOSt CFS is also unstable in p99 wakeup latency (73%). SOL, FIFO centralized, native CFS and RL vary by 0.7-2% in throughput. Repeat runs before drawing conclusions from differences of a few percent.
• The per-CPU schedulers (FIFO per-CPU, RL) and SOL have a higher start latency (≈440-570 µs) than native CFS (126 µs). With the per-CPU schedulers, new tasks are first handled by the agent on the default channel and then migrated round-robin.
• RL decisions almost never take effect: 2-3 of about 2500 requests per repetition are applied. A task waits about 27 µs in the queue (p50), while the Python environment answers in 133-139 ms under this load. So by the time an answer arrives, the task has almost always run, blocked or died. Requests on yields and preemptions add load on the environment, and about 25% of the requests are dropped because more than 1024 are outstanding. This is not a hang or a lost answer: stale is 0, so no decision was applied to the wrong task, and every request is accounted for. Making decisions matter requires either a much faster environment (batching, an in-process policy) or holding a task for a bounded time until its decision arrives.
• The three RL policies differ by about 1% at most. Some of these gaps are larger than the run-to-run spread: mean turnaround is 361.5 +/- 2.4 ms with random versus 358.1 +/- 1.9 ms with heuristic, which is detectable over 20 repetitions. The policies cannot cause this, since only 2-3 decisions per repetition are applied. The schedulers ran one after another (all 20 repetitions of one, then the next), so a slow drift of the VM during the run is the more likely cause; interleaving the repetitions of different schedulers would separate the two.

---

Original README for reference

ghOSt: Fast & Flexible User-Space Delegation of Linux Scheduling

ghOSt is a general-purpose delegation of scheduling policy implemented on top of the Linux kernel. The ghOSt framework provides a rich API that receives scheduling decisions for processes from userspace and actuates them as transactions. Programmers can use any language or tools to develop policies, which can be upgraded without a machine reboot. ghOSt supports policies for a range of scheduling objectives, from µs-scale latency, to throughput, to energy efficiency, and beyond, and incurs low overheads for scheduling actions. Many policies are just a few hundred lines of code. Overall, ghOSt provides a performant framework for delegation of thread scheduling policy to userspace processes that enables policy optimization, non-disruptive upgrades, and fault isolation.

SOSP '21 Paper
SOSP '21 Talk

The ghOSt kernel is here. You must compile and run the userspace component on the ghOSt kernel.

This is not an officially supported Google product.

---

Compilation

The ghOSt userspace component can be compiled on Ubuntu 20.04 or newer.

1. We use the Google Bazel build system to compile the userspace components of ghOSt. Go to the Bazel Installation Guide for instructions to install Bazel on your operating system.

2. Install ghOSt dependencies:

sudo apt update
sudo apt install libnuma-dev libcap-dev libelf-dev libbfd-dev gcc clang-12 llvm zlib1g-dev python-is-python3

Note that ghOSt requires GCC 9 or newer and Clang 12 or newer.

3. Compile the ghOSt userspace component. Run the following from the root of the repository:

bazel build -c opt ...

-c opt tells Bazel to build the targets with optimizations turned on. ... tells Bazel to build all targets in the BUILD file and all BUILD files in subdirectories, including the core ghOSt library, the eBPF code, the schedulers, the unit tests, the experiments, and the scripts to run the experiments, along with all of the dependencies for those targets. If you prefer to build individual targets rather than all of them to save compile time, replace ... with an individual target name, such as agent_shinjuku.

---

ghOSt Project Layout

• bpf/user/
  • ghOSt contains a suite of BPF tools to assist with debugging and performance optimization. The userspace components of these tools are in this directory.
• experiments/
  • The RocksDB and antagonist Shinjuku experiments (from our SOSP paper) and microbenchmarks. Use the Python scripts in experiments/scripts/ to run the Shinjuku experiments.
• kernel/
  • Headers that have shared data structures used by both the kernel and userspace.
• lib/
  • The core ghOSt userspace library.
• schedulers/
  • ghOSt schedulers. These schedulers include:
    • biff/, Biff (bare-bones FIFO scheduler that schedules everything with BPF code)
    • cfs/ CFS (ghOSt implementation of Linux Completely Fair Scheduler policy)
    • edf/, EDF (Earliest Deadline First)
    • fifo/centralized/, Centralized FIFO
    • fifo/per_cpu/, Per-CPU FIFO
    • shinjuku/, Shinjuku
    • sol/, Speed-of-Light (bare-bones centralized FIFO scheduler that runs as fast as possible)
• shared/
  • Classes to support shared-memory communication between a scheduler and another application(s). Generally, this communication is useful for the application to send scheduling hints to the scheduler.
• tests/
  • ghOSt unit tests.
• third_party/
  • bpf/
    • Contains the kernel BPF code for our suite of BPF tools (mentioned above). This kernel BPF code is licensed under GPLv2, so we must keep it in third_party/.
  • The rest of third_party/ contains code from third-party developers and BUILD files to compile the code.
• util/
  • Helper utilities for ghOSt. For example, pushtosched can be used to move a batch of kernel threads from the ghOSt scheduling class to CFS (SCHED_OTHER).

---

Running the ghOSt Tests

We include many different tests to ensure that both the ghOSt userspace code and the ghOSt kernel code are working correctly. Some of these tests are in tests/ while others are in other subdirectories. To view all of the tests, run:

bazel query 'tests(//...)'

To build a test, such as agent_test, run:

bazel build -c opt agent_test

To run a test, launch the test binary directly:

bazel-bin/agent_test

Generally, Bazel encourages the use of bazel test when running tests. However, bazel test sandboxes the tests so that they have read-only access to /sys and are constrained in how long they can run for. However, the tests need write access to /sys/fs/ghost to coordinate with the kernel and may take a long time to complete. Thus, to avoid sandboxing, launch the test binaries directly (e.g., bazel-bin/agent_test).

---

Running a ghOSt Scheduler

We will run the per-CPU FIFO ghOSt scheduler and use it to schedule Linux pthreads.

1. Build the per-CPU FIFO scheduler:

bazel build -c opt fifo_per_cpu_agent

2. Build simple_exp, which launches a series of pthreads that run in ghOSt. simple_exp is a collection of tests.

bazel build -c opt simple_exp

3. Launch the per-CPU FIFO ghOSt scheduler:

bazel-bin/fifo_per_cpu_agent --ghost_cpus 0-1

The scheduler launches ghOSt agents on CPUs (i.e., logical cores) 0 and 1 and will therefore schedule ghOSt tasks onto CPUs 0 and 1. Adjust the --ghost_cpus command line argument value as necessary. For example, if you have an 8-core machine and you wish to schedule ghOSt tasks on all cores, then pass 0-7 to --ghost_cpus.

4. Launch simple_exp:

bazel-bin/simple_exp

simple_exp will launch pthreads. These pthreads in turn will move themselves into the ghOSt scheduling class and thus will be scheduled by the ghOSt scheduler. When simple_exp has finished running all tests, it will exit.

5. Use Ctrl-C to send a SIGINT signal to fifo_per_cpu_agent to get it to stop.

---

Enclaves, Rebootless Upgrades, and Handling Scheduler Failures

ghOSt uses enclaves to group agents and the threads that they are scheduling. An enclave contains a subset of CPUs (i.e., logical cores) in a machine, the agents that embody those CPUs, and the threads in the ghOSt scheduling class that the enclave agents can schedule onto the enclave CPUs. For example, in the fifo_per_cpu_agent example above, an enclave is created that contains CPUs 0 and 1, though the enclave can be configured to contain any subset of CPUs in the machine, and even all of them. In the fifo_per_cpu_agent example above, two per-CPU FIFO agents enter the enclave along with the simple_exp threads when the simple_exp process is started.

Enclaves provide an easy way to partition the machine to support co-location of policies and tenants, a particularly important feature as machines scale out horizontally to contain hundreds of CPUs and new accelerators. Thus, multiple enclaves can be constructed with disjoint sets of CPUs.

Rebootless Upgrades

ghOSt supports rebootless upgrades of scheduling policies, using an enclave to encapsulate current thread and CPU state for a policy undergoing an upgrade. When you want to upgrade a policy, the agents in the new process that you launch attempt to attach to the existing enclave, waiting for the old agents running in the enclave to exit. Once the old agents exit, the new agents take over the enclave and begin scheduling.

Handling Scheduler Failures

ghOSt also recovers from scheduler failures (e.g., crashes, malfunctions, etc.) without triggering a kernel panic or machine reboot. To recover from a scheduler failure, you should generally destroy the failed scheduler's enclave and then launch the scheduler again. Destroying an enclave will kill the malfunctioning agents if necessary and will move the threads in the ghOSt scheduling class to CFS (Linux Completely Fair Scheduler) so that they can continue to be scheduled until you potentially pull them into ghOSt again.

To see all enclaves that currently exist in ghOSt, use ls to list them via ghostfs:

$ ls /sys/fs/ghost
ctl  enclave_1	version

To kill an enclave, such as enclave_1 above, run the following command, replacing enclave_1 with the name of the enclave:

echo destroy > /sys/fs/ghost/enclave_1/ctl

To kill all enclaves (which is generally useful in development), run the following command:

for i in /sys/fs/ghost/enclave_*/ctl; do echo destroy > $i; done