AndroidEnv - The Android Interaction & Learning Platform

AndroidEnv is a Python and Kotlin platform for interacting with Android devices (both virtualized emulators and physical hardware). Built for foundation model UI agents, automated testing, and Reinforcement Learning (RL) research, AndroidEnv provides low-latency device telemetry and control over any Android application.

In AndroidEnv v2, device interaction is decoupled from Reinforcement Learning task definitions:

• AndroidDevice (Direct Device Control & Telemetry): A task-free handle that connects to an on-device Kotlin Core server over WebSockets to stream screenshots, accessibility trees, system logs, and sensor data while injecting touch gestures, key events, text, and ADB commands.
• AndroidEnv & TaskWrapper (Reinforcement Learning): An optional dm_env layer on top of AndroidDevice that adds Markov Decision Process (MDP) semantics (step(), reset(), rewards, and episode terminations) via task definitions.
• AndroidEnv Playground: A responsive web workbench for live screen interaction, accessibility tree inspection, and ADB command execution.

Index

• Migrating from v1 to AndroidDevice (v2)
• Architecture & environment details
• Running AndroidDevice, Playground, and RL agents
• Setting up a virtual Android device
• Defining an RL task in AndroidEnv
• Example RL tasks available for download

Platform Features

• Real-World Android Ecosystem: Agents interact with an unaltered Android OS—on Android Studio Emulators or physical devices—exposing the exact same apps, UI frameworks, and system services that billions of human users interact with daily.
• Rich, Subscription-Based Telemetry (DeviceState): Subscribe on demand to the signals your agent needs:
  • Screenshots: High-speed, decoded RGB frame buffers (width, height, raw_pixels).
  • Accessibility Trees: Full AndroidAccessibilityForest window and UI node hierarchies streamed directly from the on-device AccessibilityService.
  • System & Sensor Streams: Foreground package/activity, logcat streams, device settings, GPS location, audio output, and IMU sensors (accelerometer, gyroscope, magnetometer, barometer, light, proximity).
• Expressive Action Interface (Action):
  • Touch & Multi-Touch Gestures: Normalized (x, y) coordinates in [0.0, 1.0] with TOUCH_DOWN, TOUCH_MOVE, and TOUCH_UP across multiple pointer_id fingers for taps, swipes, scrolls, and pinches.
  • Hardware Keys & Native Text Entry: Inject Android keycodes (BACK, HOME, ENTER), set text directly on focused UI nodes (ACTION_TYPE_SET_TEXT), or trigger clipboard actions (CUT, COPY, PASTE).
  • Direct ADB Execution: Run structured or generic AdbRequest commands via device.execute_adb_call().
• Composable RL Layer: When training RL policies, wrap AndroidDevice with android_env.rl.environment.AndroidEnv and TaskWrapper to use custom .textproto tasks, reward parsers, and dm_env / Gymnasium wrappers.

Getting Started

Installation

Install AndroidEnv from PyPI:

$ python3 -m pip install android-env

Or clone the repository from main and build/test with Bazel:

$ git clone https://github.com/google-deepmind/android_env/
$ cd android_env
$ bazel test //...

To build the PyPI wheel locally using Bazel:

$ bazel build //:android_env_wheel

1. Set Up an Android Device

Connect a physical Android device via USB debugging (adb devices) or create an Android Virtual Device (AVD) using Android Studio. See the Emulator guide for step-by-step instructions.

2. Direct Control with AndroidDevice

For UI agents, foundation model evaluations, and automation scripts that do not need RL episode loops:

import numpy as np
from android_env.components import android_device_loader
from android_env.components import config_classes
from android_env.proto import device_env_service_pb2

config = config_classes.AndroidDeviceConfig(serial='emulator-5554')
with android_device_loader.load_android_device(config) as device:
  # Subscribe to screenshots and accessibility trees:
  device.update_subscriptions({
      device_env_service_pb2.DEVICE_SIGNAL_SCREENSHOT,
      device_env_service_pb2.DEVICE_SIGNAL_ACCESSIBILITY_TREE,
  })

  # Read the latest device state:
  state = device.get_state()

  # Tap the center of the screen:
  device.send_action(
      device_env_service_pb2.Action(
          action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_DOWN,
          touch_position=device_env_service_pb2.Point(x=0.5, y=0.5),
      )
  )
  device.send_action(
      device_env_service_pb2.Action(
          action_type=device_env_service_pb2.ACTION_TYPE_TOUCH_UP,
      )
  )

3. Launch the Interactive Web Playground

Explore the device screen, inspect live accessibility trees, and run ADB commands from your browser:

$ bazel run //android_env/playground:launch -- \
    --config=android_env/configs/emulator_config.py

4. Reinforcement Learning (dm_env)

To define MDP tasks with rewards and episode resets, see the Tasks guide, Example tasks, and detailed instructions.

About

This library is developed and maintained by Google DeepMind. You can find the original technical report on arXiv, as well as the introductory blog post.

If you use AndroidEnv in your research, you can cite the paper using the following BibTeX:

@article{ToyamaEtAl2021AndroidEnv,
  title     = "{AndroidEnv}: A Reinforcement Learning Platform for Android",
  author    = {Daniel Toyama and Philippe Hamel and Anita Gergely and
               Gheorghe Comanici and Amelia Glaese and Zafarali Ahmed and Tyler
               Jackson and Shibl Mourad and Doina Precup},
  year      = {2021},
  eprint    = {2105.13231},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  volume    = {abs/2105.13231},
  url       = {http://arxiv.org/abs/2105.13231},
}

Disclaimer: This is not an official Google product.