Repository navigation
[doc] Add an Android section to reducing_variance - #2356
Conversation
reducing_variance.md is written for desktop and server. On Android, cpupower is absent, scaling_governor is often not accessible to the shell user, there is no BIOS in which to disable SMT, and the effects that dominate variance are not mentioned. The first item is the one that caught me. Android compiles installed applications ahead of time in a background job gated on the device being idle and charging, which is exactly the state you create in order to benchmark it. On a Galaxy M21 (Exynos 9611, Android 12), the same memory-bandwidth microbenchmark with the same arguments and the same power state read 6.35 GB/s while that job was running and 9.32 GB/s once the device was quiet, 32% lower. Both runs showed 15% repetition-to-repetition variation, so a check based on the spread between repetitions does not see it: the job runs for the whole measurement and depresses every repetition by a similar amount. The job runs at nice 10, which yields the CPU but not memory bandwidth or cache. The other five items are the ones that cost me a measurement each: measuring quietness instead of trusting load average, which reads 19 to 21 on a device whose busy fraction is 3%; recording the device power mode without assuming a direction, since it can leave scaling_max_freq untouched; letting the device settle after a reboot; not using every core; and treating battery temperature with suspicion, since on a charging device it tracks the battery rather than the SoC and fell steadily through six minutes of all-core compute. Appended at the end of the file so the existing "Further resources" list stays attached to the section it belongs to.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
@googlebot I signed it! |
The command I suggested, ps -A -T across every thread, is itself heavy enough on a device with a few thousand threads to leave adbd unresponsive. That happened twice while gathering these numbers, and recovering needed a physical replug, which is a poor way to find out your device is busy. The compiler threads belong to dex2oat, so matching the process with pgrep is sufficient and costs nothing.
| Most of the advice above assumes a desktop or server. On an Android device little of it | ||
| applies: there is no `cpupower`, `scaling_governor` is often not readable or writable by the |
There was a problem hiding this comment.
maybe instead of this paragraph we can just add something to the above that makes it clear it's server/desktop, then this section stands alone
There was a problem hiding this comment.
Done. Dropped the paragraph and put the scoping in the Reducing Variance in Benchmarks intro instead, which now says it assumes a desktop or server and links down here. Added an anchor so the link matches the style of the existing ones.
| ```sh | ||
| adb shell pgrep dex2oat | ||
| ``` | ||
| If that prints anything, wait. `pgrep` is deliberate here: listing every thread with |
There was a problem hiding this comment.
is there anything programmatic that a user might be able to do?
There was a problem hiding this comment.
Yes, and I think you are right that it belongs here rather than in prose.
You cannot stop the job on a retail device, but you can wait for it, which is the part a harness should own rather than leaving the operator to remember. I added a snippet that blocks until the device has been quiet for three consecutive checks, combining the dex2oat check with the busy-fraction check from item 2. It reads /proc/stat deltas rather than load average, because on Android load average counts threads blocked in D state and read 19 to 21 on a device whose measured busy fraction was 3%.
Tested on the device the numbers came from: it returns in about 10 s when the phone is idle, and blocks for the full window with eight spinners running.
There was a problem hiding this comment.
i meant that we should recommend something for a "main" that could be added to an android benchmark binary, but i guess permissions wouldn't allow for that.
…id wait programmatic Two review comments. The contrast paragraph that opened the Android section is gone. Instead the 'Reducing Variance in Benchmarks' intro now says it assumes a desktop or server and points at the Android section, which stands alone. Added an anchor so the link works the same way as the existing ones. For 'is there anything programmatic that a user might be able to do': you cannot stop the dexopt job on a retail device, but you can wait for it rather than leaving the operator to remember. Added a snippet that blocks until the device has been quiet for three consecutive checks, combining the dex2oat check with the busy-fraction check from item 2. Tested on a Galaxy M21: returns in 10 s on a quiet device, and blocks for the full window when eight spinners are running.
|
if the CLA check fails again it's because "noreply@anthropic.com" is a contributor and they can't sign the CLA. i need to figure out internally how i can handle this case. |
72b2aaf to
2eb4e07
Compare
|
thank you! |
docs/reducing_variance.mdis written for desktop and server. On Android,cpupoweris absent,scaling_governoris often not accessible to theshelluser, there is no BIOS in which to disable SMT, and the effects that dominate variance there are not mentioned.The first item is the one that caught me, and it is why I bothered writing this up.
Android compiles installed applications ahead of time in a background job gated on the device being idle and charging, which is exactly the state you create in order to benchmark it. On a Galaxy M21 (Exynos 9611, Android 12), the same memory-bandwidth microbenchmark with the same arguments in the same power state read 6.35 GB/s while that job was running and 9.32 GB/s once the device was quiet, so the contaminated figure was 32% low.
What makes it worth documenting rather than just avoiding: both runs showed 15% repetition-to-repetition variation, so a check based on the spread between repetitions does not catch it. The job runs for the whole measurement and depresses every repetition by a similar amount. It also runs at
nice10, so it yields the CPU readily and looks harmless, but niceness does not yield memory bandwidth or cache.The other five items each cost me a measurement:
/proc/statrather than trusting load average, which read 19 to 21 on a device whose measured busy fraction was 3%, because Android's load average counts threads blocked inDstate;settings put global low_power 1can leavescaling_max_freqat its full value;I could not find an existing issue or PR on measurement hygiene for Android in this repo. The section is appended at the end of the file so the existing "Further resources" list stays attached to the section it belongs to.
Happy to cut this down if it is more than the file wants, or to drop the specific numbers and keep only the advice.
🤖 Generated with Claude Code