RUM-18135: Add a cold-start benchmarking harness and methodology guide

README.md

@@ -106,6 +106,10 @@ If you encounter any issue when using the Datadog SDK for Android and Android TV
 the [troubleshooting checklist][6], [common problems](docs/advanced_troubleshooting.md), or at
 the existing [issues](https://github.com/DataDog/dd-sdk-android/issues?q=is%3Aissue).
 
+If you are investigating the SDK's impact on your application's startup time, the
+[cold-start benchmarking guide](docs/benchmarking_sdk_cold_start.md) walks through measuring
+it on your own app, with the scripts we use internally.
+
 <div class="alert alert-warning">
 Datadog cannot guarantee the Android and Android TV SDK's performance on Roku devices running with Android OS. If you encounter any issues when using the SDK for these devices, contact <a href="https://docs.datadoghq.com/help/">Datadog Support</a> or open an issue in our GitHub project.
 </div>

tools/coldstart-benchmark/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.pyc
+results_*.csv
+bench_*.log
+*.pftrace
+.venv/

tools/coldstart-benchmark/fp_simulation.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+# Unless explicitly stated otherwise all files in this repository are licensed
+# under the Apache License Version 2.0.
+# This product includes software developed at Datadog (https://www.datadoghq.com/).
+# Copyright 2016-Present Datadog, Inc.
+"""
+Reproduces the false-positive table that justifies this harness's primary endpoint.
+
+The claim under test
+--------------------
+`ab_stats.py` analyses one delta per BLOCK and runs a paired test on those,
+rather than pooling every launch into an unpaired test. The reason is that
+launches inside one arm x block cell are not independent -- they share an
+install, an AOT compilation, a thermal state and a page-cache state -- so an
+unpaired test estimates the standard error from WITHIN-cell scatter only and
+ignores the BETWEEN-cell component. That makes it anti-conservative.
+
+"Anti-conservative" is easy to assert and easy to check, so this checks it.
+
+The model
+---------
+Launch time = 0 (no true effect, by construction) + a per-cell shift drawn from
+N(0, sigma_b) + per-launch noise from N(0, 11 ms). 11 ms is the pooled launch sd
+from the reference A/A run. sigma_b sweeps 0, 2, 4 and 8 ms; 4 ms is an ordinary
+between-block shift on a real device.
+
+Both designs then compute a nominal-95% interval and we count how often it
+excludes zero. A correct procedure lands at 5%.
+
+Usage:
+    ./fp_simulation.py                 # the published table
+    ./fp_simulation.py --trials 50000  # tighter Monte-Carlo error
+"""
+import argparse
+import math
+import random
+import statistics as st
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from ab_stats import t_crit, welch  # noqa: E402  (path set above)
+
+SD_WITHIN = 11.0
+SIGMA_B = (0, 2, 4, 8)
+
+
+def _cell(n_launches, shift, rng):
+    return [rng.gauss(shift, SD_WITHIN) for _ in range(n_launches)]
+
+
+def unpaired_rejects(n_blocks, n_launches, sigma_b, rng):
+    """Pool every launch, Welch t-interval. What most tools report."""
+    a, b = [], []
+    for _ in range(n_blocks):
+        a += _cell(n_launches, rng.gauss(0, sigma_b), rng)
+        b += _cell(n_launches, rng.gauss(0, sigma_b), rng)
+    d, se, df, _ = welch(a, b)
+    tc = t_crit(df)
+    return not (d - tc * se <= 0 <= d + tc * se)
+
+
+def paired_rejects(n_blocks, n_launches, sigma_b, rng):
+    """One delta per block, paired t-interval on those. What ab_stats.py reports.
+
+    Each ARM x BLOCK cell draws its OWN shift, matching the model: every cell gets
+    its own uninstall / install / AOT compile, so the thing that moves it is not
+    shared with the other arm in that block. Drawing one shift per block and
+    applying it to both arms would cancel it exactly in the delta, leaving sigma_b
+    with no effect at all -- the paired column would read ~5% for every sigma_b
+    whether or not the design actually worked, which proves nothing.
+    """
+    deltas = []
+    for _ in range(n_blocks):
+        deltas.append(st.mean(_cell(n_launches, rng.gauss(0, sigma_b), rng))
+                      - st.mean(_cell(n_launches, rng.gauss(0, sigma_b), rng)))
+    m = st.mean(deltas)
+    se = st.stdev(deltas) / math.sqrt(n_blocks)
+    tc = t_crit(n_blocks - 1)
+    return not (m - tc * se <= 0 <= m + tc * se)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--trials", type=int, default=20_000)
+    ap.add_argument("--seed", type=int, default=7)
+    args = ap.parse_args()
+    rng = random.Random(args.seed)
+
+    designs = (
+        ("2 blocks x 15 launches, unpaired", unpaired_rejects, 2, 15),
+        ("8 blocks x 4 launches, paired", paired_rejects, 8, 4),
+    )
+    # +-1 Monte-Carlo standard error on a ~5% rate, so a reader can tell a real
+    # difference from sampling noise without re-deriving it.
+    mc_se = 100 * math.sqrt(0.05 * 0.95 / args.trials)
+    print(f"true effect = 0, within-launch sd = {SD_WITHIN:g} ms, "
+          f"{args.trials} trials (MC SE at 5% ~ {mc_se:.2f}pp)")
+    print("false-positive rate of a nominal-95% interval\n")
+    print(f"{'design':36s}" + "".join(f"{f'sigma_b={s}':>12s}" for s in SIGMA_B))
+    for name, fn, nb, nl in designs:
+        rates = [100 * sum(fn(nb, nl, s, rng) for _ in range(args.trials)) / args.trials
+                 for s in SIGMA_B]
+        print(f"{name:36s}" + "".join(f"{r:11.1f}%" for r in rates))
+    print("\nA correct procedure sits at 5.0% in every column. The unpaired design")
+    print("does not, and the gap widens with the between-block shift.")
+
+
+if __name__ == "__main__":
+    main()

tools/coldstart-benchmark/lib.sh

@@ -0,0 +1,219 @@
+#
+# Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0.
+# This product includes software developed at Datadog (https://www.datadoghq.com/).
+# Copyright 2016-Present Datadog, Inc.
+#
+# Shared helpers for the cold-start harness. Source this, don't execute it.
+#
+# Resolves the Android tools WITHOUT depending on the caller's PATH, so the
+# harness works from cron, CI, an IDE terminal, or a shell whose profile has
+# not been fixed. Precedence:
+#   1. $ADB / $ANDROID_HOME / $ANDROID_SDK_ROOT if already set
+#   2. PATH
+#   3. well-known macOS/Linux SDK locations
+
+_dd_find_sdk() {
+  local c
+  for c in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" \
+           "$HOME/Library/Android/sdk" "$HOME/Android/Sdk" "$HOME/Android/sdk" \
+           "/usr/local/share/android-sdk" "/opt/homebrew/share/android-sdk"; do
+    [ -n "$c" ] && [ -x "$c/platform-tools/adb" ] && { echo "$c"; return 0; }
+  done
+  return 1
+}
+
+dd_resolve_tools() {
+  if [ -n "${ADB:-}" ] && [ -x "${ADB}" ]; then
+    :
+  elif command -v adb >/dev/null 2>&1; then
+    ADB="$(command -v adb)"
+  else
+    local sdk
+    sdk="$(_dd_find_sdk)" || {
+      echo "FATAL: cannot find adb. Set ADB=/path/to/adb or ANDROID_HOME." >&2
+      return 1
+    }
+    ANDROID_HOME="$sdk"
+    ADB="$sdk/platform-tools/adb"
+  fi
+
+  # aapt2 (optional): newest build-tools wins. Used by coldstart_bench.sh's
+  # preflight to read each APK's package name and version out of the manifest.
+  # Absent aapt2 downgrades that check to a warning rather than failing the run.
+  if [ -z "${AAPT2:-}" ]; then
+    local sdk bt
+    sdk="${ANDROID_HOME:-$(_dd_find_sdk 2>/dev/null || true)}"
+    if [ -n "$sdk" ] && [ -d "$sdk/build-tools" ]; then
+      # Glob + sort -V, never `ls`: `ls` output is at the mercy of the caller's
+      # environment. On this machine an `ls -aFh` alias makes it emit `37.0.0/`
+      # (and `.`/`..`), which produced a working-but-wrong `build-tools/37.0.0//aapt2`.
+      # Ordering must be by VERSION anyway -- lexically, "9.0.0" beats "10.0.0".
+      bt=$(printf '%s\n' "$sdk"/build-tools/*/ | sed 's:/*$::; s:.*/::' | sort -V | tail -1)
+      [ -n "$bt" ] && [ -x "$sdk/build-tools/$bt/aapt2" ] && AAPT2="$sdk/build-tools/$bt/aapt2"
+    fi
+  fi
+
+  export ADB AAPT2 ANDROID_HOME
+  return 0
+}
+
+# Name of the currently resumed activity, as "pkg/component".
+#
+# The obvious `grep mResumedActivity` does NOT work everywhere: Android 12 on
+# some devices (observed on a Motorola moto g60s, SDK 31) prints the record as
+# `ResumedActivity:` inside the Task dump with no `m` prefix, so the anchored
+# pattern matches zero lines and every foreground assertion reports OTHER.
+# Match both spellings, and fall back to the window manager's focused app.
+dd_top_activity() {
+  local top
+  top=$("$ADB" shell dumpsys activity activities 2>/dev/null \
+        | grep -m1 -E '(^|[^a-zA-Z])m?ResumedActivity[:=]' \
+        | grep -oE '[a-zA-Z0-9_.]+/[a-zA-Z0-9_.]+' | head -1 | tr -d '\r') || true
+  if [ -z "$top" ]; then
+    top=$("$ADB" shell dumpsys window 2>/dev/null \
+          | grep -m1 'mFocusedApp=' \
+          | grep -oE '[a-zA-Z0-9_.]+/[a-zA-Z0-9_.]+' | head -1 | tr -d '\r') || true
+  fi
+  printf '%s' "$top"
+}
+
+# Refuse to measure behind a lockscreen. A locked device still RESUMES the
+# activity -- `ResumedActivity` names the app correctly -- but no frame is ever
+# drawn, so `am start -W` reports no TotalTime and LaunchState=UNKNOWN. Without
+# this check the run dies later with a misleading "not fully reaped" message.
+dd_require_unlocked() {
+  local locked focus attempt
+  # The notification shade and the always-on display are not locks -- they are
+  # just windows sitting on top, and either can be left behind by an earlier
+  # `input` command or by a notification arriving between runs. Clear them and
+  # look again before failing, otherwise an unattended sequence of captures
+  # loses every run after the first stray swipe.
+  for attempt in 1 2; do
+    locked=$("$ADB" shell dumpsys trust 2>/dev/null | grep -oE 'deviceLocked=[01]' | head -1 | cut -d= -f2 | tr -d '\r') || true
+    focus=$("$ADB" shell dumpsys window 2>/dev/null | grep -m1 mCurrentFocus | tr -d '\r') || true
+    case "$focus" in
+      *Keyguard*|*NotificationShade*|*AOD*|*DreamActivity*) locked=1 ;;
+    esac
+    { [ "${locked:-0}" = "1" ] && [ "$attempt" = 1 ]; } || break
+    "$ADB" shell cmd statusbar collapse >/dev/null 2>&1 || true
+    "$ADB" shell input keyevent KEYCODE_WAKEUP >/dev/null 2>&1 || true
+    "$ADB" shell wm dismiss-keyguard >/dev/null 2>&1 || true
+    sleep 2
+  done
+  if [ "${locked:-0}" = "1" ]; then
+    echo "FATAL: the device is locked, or the lockscreen/shade is on top." >&2
+    echo "  A locked device still resumes the activity, but never draws a frame, so" >&2
+    echo "  'am start -W' returns no TotalTime and LaunchState=UNKNOWN. Every launch" >&2
+    echo "  would be unmeasurable." >&2
+    echo "  Fix: unlock the phone by hand and leave it on the home screen. If it has a" >&2
+    echo "  PIN/pattern/password, adb cannot dismiss it ('wm dismiss-keyguard' only" >&2
+    echo "  works for a swipe-only lock)." >&2
+    echo "  Current focus: ${focus:-unknown}" >&2
+    return 1
+  fi
+  return 0
+}
+
+# A Python that can actually import `perfetto`. The trace scripts need it, and the
+# usual failure is invisible: `verify_trace.py`'s shebang is `#!/usr/bin/env python3`,
+# i.e. the SYSTEM interpreter, while the documented install puts `perfetto` in a
+# local venv. Prefer the venv next to the scripts, then $PY, then anything on PATH
+# that can import it.
+dd_resolve_python() {
+  local here c
+  here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+  for c in "${PY:-}" "$here/.venv/bin/python" "$here/.venv/bin/python3" \
+           "$(command -v python3 2>/dev/null)" "$(command -v python 2>/dev/null)"; do
+    [ -n "$c" ] && [ -x "$c" ] || continue
+    if "$c" -c 'import perfetto' >/dev/null 2>&1; then PY="$c"; export PY; return 0; fi
+  done
+  echo "FATAL: no Python with the 'perfetto' package. Install it, then re-run:" >&2
+  echo "    python3 -m venv $here/.venv && $here/.venv/bin/pip install perfetto" >&2
+  echo "  (verify_trace.py's shebang is the system python3, which will not see a venv," >&2
+  echo "   so pass the venv interpreter explicitly when running it by hand.)" >&2
+  return 1
+}
+
+# md5 of a local file, portable between macOS and Linux.
+dd_md5() {
+  [ -f "$1" ] || { echo "FATAL: file not found: $1" >&2; return 1; }
+  if command -v md5 >/dev/null 2>&1; then md5 -q "$1"
+  else md5sum "$1" | awk '{print $1}'; fi
+}
+
+# Fail unless exactly one device is attached AND authorized. Distinguishes the
+# common failure modes instead of reporting a generic "no device".
+dd_require_device() {
+  local out n_auth n_unauth n_off
+  out="$("$ADB" devices | tail -n +2 | grep -v '^$' || true)"
+  # Honour ANDROID_SERIAL, which adb itself respects: a Linux CI box or a
+  # workstation with an emulator running alongside a phone is a normal setup.
+  if [ -n "${ANDROID_SERIAL:-}" ]; then
+    if printf '%s\n' "$out" | grep -qE "^${ANDROID_SERIAL}[[:space:]]+device$"; then
+      return 0
+    fi
+    echo "FATAL: ANDROID_SERIAL='$ANDROID_SERIAL' is not attached and authorized." >&2
+    printf '%s\n' "$out" >&2
+    return 1
+  fi
+  n_auth=$(echo "$out"   | grep -cE '[[:space:]]device$'       || true)
+  n_unauth=$(echo "$out" | grep -cE '[[:space:]]unauthorized$' || true)
+  n_off=$(echo "$out"    | grep -cE '[[:space:]]offline$'      || true)
+
+  if [ "$n_unauth" -gt 0 ]; then
+    echo "FATAL: device attached but UNAUTHORIZED." >&2
+    echo "  Unlock the phone; accept the 'Allow USB debugging?' prompt and tick" >&2
+    echo "  'Always allow from this computer'. If no prompt appears:" >&2
+    echo "    $ADB kill-server && $ADB start-server && $ADB devices" >&2
+    echo "  (or revoke old keys: Developer options -> Revoke USB debugging authorisations)" >&2
+    return 1
+  fi
+  [ "$n_off" -eq 0 ] || { echo "FATAL: device is offline; replug the cable." >&2; return 1; }
+  if [ "$n_auth" -ne 1 ]; then
+    echo "FATAL: need exactly one authorized device, found $n_auth." >&2
+    echo "  Set ANDROID_SERIAL=<serial> to pick one." >&2
+    "$ADB" devices -l >&2
+    return 1
+  fi
+  return 0
+}
+
+# Put the radios into the requested state and PROVE they got there.
+#
+# `svc` is best-effort and silently no-ops without root -- `svc data disable` needs
+# root on most retail devices -- so without a read-back a run can be stamped
+# airplane=1 while mobile data is still up. That is not merely noisy, it is
+# mislabelled, and uncomparable against a genuinely offline run.
+#
+# Shared because BOTH the benchmark and the trace capture need the same network
+# condition: a trace taken online cannot explain a delta measured offline.
+# $1 = 1 for offline, 0 for online. Only offline is enforced; in online mode a radio
+# that will not come up is reported, since the run is still "online" as labelled.
+dd_apply_radio_state() {
+  local want="$1"
+  if [ "$want" = 1 ]; then
+    "$ADB" shell svc wifi disable >/dev/null 2>&1 || true
+    "$ADB" shell svc data disable >/dev/null 2>&1 || true
+  else
+    "$ADB" shell svc wifi enable >/dev/null 2>&1 || true
+  fi
+  sleep 2
+  local w d
+  w=$("$ADB" shell settings get global wifi_on 2>/dev/null | tr -d '\r') || true
+  d=$("$ADB" shell settings get global mobile_data 2>/dev/null | tr -d '\r') || true
+  if [ "$want" = 1 ]; then
+    case "$w" in 1) echo "FATAL: AIRPLANE=1 but Wi-Fi is still on ('svc wifi disable' had no" >&2
+       echo "       effect -- it needs root on many devices). Refusing to record a run" >&2
+       echo "       stamped airplane=1 that was not offline. Turn Wi-Fi off by hand." >&2
+       return 1 ;; esac
+    case "$d" in 1) echo "FATAL: AIRPLANE=1 but mobile data is still on ('svc data disable'" >&2
+       echo "       needs root on most retail devices). Refusing to record a run stamped" >&2
+       echo "       airplane=1 that was not offline. Disable it by hand." >&2
+       return 1 ;; esac
+    echo "[radios verified OFF (wifi_on=$w mobile_data=$d)]" >&2
+  else
+    echo "[radios: wifi_on=${w:-unknown} mobile_data=${d:-unknown}]" >&2
+    case "$w" in 0) echo "[NOTE: Wi-Fi did not come up; device may be on mobile data or offline.]" >&2 ;; esac
+  fi
+  return 0
+}