Vahag Byurat  ·  All Projects
Appium · Kubernetes · Streaming QA at Scale

Distributed Device Farm

A cross-platform test grid for streaming clients: one page-object layer drives web, phones and TVs, a capability-as-config matrix decides where each suite runs, and the grid reaches past localhost to real hardware sitting on another network entirely.

Java 21 · JUnit 5 Appium 3 · Playwright · Selenium 4 Toxiproxy · ABR Kubernetes · Argo
The Problem
A streaming client is not one program: it is the same program on a dozen screens

The same release ships to a browser, a phone, a tablet, a set-top box and three flavours of smart TV, and it fails differently on each. A television has no pointer, only a focus engine and a D-pad. A phone suspends your process mid-playback. A browser gives you developer tooling that no TV will ever give you. Coverage means running against all of them, which turns a testing problem into an infrastructure problem: you need somewhere for all those devices to live, and a way for one test run to reach them.

That is what this project is. A grid, a device matrix, and one page-object layer thin enough that the same test body survives the jump from Chrome to a living-room TV. Everything else here (the mock service, the streams, the fault injection) exists so the grid has something real to prove itself against.

Where the devices actually are

Three kinds of place, and a test body cannot tell them apart: containers in a Kubernetes cluster, simulators on the machine running the suite, and real hardware on a completely different network, reached over a mesh VPN. Each was proven on its own run; what they share is that execution location is configuration, not code.


The System Under Test
To test a streaming app properly, you first need a streaming app

Every interesting question in streaming QA (does adaptation actually down-switch under load, does the join time hold, does the same page object survive a jump from a browser to a TV remote) needs a real player on a real screen to ask. Public demo streams and headless stubs answer none of them.

So the app came first: Vahagmount++, a deliberately fake streaming service named after its author and incremented once, because the interesting part was never the streaming. It exists in five flavours, each honouring the same accessibility contract so that the same page objects (and in most cases the same test bodies) drive all of them. Three of the five are built without their conventional build system on purpose: no Gradle for Android, and no .xcodeproj for either Apple build. That keeps the toolchain to an SDK and a compiler, and it builds in constrained environments where a Gradle daemon's loopback sockets are blocked.

Build What it is How it's built
mock-app Web client: HLS.js and Shaka behind one engine abstraction, live window.__qoe telemetry static files
mock-app-android Native APK: login, home, player javac → d8 → apksigner
mock-app-ios SwiftUI iPhone app bare swiftc
mock-app-tvos SwiftUI Apple TV app, focus-engine navigation bare swiftc
mock-app-webos LG webOS .ipk repackaging the web client ares packaging
The service is fake. The failure modes are not.

The Keystone
One PlayerPage, five platforms, zero branches

The usual way a cross-platform suite rots is triplication: a WebLoginPage, an AndroidLoginPage, a TvLoginPage, and three copies of every flow that walks through them. The alternative most teams reach for, if (platform == …) inside the page object, is the same rot with worse ergonomics.

The fix here is to move the platform decision inside the locator. ByPlatform extends Selenium's own By, so it drops into driver.findElement(...), every BasePage helper and ExpectedConditions with no special handling. Resolution is most-specific-wins, and the platform comes from a ThreadLocal the driver factory binds at session creation, so the page object never receives or threads a platform argument at all.

private static final ByPlatform SCREEN = ByPlatform.builder() .web(testId("player-screen")) .android(AppiumBy.id("com.vahagmountplus:id/player_root")) .ios(AppiumBy.accessibilityId("playerScreen")) .tv(AppiumBy.accessibilityId("player-screen")) // any TV-family platform .build();
one PlayerPage · one test body no per-platform page objects ByPlatform.resolve(platform): most-specific-wins exact platform Android-flavoured family fallback WEB data-testid HLS.js/Shaka web family ANDROID resource-id signed APK exact iOS accessibility id · SwiftUI exact tvOS accessibility id · D-pad TV family webOS accessibility id · .ipk TV family
Teal targets were given an exact locator; amber ones inherited a family locator they never had to declare.
The branch that earned its own test class

Android TV and Fire TV belong to the TV family but are Android underneath, and they run the same APK as the phone, so a plain family lookup would hand them the accessibility-id locators meant for webOS and tvOS. The resolver therefore inserts one step: an Android-flavoured target with no TV-specific locator prefers the ANDROID exact locator over the TV family. Nine unit tests pin the resolver down, including the cases that matter most: that an explicit androidTv() locator still beats that fallback, and that Apple TV and webOS are not dragged into the Android branch.

What the page objects deliberately do not use

No implicit waits anywhere: implicit and explicit waits compound unpredictably and mask races rather than fixing them, so every find, click and type is a bounded WebDriverWait. No @FindBy / PageFactory either: its lazy element proxies are a well-known source of StaleElementReferenceException, and they do not compose with a polymorphic locator in the first place.


Proving Adaptation
A down-switch you can force, not one you hope to catch

Most ABR "tests" watch a stream and hope adaptation happens. That is a coin-flip dressed as an assertion. Here the ladder is built to make the outcome deterministic: one media corpus is encoded to three well-separated rungs (240p at roughly 250 kbps, 480p at 600, 720p at 1.1 Mbps), and the throttle is set at about 400 kbps, deliberately parked between the floor and the 480p rung. Only one rung fits. A player that does not drop to it has failed.

throttle ≈ 400 kbps 720p 1.1 Mbps beyond the pipe 480p 600 kbps over budget 240p 250 kbps the only rung that fits DASH · HLS · CMAF all carry this ladder, and all three are asserted
The same media, packaged three ways, throttled at the same point, asserted with the same code.

There are two throttles in the suite. The plain Playwright lane shapes bandwidth through Chrome DevTools, which is fine for a browser on the same machine; the lane that proves the mechanism generalises applies the same shaping at the network layer through Toxiproxy. That second choice is the load-bearing one: CDP's setNetworkConditions cannot reach a browser running on a Grid node or a phone on the other end of an Appium session, whereas a proxy in front of the origin shapes bytes for anything downstream of it. The framework spawns the native toxiproxy-server binary itself over ProcessBuilder and blocking HTTP, so no Docker and no cluster is required to run the lane, and the identical conditioner drives an in-cluster Toxiproxy with only an admin URL changed.

What gets asserted is not "the video element exists" but the telemetry the player publishes: window.__qoe carries video start time, rebuffer count and ratio, ladder position, bitrate-switch history and dropped frames (deliberately the vocabulary Conviva, MUX and NPAW expose for real). Both browser engines write the same object, so a CMAF stream played through Shaka and an HLS stream played through HLS.js are checked by literally the same assertion code.

An up-switch the suite refuses to assert

Symmetry would suggest also asserting a mid-stream up-switch once the pipe widens. It is not asserted, because Shaka's estimator only samples bandwidth on a segment fetch: the first load measures a cold network while later loads see warm, near-instant transfers it discards. Reliable for HLS.js, flaky for Shaka. The down-switch is deterministic and a fast-pipe ceiling check covers the other direction. An assertion that is sometimes true is worse than one that isn't there.


CI Honesty
A suite that is never falsely red

A device farm's tests run on machines that mostly do not have the devices. The failure mode this creates, red builds that mean "no Fire TV plugged into this agent", trains everyone to ignore the colour. So absence is modelled explicitly rather than handled by accident.

Missing hardware skips, never fails

The Appium base class starts a session inside a try and calls Assumptions.abort when one cannot be created, so a device-backed test on a bare agent is skipped with the reason attached. Green when devices are present, cleanly skipped when they are not.

Adding a device is adding a file

Sixteen tagged JSON capability files describe the fleet. DeviceMatrix assembles each test's device list by tag (mobile, tv, android-tv, web), so a new device joins every relevant matrix with no test edit and no recompile of a test body. A run narrows with -Ddevices=… for CI sharding.

Where a test runs is configuration

Local, Selenium Grid, a remote Appium over a mesh VPN, or a cloud lab are interchangeable through TestEnvironment, appium.server.url and web.hub.url. The same MobilePlaybackTest ran green on an iPhone simulator, an Android emulator matrix and a real Pixel over USB; a living-room Android TV on network adb ran the CTV lane against the same page objects and the same APK.

Counting tests honestly

The Toxiproxy lanes gate in a class-level @BeforeAll assumption, so without the binary they are never started: they leave the total rather than appear as skips. -Dgroups=pw,api is 39 green with the binary on PATH and 25 without, and both numbers are correct. Saying so is cheaper than explaining a moving total later.

39Green in the
web + API lane
5App builds,
one contract
16Device configs,
zero test edits
9Unit tests pinning
locator resolution

Reach
Devices that live somewhere else

A grid confined to one machine can only ever test what fits on that machine. The interesting hardware (a real living-room television, a physical phone) tends to be somewhere else, on a different network, behind NAT. Commercial device clouds solve this by owning the hardware; the same shape is buildable from a mesh VPN and an Appium server.

The orchestrating laptop joins a WireGuard mesh with a machine at another location. That machine runs Appium against the hardware physically attached to it (an Android TV over network ADB, a phone over USB) while the laptop contributes its own iOS and tvOS simulators locally. One run dispatches to all of them.

Why it needed no new code

Because remoteness was never a property of the test. A suite already addresses its Appium server by URL and its device by capability file, so a node across the internet differs from a node on localhost by a hostname. The capability files for the real TV and phone were reused unchanged; what the mesh VPN added was reachability, not a new execution model.

What that costs

Honesty about the trade: this is a proof of concept, not a hardened farm. The remote machine has to be awake, its Appium server up, and the TV on. There is no queue, no lease, no reclaim of a wedged device. It demonstrates the topology and the config-not-code claim; it does not survive a team pointing CI at it.


Scale-Out
The farm, as manifests

The orchestration is real rather than illustrative: a kind cluster runs a Selenium Grid 4 hub with Chrome nodes, the mock app, and a Toxiproxy pod, with an Argo Workflows DAG fanning out one pod per shard and joining on an aggregate step. Nine web tests ran in-cluster, the proxy throttle drove a 240p down-switch through the Grid where CDP could not reach, and the Argo fan-out finished four of four. A KEDA ScaledObject that autoscales nodes off the Grid's session queue is written but not yet driven hard enough to watch it scale. It sits on the backlog, labelled as such.

For anyone who just wants the green lane, there is a self-contained image: a Temurin 21 Maven base with real Google Chrome installed, because Playwright's bundled Chromium ships without H.264 and AAC and therefore cannot play the HLS corpus at all. That gotcha costs a video-QA team standardising on Playwright an afternoon; here it is a line in a Dockerfile and a note in the README.


Limits
What this does not prove

The repository labels every claim as proven or compiles, and the gap between the two is where the honest reading lives.

  • Roku, Samsung Tizen and Fire TV are configured, not proven. Capability files, driver paths and skip behaviour exist; the hardware never reached the desk. Roku is modelled as non-Appium (it has its own WebDriver) rather than faked into the Appium path.
  • webOS runs but does not pair. The .ipk installs and launches on a webOS OSE emulator under QEMU/KVM; LG's SSAP remote-pairing step is a commercial-TV feature the open-source edition lacks.
  • The apps are stand-ins. They honour the accessibility contract a real client would expose, but they are not production streaming clients, and DRM is a contract field rather than real encryption.
  • Device-lane buffering is untested. The Appium stand-in apps play a local file rather than a network stream, so ABR and rebuffer behaviour are proven on web only. A streaming build of the native apps is the precondition, and it is written down as one.
  • The buffering lane is wall-clock sensitive. It waits on real ABR recovery through a shaped proxy, so on a loaded machine it can exceed its 30-second budget and error. The count is stable; the timing is not.
  • One test clip. The catalogue lists several titles. They all play the same ladder.

The service the farm tests (Vahagmount++, named after its author and then incremented, because the streaming was never the interesting part) is fictional, built for QA practice and released for educational use only. Every catalogue entry in it, Vishap Wars: Kernel Panic, Failover, Tomato Dragon Cluster and the rest, is an invented placeholder written for test fixtures; none of them refers to a real programme, studio or provider.