A streaming service may target a browser, a phone, a tablet, a set-top box and several flavors of smart TV, and it fails differently on each. A television has no pointer, only a focus engine and a D-pad. A phone suspends your process mid-playback. A browser gives you developer tooling that no TV will ever give you. Coverage means running against all of them, which turns a testing problem into an infrastructure problem: you need somewhere for all those devices to live, and a way for one test run to reach them.
This project combines a grid, a device matrix, and a page-object layer that shares test flows where the client behavior is equivalent. The mock service, streams, and fault injection provide a controlled system under test.
The available targets include containers in a Kubernetes cluster, simulators on the test machine, and real hardware reached over a mesh VPN. Each path was exercised in a separate run. Execution location is selected through configuration rather than changes to the test body.
Every interesting question in streaming QA (does adaptation actually down-switch under load, does the join time hold, does the same page object survive a jump from a browser to a TV remote) needs a real player on a real screen to ask. Public demo streams and headless stubs answer none of them.
The system under test is Vahagmount++, a fictional streaming service implemented in five client variants. Each exposes the same accessibility contract so the page objects, and most test bodies, can be shared. The Android and Apple stand-ins also have compiler-only build paths for constrained environments where the conventional project tooling is unavailable.
| Build | What it is | How it's built |
|---|---|---|
| mock-app | Web client: HLS.js and Shaka behind one engine abstraction, live window.__qoe telemetry |
static files |
| mock-app-android | Native APK: login, home, player | javac → d8 → apksigner |
| mock-app-ios | SwiftUI iPhone app | bare swiftc |
| mock-app-tvos | SwiftUI Apple TV app, focus-engine navigation | bare swiftc |
| mock-app-webos | LG webOS .ipk repackaging the web client |
ares packaging |
The usual way a cross-platform suite rots is triplication: a
WebLoginPage, an AndroidLoginPage, a
TvLoginPage, and three copies of every flow that walks through them.
The alternative most teams reach for, if (platform == …) inside
the page object, is the same rot with worse ergonomics.
The fix here is to move the platform decision inside the locator.
ByPlatform extends Selenium's own By, so it drops into
driver.findElement(...), every BasePage helper and
ExpectedConditions with no special handling. Resolution is
most-specific-wins, and the platform comes from a ThreadLocal the
driver factory binds at session creation, so the page object never receives
or threads a platform argument at all.
Android TV and Fire TV belong to the TV family but are Android underneath, and
they run the same APK as the phone, so a plain family lookup would hand
them the accessibility-id locators meant for webOS and tvOS. The resolver
therefore inserts one step: an Android-flavored target with no TV-specific
locator prefers the ANDROID exact locator over the TV family.
Nine unit tests pin the resolver down, including the cases that matter most:
that an explicit androidTv() locator still beats that
fallback, and that Apple TV and webOS are not dragged into the
Android branch.
No implicit waits anywhere: implicit and explicit waits compound unpredictably
and mask races rather than fixing them, so every find, click and type is a
bounded WebDriverWait. No @FindBy / PageFactory
either: its lazy element proxies are a well-known source of
StaleElementReferenceException, and they do not compose with a
polymorphic locator in the first place.
Most ABR "tests" watch a stream and hope adaptation happens. That is a coin-flip dressed as an assertion. Here the ladder is built to make the outcome deterministic: one media corpus is encoded to three well-separated rungs (240p at roughly 250 kbps, 480p at 600, 720p at 1.1 Mbps), and the throttle is set at about 400 kbps, deliberately parked between the floor and the 480p rung. Only one rung fits. A player that does not drop to it has failed.
There are two throttles in the suite. The plain Playwright lane shapes bandwidth
through Chrome DevTools, which is fine for a browser on the same machine; the lane
that proves the mechanism generalises applies the same shaping at the
network layer through Toxiproxy. That second choice is the load-bearing one: CDP's
setNetworkConditions cannot reach a browser running on a Grid node or
a phone on the other end of an Appium session, whereas a proxy in front of the
origin shapes bytes for anything downstream of it. The framework spawns the native
toxiproxy-server binary itself over ProcessBuilder and
blocking HTTP, so no Docker and no cluster is required to run the lane, and the
identical conditioner drives an in-cluster Toxiproxy with only an admin URL changed.
What gets asserted is not "the video element exists" but the telemetry the player
publishes: window.__qoe carries video start time, rebuffer count and
ratio, ladder position, bitrate-switch history and dropped frames
(deliberately the vocabulary Conviva, MUX and NPAW expose for real). Both browser
engines write the same object, so a CMAF stream played through Shaka and an HLS
stream played through HLS.js are checked by literally the same assertion code.
Symmetry would suggest also asserting a mid-stream up-switch once the pipe widens. It is not asserted, because Shaka's estimator only samples bandwidth on a segment fetch: the first load measures a cold network while later loads see warm, near-instant transfers it discards. Reliable for HLS.js, flaky for Shaka. The down-switch is deterministic and a fast-pipe ceiling check covers the other direction. The unstable up-switch is documented rather than included as a release-gating assertion.
Developer machines rarely have every configured device. The framework records unavailable optional targets explicitly, but a production CI lane would also need to distinguish optional devices from hardware that the job requires.
The Appium base class starts a session inside a try and calls
Assumptions.abort when one cannot be created, so a device-backed
test on a bare agent is skipped with the reason attached. This is useful for
local discovery, but it is not sufficient for a required hardware CI lane;
that job should fail when its expected device is unavailable.
Sixteen tagged JSON capability files describe the fleet. DeviceMatrix
assembles each test's device list by tag (mobile,
tv, android-tv, web), so a new
device joins every relevant matrix with no test edit and no recompile of a
test body. A run narrows with -Ddevices=… for CI sharding.
Local, Selenium Grid, a remote Appium over a mesh VPN, or a cloud lab are
interchangeable through TestEnvironment, appium.server.url
and web.hub.url. The same MobilePlaybackTest ran
green on an iPhone simulator, an Android emulator matrix and a real Pixel over
USB; a living-room Android TV on network adb ran the CTV lane against the same
page objects and the same APK.
The Toxiproxy lanes gate in a class-level @BeforeAll assumption,
so without the binary they are never started: they leave the total
rather than appear as skips. -Dgroups=pw,api currently reports
39 tests with the binary on PATH and 25 without. A production
report should keep an expected lane count alongside the executed total so
missing infrastructure cannot look like complete coverage.
A grid confined to one machine can only ever test what fits on that machine. The interesting hardware (a real living-room television, a physical phone) tends to be somewhere else, on a different network, behind NAT. Commercial device clouds solve this by owning the hardware; the same shape is buildable from a mesh VPN and an Appium server.
The orchestrating laptop joins a WireGuard mesh with a machine at another location. That machine runs Appium against the hardware physically attached to it (an Android TV over network ADB, a phone over USB) while the laptop contributes its own iOS and tvOS simulators locally. The configuration can address all of them, although the published evidence was collected in separate target runs.
The suite already addresses its Appium server by URL and its device by capability file, so a node across the internet differs from a node on localhost by a hostname. The capability files for the real TV and phone were reused unchanged; what the mesh VPN added was reachability, not a new execution model.
This is a proof of concept, not a hardened farm. The remote machine has to be awake, its Appium server up, and the TV on. There is no queue, no lease, no reclaim of a wedged device. It demonstrates the topology and the config-not-code claim; it does not survive a team pointing CI at it.
The orchestration is real rather than illustrative: a kind cluster runs a Selenium
Grid 4 hub with Chrome nodes, the mock app, and a Toxiproxy pod, with an Argo
Workflows DAG fanning out one pod per shard and joining on an aggregate step.
Nine web tests ran in-cluster, the proxy throttle drove a 240p down-switch through
the Grid where CDP could not reach, and the Argo fan-out finished four of four.
A KEDA ScaledObject for scaling nodes from the Grid session queue is
present but unverified and is not included in the demonstrated capability set.
For anyone who just wants the green lane, there is a self-contained image: a Temurin 21 Maven base with real Google Chrome installed, because Playwright's bundled Chromium ships without H.264 and AAC and therefore cannot play the HLS corpus at all. That gotcha costs a video-QA team standardising on Playwright an afternoon; here it is a line in a Dockerfile and a note in the README.
The repository labels each capability as proven or compiles so configured targets are not presented as executed tests.
.ipk installs and launches
on a webOS OSE emulator under QEMU/KVM; LG's SSAP remote-pairing step is a
commercial-TV feature the open-source edition lacks.
Vahagmount++ is a fictional service built for QA practice and educational use. Every catalog entry, including Vishap Wars: Kernel Panic, Failover, Tomato Dragon Cluster and the rest, is an invented placeholder written for test fixtures; none refers to a real program, studio, or provider.