The same release ships to a browser, a phone, a tablet, a set-top box and three flavours of smart TV, and it fails differently on each. A television has no pointer, only a focus engine and a D-pad. A phone suspends your process mid-playback. A browser gives you developer tooling that no TV will ever give you. Coverage means running against all of them, which turns a testing problem into an infrastructure problem: you need somewhere for all those devices to live, and a way for one test run to reach them.
That is what this project is. A grid, a device matrix, and one page-object layer thin enough that the same test body survives the jump from Chrome to a living-room TV. Everything else here (the mock service, the streams, the fault injection) exists so the grid has something real to prove itself against.
Three kinds of place, and a test body cannot tell them apart: containers in a Kubernetes cluster, simulators on the machine running the suite, and real hardware on a completely different network, reached over a mesh VPN. Each was proven on its own run; what they share is that execution location is configuration, not code.
Every interesting question in streaming QA (does adaptation actually down-switch under load, does the join time hold, does the same page object survive a jump from a browser to a TV remote) needs a real player on a real screen to ask. Public demo streams and headless stubs answer none of them.
So the app came first: Vahagmount++, a deliberately fake
streaming service named after its author and incremented once, because the
interesting part was never the streaming. It exists in five flavours, each
honouring the same accessibility contract so that the same page objects (and
in most cases the same test bodies) drive all of them. Three of the five are built without their conventional build
system on purpose: no Gradle for Android, and no .xcodeproj for
either Apple build. That keeps the toolchain to an SDK and a compiler, and it
builds in constrained environments where a Gradle daemon's loopback sockets are
blocked.
| Build | What it is | How it's built |
|---|---|---|
| mock-app | Web client: HLS.js and Shaka behind one engine abstraction, live window.__qoe telemetry |
static files |
| mock-app-android | Native APK: login, home, player | javac → d8 → apksigner |
| mock-app-ios | SwiftUI iPhone app | bare swiftc |
| mock-app-tvos | SwiftUI Apple TV app, focus-engine navigation | bare swiftc |
| mock-app-webos | LG webOS .ipk repackaging the web client |
ares packaging |
The usual way a cross-platform suite rots is triplication: a
WebLoginPage, an AndroidLoginPage, a
TvLoginPage, and three copies of every flow that walks through them.
The alternative most teams reach for, if (platform == …) inside
the page object, is the same rot with worse ergonomics.
The fix here is to move the platform decision inside the locator.
ByPlatform extends Selenium's own By, so it drops into
driver.findElement(...), every BasePage helper and
ExpectedConditions with no special handling. Resolution is
most-specific-wins, and the platform comes from a ThreadLocal the
driver factory binds at session creation, so the page object never receives
or threads a platform argument at all.
Android TV and Fire TV belong to the TV family but are Android underneath, and
they run the same APK as the phone, so a plain family lookup would hand
them the accessibility-id locators meant for webOS and tvOS. The resolver
therefore inserts one step: an Android-flavoured target with no TV-specific
locator prefers the ANDROID exact locator over the TV family.
Nine unit tests pin the resolver down, including the cases that matter most:
that an explicit androidTv() locator still beats that
fallback, and that Apple TV and webOS are not dragged into the
Android branch.
No implicit waits anywhere: implicit and explicit waits compound unpredictably
and mask races rather than fixing them, so every find, click and type is a
bounded WebDriverWait. No @FindBy / PageFactory
either: its lazy element proxies are a well-known source of
StaleElementReferenceException, and they do not compose with a
polymorphic locator in the first place.
Most ABR "tests" watch a stream and hope adaptation happens. That is a coin-flip dressed as an assertion. Here the ladder is built to make the outcome deterministic: one media corpus is encoded to three well-separated rungs (240p at roughly 250 kbps, 480p at 600, 720p at 1.1 Mbps), and the throttle is set at about 400 kbps, deliberately parked between the floor and the 480p rung. Only one rung fits. A player that does not drop to it has failed.
There are two throttles in the suite. The plain Playwright lane shapes bandwidth
through Chrome DevTools, which is fine for a browser on the same machine; the lane
that proves the mechanism generalises applies the same shaping at the
network layer through Toxiproxy. That second choice is the load-bearing one: CDP's
setNetworkConditions cannot reach a browser running on a Grid node or
a phone on the other end of an Appium session, whereas a proxy in front of the
origin shapes bytes for anything downstream of it. The framework spawns the native
toxiproxy-server binary itself over ProcessBuilder and
blocking HTTP, so no Docker and no cluster is required to run the lane, and the
identical conditioner drives an in-cluster Toxiproxy with only an admin URL changed.
What gets asserted is not "the video element exists" but the telemetry the player
publishes: window.__qoe carries video start time, rebuffer count and
ratio, ladder position, bitrate-switch history and dropped frames
(deliberately the vocabulary Conviva, MUX and NPAW expose for real). Both browser
engines write the same object, so a CMAF stream played through Shaka and an HLS
stream played through HLS.js are checked by literally the same assertion code.
Symmetry would suggest also asserting a mid-stream up-switch once the pipe widens. It is not asserted, because Shaka's estimator only samples bandwidth on a segment fetch: the first load measures a cold network while later loads see warm, near-instant transfers it discards. Reliable for HLS.js, flaky for Shaka. The down-switch is deterministic and a fast-pipe ceiling check covers the other direction. An assertion that is sometimes true is worse than one that isn't there.
A device farm's tests run on machines that mostly do not have the devices. The failure mode this creates, red builds that mean "no Fire TV plugged into this agent", trains everyone to ignore the colour. So absence is modelled explicitly rather than handled by accident.
The Appium base class starts a session inside a try and calls
Assumptions.abort when one cannot be created, so a device-backed
test on a bare agent is skipped with the reason attached. Green when devices
are present, cleanly skipped when they are not.
Sixteen tagged JSON capability files describe the fleet. DeviceMatrix
assembles each test's device list by tag (mobile,
tv, android-tv, web), so a new
device joins every relevant matrix with no test edit and no recompile of a
test body. A run narrows with -Ddevices=… for CI sharding.
Local, Selenium Grid, a remote Appium over a mesh VPN, or a cloud lab are
interchangeable through TestEnvironment, appium.server.url
and web.hub.url. The same MobilePlaybackTest ran
green on an iPhone simulator, an Android emulator matrix and a real Pixel over
USB; a living-room Android TV on network adb ran the CTV lane against the same
page objects and the same APK.
The Toxiproxy lanes gate in a class-level @BeforeAll assumption,
so without the binary they are never started: they leave the total
rather than appear as skips. -Dgroups=pw,api is 39 green with the
binary on PATH and 25 without, and both numbers are correct.
Saying so is cheaper than explaining a moving total later.
A grid confined to one machine can only ever test what fits on that machine. The interesting hardware (a real living-room television, a physical phone) tends to be somewhere else, on a different network, behind NAT. Commercial device clouds solve this by owning the hardware; the same shape is buildable from a mesh VPN and an Appium server.
The orchestrating laptop joins a WireGuard mesh with a machine at another location. That machine runs Appium against the hardware physically attached to it (an Android TV over network ADB, a phone over USB) while the laptop contributes its own iOS and tvOS simulators locally. One run dispatches to all of them.
Because remoteness was never a property of the test. A suite already addresses its Appium server by URL and its device by capability file, so a node across the internet differs from a node on localhost by a hostname. The capability files for the real TV and phone were reused unchanged; what the mesh VPN added was reachability, not a new execution model.
Honesty about the trade: this is a proof of concept, not a hardened farm. The remote machine has to be awake, its Appium server up, and the TV on. There is no queue, no lease, no reclaim of a wedged device. It demonstrates the topology and the config-not-code claim; it does not survive a team pointing CI at it.
The orchestration is real rather than illustrative: a kind cluster runs a Selenium
Grid 4 hub with Chrome nodes, the mock app, and a Toxiproxy pod, with an Argo
Workflows DAG fanning out one pod per shard and joining on an aggregate step.
Nine web tests ran in-cluster, the proxy throttle drove a 240p down-switch through
the Grid where CDP could not reach, and the Argo fan-out finished four of four.
A KEDA ScaledObject that autoscales nodes off the Grid's session
queue is written but not yet driven hard enough to watch it scale. It sits
on the backlog, labelled as such.
For anyone who just wants the green lane, there is a self-contained image: a Temurin 21 Maven base with real Google Chrome installed, because Playwright's bundled Chromium ships without H.264 and AAC and therefore cannot play the HLS corpus at all. That gotcha costs a video-QA team standardising on Playwright an afternoon; here it is a line in a Dockerfile and a note in the README.
The repository labels every claim as proven or compiles, and the gap between the two is where the honest reading lives.
.ipk installs and launches
on a webOS OSE emulator under QEMU/KVM; LG's SSAP remote-pairing step is a
commercial-TV feature the open-source edition lacks.
The service the farm tests (Vahagmount++, named after its author and then incremented, because the streaming was never the interesting part) is fictional, built for QA practice and released for educational use only. Every catalogue entry in it, Vishap Wars: Kernel Panic, Failover, Tomato Dragon Cluster and the rest, is an invented placeholder written for test fixtures; none of them refers to a real programme, studio or provider.