Recently, we joined the Frontendisti community at their November Prague meetup. Tomáš Zeman, Showmax Frontend team leader of the web and smart TV team, delivered a talk about how we tackle remote testing on smart TVs. This blog post summarizes the story, and shares a useful how-to for everybody who would like to follow our example.
Showmax smart TV wall
Developing a smart TV app implies its continuous testing on various types of devices. We test many different brands and categories, from low-end to high-end, and have a cluttered smart TV wall in the office. In the past, pinning the device to the wall was the only real problem. We had all of the devices at our fingertips, and if you were planning your tests with respect to others, performing a test was pretty easy. To tackle the single obstacle — having all TVs affected by the signals sent from some other remote control — we came up with simple 3D printed adapters (image below).
Everything worked perfectly until…
After the COVID-19 outbreak, everything changed. We stopped working from the office, pulled back to remote mode, and testing anything directly on the smart TV wall was no longer possible. For some time, we improvised and moved some devices to our respective homes. We continued running tests basically on the good will of our colleagues and families. Imagine you have several large smart TVs in a tiny flat, or you have curious kids playing around them (or both) — we knew this was only a temporary solution, so we started searching for a better one.
What we did next
It’s no secret that we love to experiment, and if we can build a solution using hardware and software, we will. So, despite having no prior experience with remote testing, we decided this is the way to go. We searched the sources, learned new stuff, and came up with a solution built on Suite.st hardware.
We already knew Suite.st from automated testing we were doing. The new equipment was CandyBox and a remote-controlled IR transmitter.
IR transmitter & CandyBox by Suite.st
The IR transmitter is connected to the TV using a hemisphere-shaped, semi-transparent plastic case. The CandyBox that sends signals to the IR transmitter is a simple raspberry with eth ports. Each port is connected to one blaster to pick which TV you want to control. The connection between CandyBox and IR transmitter enables you to run tests as if you were holding a remote controller in hand — you control the TV using a virtual remote control with a javascript library.
The big advantage is that you can teach the IR transmitter any IR signal and control various devices. Another advantage is in the semi-transparency of the IR blaster — it doesn’t block signals coming from the real remote control, and you can continue running tests in-person if you are occasionally in the office. Finally, the IR blaster can be easily stuck to any part of the device so it fits any size or shape.
To get to the final setup that you can see on the image below, we combined that CandyBox and IR transmitter with IP cameras, and supported it all by tweaking our internal networks.
Demo
Issues we solved along the way
We had a very painful issue with video stream resolution, especially for UI testing. You definitely need a camera with optical zoom.
Latencies in the video were a big problem too. Anything above 500ms made the process very difficult, and latencies above 1s led to true pain and sacrifice when testing anything. We measured latencies on individual parts of the setup — you can find the rounded average values in the table below to see the scale.
| Part of the setup | Specification | Latency |
|---|---|---|
| VPN | ||
| 3rd party | 300ms | |
| Showmax VPN | 30ms | |
| CandyBox controller | <50ms | |
| Camera | RTSP stream | 1-2s |
| Camera's web interface | 400ms |
Only the CandyBox was working decently from the very start without the need for any tweaks.
VPN
Due to security blocks, we needed to separate TVs on their own network. At first, we tried an external VPN solution by Zerotier, but, due to server location and difficult routing, the latencies mounted to 300ms. After configuring our own VPN the latency dropped to 30ms.
Camera
To offer a decent user experience for the tester, you need to find a suitable camera with a latency <500ms counted between input and stream. You may either choose a camera with a super fast RTSP streaming, or a camera with its own web interface (these tend to be much faster). In the past, we used remote POC as well, which we wouldn’t recommend at all due to high Zerotier and RTSP latencies.
Next step
We are pretty happy with the current setup, and we’re improving camera coverage on the TV wall now. By adding more, we’ll be able to focus on the selected TV quickly and easily. Another improvement might be to add remote-controlled power sockets. Sometimes when the TVs freeze, you need to switch them off and on again to get some response — and remote unplugging is an easy solution.
