Remote testing on Smart TVs
Recently, we joined the Frontendisti community at their November Prague meetup. Tomáš Zeman, Showmax Frontend team leader of the web and smart TV team, delivered a talk about how we tackle remote testing on smart TVs. This blog post summarizes the story, and shares a useful how-to for everybody who would like to follow our example.
Showmax smart TV wall
Developing a smart TV app implies its continuous testing on various types of devices. We test many different brands and categories, from low-end to high-end, and have a cluttered smart TV wall in the office. In the past, pinning the device to the wall was the only real problem. We had all of the devices at our fingertips, and if you were planning your tests with respect to others, performing a test was pretty easy. To tackle the single obstacle — having all TVs affected by the signals sent from some other remote control — we came up with simple 3D printed adapters (image below).
Everything worked perfectly until…
After the COVID-19 outbreak, everything changed. We stopped working from the office, pulled back to remote mode, and testing anything directly on the smart TV wall was no longer possible. For some time, we improvised and moved some devices to our respective homes. We continued running tests basically on the good will of our colleagues and families. Imagine you have several large smart TVs in a tiny flat, or you have curious kids playing around them (or both) — we knew this was only a temporary solution, so we started searching for a better one.
What we did next
It’s no secret that we love to experiment, and if we can build a solution using hardware and software, we will. So, despite having no prior experience with remote testing, we decided this is the way to go. We searched the sources, learned new stuff, and came up with a solution built on Suite.st hardware.
We already knew Suite.st from automated testing we were doing. The new equipment was CandyBox and a remote-controlled IR transmitter.
IR transmitter & CandyBox by Suite.st
The big advantage is that you can teach the IR transmitter any IR signal and control various devices. Another advantage is in the semi-transparency of the IR blaster — it doesn’t block signals coming from the real remote control, and you can continue running tests in-person if you are occasionally in the office. Finally, the IR blaster can be easily stuck to any part of the device so it fits any size or shape.
To get to the final setup that you can see on the image below, we combined that CandyBox and IR transmitter with IP cameras, and supported it all by tweaking our internal networks.
Issues we solved along the way
We had a very painful issue with video stream resolution, especially for UI testing. You definitely need a camera with optical zoom.
Latencies in the video were a big problem too. Anything above 500ms made the process very difficult, and latencies above 1s led to true pain and sacrifice when testing anything. We measured latencies on individual parts of the setup — you can find the rounded average values in the table below to see the scale.
|Part of the setup
|Camera's web interface
Only the CandyBox was working decently from the very start without the need for any tweaks.
Due to security blocks, we needed to separate TVs on their own network. At first, we tried an external VPN solution by Zerotier, but, due to server location and difficult routing, the latencies mounted to 300ms. After configuring our own VPN the latency dropped to 30ms.
To offer a decent user experience for the tester, you need to find a suitable camera with a latency <500ms counted between input and stream. You may either choose a camera with a super fast RTSP streaming, or a camera with its own web interface (these tend to be much faster). In the past, we used remote POC as well, which we wouldn’t recommend at all due to high Zerotier and RTSP latencies.
We are pretty happy with the current setup, and we’re improving camera coverage on the TV wall now. By adding more, we’ll be able to focus on the selected TV quickly and easily. Another improvement might be to add remote-controlled power sockets. Sometimes when the TVs freeze, you need to switch them off and on again to get some response — and remote unplugging is an easy solution.