Tackling a device-specific problem with playback on the Xiaomi Redmi 4A.
This all started with a spike in the number of errors on the Xiaomi Redmi 4A, specifically. We provide a video-on-demand service, so, when we see a massive device-wide playback failure, it’s a big deal. In this case, we had no idea what was going on and our ideas were basically shattered with every refresh of Kibana (more on what Kibana is here).
The problem was not a sudden immediate peak (or, in this case, trough), which confused us even more. We tried to scope the affected devices, but some that were failing had successful playbacks as well, and vice versa - with the same app version. The reported results were saying that 30% of users running the same OS were still able to play the content. There were no distinct patterns to their playback habits, the groups of content with errors were overlapping those without errors, no time patterns, cats were playing with dogs, etc. The only hard fact we had was that playback was at least partially-broken on one specific OS version.
We had something.
The Redmi 4A was prohibited from downloading our app from the Play store. Customer Care was notified that we were working on a solution. In the meantime, we decided to buy one of the devices and try it out. Launch > home screen > download the app > All good. It worked. Which was perfect because it created a dead end.
But, after updating the device OS, it has started to behave as users had reported:
android.media.drm$media drm state exception: failed to get provision request: drm vendor defined error: -2990So, there was a clear connection in some way to the OS update. We checked other video services: pre-update it worked, post-update it didn’t. This was telling as it meant that the issue wasn’t in our code, but was caught up in a wider problem - later confirmed via research posted on github (more here, here, and here).
The next step was to find out what was going on inside the OS when this error occured. We couldn’t find out the root of the problem by ourselves, so we contacted Xiaomi directly to help us solve the issue.
After a few bug reports, a Xiaomi representative visited our Showmax office in Poland to take a look at the problem together. This led to a revelation: A factory reset solved the problem. On its face, we had a solution, but we had a solution that meant tens of thousands of people would have to do factory resets. This was not at all optimal.
Xiaomi wanted more information and we wanted to solve the issue. So, a special build was installed on the device, a new bug report was generated, and we handed it over to Xiaomi.
The next step was to wait for the OTA to come to our devices and test it. A new version of the OS came to the test device with the broken playback, we crossed our fingers, and updated the OS.
It worked.
Three months of preparation and three months of waiting were over, and we had a solution for this issue. Game over. We asked some friends to install it, try playing videos on our app, and also checked our logs: There were still some users with the app installed, some of them updated to the newest OS, all of them signaled no DRM-related playback issues.
Finally, after seeing that playback actually works for the new OS, the device was allowed to install the Showmax app in the Google Play store.
Also, the Customer Care team was informed with instructions for users who continued to experience the issue. If they can’t play movies - the solution is to update the system to the latest available MIUI version. No need to factory reset, the problem is fixed seamlessly.
To be aware of the scope of the issue, changes in users, and to monitor possible failures, a new dashboard was created.
We won this battle on the Redmi 4A device, but it could happen again at any time on any one of the hundreds (thousands, really) of different devices on the market. It was very important that we were able to cooperate with other video providers, researchers, and the manufacturer directly, and we couldn’t have solved this problem had we not worked together.
Thanks everyone for diving with us into this intriguing journey!
Finally, we want to thank Martin Šimka, the Xiaomi team and our in-field agent in Poland who first tested the Showmax app with the new OS version.
The artwork in the header is “Mental Calculation. In Public School of S. A. Rachinsky” from Nikolay Bogdanov-Belsky.
