More than two years ago, in July’s high season, we launched a new top-notch product, Showmax PRO. In Forbes, Showmax Pro was called a Streaming Soccer Service. But there is much more to Showmax Pro than just live soccer games (find out more). This brand new subscription plan bundles the existing Showmax library together with various live sports from SuperSport and adds music channels and news on top.
Have a look at Showmax Pro’s launching video
The Business and Product teams did a great job coming up with an idea of live sports on Showmax. In Africa, live sport streaming brought us an important competitive edge. Rugby and football World Cups are the most popular sporting events in both South Africa and the rest of Africa. We’ve also covered the majority of the 2020 Summer Olympics and are getting ready for the 2022 FIFA World Cup. Needless to say, the new plan and functionalities brought our engineering team a bunch of interesting problems and challenges to tackle.
We’ve been in the video on demand (VOD) business for a few years already and came up with a solution to (almost) every issue, so it was time to open another door and take on another challenge – live streaming.
Now that the new product has been launched and running for some time, I can say this: with a bit of exaggeration, streaming movies on demand is easy when compared to streaming live sports. Simply put, all you have to do in the VOD business is cut and encode the videos, load them into your internal CMS, set up the metadata and upload the images, run it through QC, publish, and after a few minutes the movie appears in your apps. The rest is about streaming the content using the CDNs. Sounds simple, right? There’s nothing to stop you from making the content available for months or years. If it works once, it should keep working.
But when it comes to live sports streaming, there’s a lot more to handle on top of what’s needed for VOD. The data is always outdated. There are always last minute changes to the tournament roster, delays or overtimes, or even cancellations due to bad weather. These are just some of the scenarios that we’ve faced and that led us to re-think how our platform works. With live sports, you simply cannot plan for the future. You cannot use the processes fine-tuned for VOD; they are neither fast nor flexible enough and you need constant supervision and full control over the content. If a live stream goes down, fixing it in 10 minutes is not good enough as the match goes on – users would have already missed Ronaldo’s awesome goal.
On the other hand, sports were not an entirely new challenge for us. Our sister company SuperSport has decades of experience in broadcasting sporting events to all of Africa. Additionally, DStv, another sister of Showmax providing catch-up service with live channels, helped us with the live streaming experience and infrastructure. So along with the content we gained a lot of useful know-how. It’s great to be a part of a big family, right? Since we had the content and know-how, it may seem like we were all set, but there was still a lot to learn.
Getting fans to the stadium
Showmax PRO was launched as a new type of Showmax subscription, with both a normal and a mobile-only variant. Once the users upgrade to the new PRO subscription, they can immediately start watching the sport content. But making that content available is not such a straightforward process.
Event scheduling and real time updates
Manual curation and scheduling is not possible for live streaming, because, as explained above, it would mean a tremendous and neverending load of work. To solve that, we integrated Showmax with the SuperSport systems to synchronise our systems with the latest schedules available. We rely on full automation with rare manual interventions – used in cases of, for example, unexpected overtime.
The content database that we’re using for VOD has long update times, and to tackle this problem, we have developed a parallel metadata pipeline. It is used solely for live content and enables the immediate propagation of any change to production. Immediate in this case means that the process of updating the schedules across our applications is finished within one minute, compared to ~10 minutes for our VOD content.
Events vs. channels
To create the actual live stream, we are using the same video feeds (channels) as we do for SuperSport’s satellite TV broadcast. We run it through on-premise AWS Elementals and control when or for how long the feed is available for the users to stream the match or event they want to watch. From the user’s perspective, this is quite different from what most of the competitors offer and we believe it makes perfect sense. The concept of 24/7 live channels works for TV, but not necessarily for an online VOD service.
Focusing on the customer experience, we want to stream the match from start to finish, nothing more and nothing less. It may seem easy at a first glance, but it proved to be very challenging. Different countries need to receive different video feeds, sometimes at different times. The requirements for serving ads differ as well. For example in Mauritius, alcohol advertising is illegal. We needed to come up with a solution for that – in the first phase, the solution was to dynamically replace the ad with a still image. Later on, we plan to replace the ads within the video feed with dynamic ads.
The live feed can also be influenced by other events and the stream can actually change before the event is finished. And that’s where the need for full real-time control originates. Still, we do believe that our approach of “events” rather than “channels” makes sense from the user’s perspective and we plan to keep it in the future.
Avoiding the overload
Now, when the live stream is set up and everything is in order, the users can start tuning in for the kick off. There’s one small problem though – a majority of the users will launch the app and the live stream at almost the same time, which results in a huge spike of traffic and requests that need processing. Surprisingly, very similar spikes can also be seen during half-times or at the end of the match, when people start leaving the stream and potentially looking for another movie to watch.
The playback starts pattern for one of the live football matches.
We took multiple steps to avoid overloading the platform. Apart from several small optimizations, two features play the key role.
Firstly, we have improved our “overload mitigation” functionality. Based on the current need, we simply set the mitigation value (0 - no mitigation, 1 - maximum mitigation) in our HAProxies, which pass it in a header as part of all requests to our services. Each service can then work with the header as necessary. In general, the higher the value, the less personalised and more cached the platform is.
The second feature is much simpler – we allow the users to start the stream 10 minutes before the actual start of the match. During these 10 minutes, push notifications are also sent out, staggered across the duration. This spreads the user requests out over the time period and significantly helps with the load distribution. It’s also convenient for the users who do not need to wait for a specific time to start the stream.
In the near future, after the transition to AWS is finished, the platform will become much more scalable within a short time, which will also help significantly.
Everyone hates when there’s an outage and you have to learn about your home team’s goal from the ecstatic screaming of your neighbour…we’ve all experienced that, and our customers deserve better service. It’s crucial to catch such interruptions as soon as possible. In Connected Video, we manage more than five hundred live channels and use several levels of monitoring.
We leverage Prometheus and Alertmanager heavily as the primary monitoring tools of our services. For the live channels we use Blackbox exporter, allowing us to probe channel playlists and manifests in regular intervals. We make a simple assumption, that if a probing HTTP request returns 200 OK and the last-modified header is not older than segment length, then the live channel is alive. It’s a relatively cheap approach to probe availability from different destinations and CDN paths.
On top of that, we also monitor statistics and events from devices during playback, which provide effective insight into the state of live streaming (and VOD as well). This helps us catch various issues across a variety of devices.
Premium and most popular SuperSport channels or events are additionally monitored by a commercial monitoring tool called Stream Probe and Witbe. It’s another level of active monitoring to ensure the top quality.
We have also extended our CMS with a page allowing the sports team to monitor all the running live streams simultaneously, and to easily adjust the schedule if necessary.
The live stream monitoring page in our CMS.
Quality of experience
In general, you can say that quality of experience arises from the convenience of use. To help users easily find the matches they want to watch, we designed a brand new timeline-based sports page which looks nothing like a typical EPG (Electronic Programme Guide). Users can filter the content based on the sport genre or time and also set up notifications for selected future live events.
From the technical point of view, we did several upgrades compared to the first solution. At the beginning, we provisioned dedicated live streams for Showmax provided by the DStv infrastructure. It was not optimal, as the same content which was broadcast on both platforms was packaged and cached on the CDN twice. In the next iteration, we focused on sharing the content on the CDN better, as the events were often broadcast on both platforms (even though the platforms target different groups of customers). It helped lower cache fragmentation.
Another enhancement is active content prefetching. It means that the CDN tries to fetch the next segment of the stream before the first customer asks for it. It lowers the response time for the first customers, as well as on less popular channels in general or in sparse areas. This feature is provided by Akamai, which we use as the primary CDN provider for our live streaming and it’s also supported by our Unified Origin component via hinting the next segment in headers.
The final indicator of good quality of experience is “glass to glass” latency. It’s a delay between real live action in the stadium and what you see on your screen. Standard latency of HTTP based live streaming is between 20 to 40 seconds. By tuning parameters (like segment length or buffer size in player), you can get significantly below these values. But it’s always a matter of balance between picture quality, buffering issues, and other attributes of good experience. We took inspiration from Zatoo’s posts 1, 2, and 3 to tweak it on our side.
Content piracy is a big issue all over the world, and Africa is no exception. Live sports streaming is more targeted and at higher risk of being pirated than VOD. Content rights are sold for shocking amounts of money and streaming companies do their best not only to protect the content but also track the leaks and catch the violators.
Nowadays, all content is protected by DRM, and surprisingly even DRM can’t provide sufficient protection. We all remember Widevine being breached several times in the last year and it probably won’t be completely fixed ever again. Although it still works fine against casual attackers (understood as BFU).
Anti-piracy efforts are mainly focused on a different group of violators – highly specialised crackers who build their business on re-selling stolen content. They don’t just cause losses by re-selling but also re-using the CDN infrastructure of the content owner. For such cases, we incorporate geo-blocking on the CDN to mitigate that behaviour. We work closely with Irdeto (another sister company) who help us identify and track pirates. Together we look for suspicious behaviour in the streaming environment, like unexpectedly high traffic on specific IPs. Thanks to them, several piracy businesses were traced and closed down eventually.
When the match ends: Highlights
Missed the live action? Highlights are short previews of matches that our users can watch to quickly catch up on the games they’ve missed, or to enjoy the best moments again. To ensure we receive sport highlights as soon as possible, we had to significantly speed up our video delivery workflow. We imposed a rule of releasing these highlights within one hour after the game ended, and in order to comply, we started using standardised “mezzanine” video input within our encoding pipeline and limited the number of resulting video profiles.
We briefly mentioned at the very beginning that enabling live streaming included not just sports but also 24/7 streams of multiple news and music channels, including the South African national TV channel SABC with regular COVID-19 updates by the country’s president, Cyril Ramaphosa. The Nigerian reality show “Big Brother Naija”, which we were able to stream 24/7, was a great success too. We can also stream live concerts, beauty contests, or other live shows. Live streams are a huge leap forward in the variety of Showmax content libraries and the effort definitely paid off.
We already have plenty of ideas of what to do next and what else to improve. One of the areas we plan to focus on is content discovery – making sure the right people find the right content. We have already done a great job in VOD, so now it’s time to use the know-how for sports. For that, we plan to utilise more of the league’s and teams’ data.
Would you like to understand some of the above even further? Let us know on Twitter.