Recently, our Media Engineering team participated in the Demuxed 2020 conference for video engineers. Due to the COVID-19 pandemic, the conference was held virtually for the first time, but that didn’t stop 27 renowned speakers from sharing their expertise.
The speakers all had pre-recorded presentations that were followed by live QA sessions organized via the conference Slack Channel #demuxed. We must say, as media engineering geeks, we enjoyed the conference, and not only because of interesting content — the stream was very high-quality. There were only minor issues during the three days of the event and the interaction between speakers and participants on Slack was well organized. Plus there was a fun party at the end of the event that looked very much like the real ones we remember from the pre-pandemic days of in-person IT conferences.
Of the 27 sessions, we found the most relevant to us to be the one by Denise Noyes from Facebook, “Providing better video experiences for the next billion users: What Facebook learned from focusing on improving video playback in India.” We learned that Facebook has been facing very similar challenges to those that we took on in Africa. In both India and Africa, the challenges with video streaming are very similar, mainly the issue of how to tackle streaming over slow networks while still delivering a high-quality video entertainment experience. In fact, we’ve written on our blog about exactly this topic — read more in “Pushing video bitrate to the limit”.
The main goal for the Facebook team was to improve Quality of Experience (QoE) by reducing buffering events and playback errors. As Denise mentioned, there are several things you can do. Lowering the average bitrate (preferably without degrading video quality) is a very important one to consider, as it positively affects bandwidth consumption, reduces storage needs, and improves cache hit ratio on CDNs. Within the session, Facebook mentioned two approaches to lowering bitrates. We tested both here at Showmax, and we may share additional data to help assess the two approaches in the future.
Increasing coverage for low bitrate profiles
The first attempt Facebook tried was to increase the low bitrate coverage by adding 144p 100kpbs and 240p 200kbps to the ABR ladder. At Showmax, we’ve had a similar configuration since March 2019, starting even a bit lower at 144p 75kbps. The table below summarizes Facebook’s and our choices for resolution and bitrate, focused on users with lower connectivity.
ShowmaxFacebook144p 75kbps144p 100kbps216p 110kbps-288p 200kbps240p 200kbps360p 400kbps360p 400kbps
Later on, Facebook used their own proprietary tool, FB-MOS, to assess the quality of the videos at different profiles. The team discovered that higher resolutions at lower bitrates can provide a better score. So their final ABR ladder changed again, to this one below:
FromToFB-MOS Improvement144p 100kbps360p 100kbps11%240p 200kbps360p 200kbps12.5%360p 400kbps.480p 400kbps1.67%
We were curious to see whether the same behavior could be observed with the perceptual quality metrics we use — PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity) and VMAF (Video Multi-method Assessment Fusion). So, we encoded our 144p, 216p, and 288p resolutions as 360p while keeping the same bitrate. The results for the HEVC and H264 codecs are in the table below, grouped by codec and resolution:
CodecOriginal ResolutionNew ResolutionBitrate AverageVMAFSSIMPSNRHEVC144p360p4.57%557.40%2.56%4.13%216p2.78%180.76%1.39%2.83%288p2.31%40.90%0.48%1.67%H264144p-11.64%248.79%-4.59%-2.89%216p-7.08%108.55%-1.91%-1.17%288p-2.44%26.76%-0.60%-0.12%
The first row displays a hard-to-believe 557.40% improvement on VMAF for the former 144p (now 360p), but watching the video we had to agree with the numbers. Here are two examples to illustrate the difference:
However, for H264 we noticed that PSNR and SSIM greatly disagreed with VMAF about the improvement. And it is clear when watching the video that VMAF made a bad judgement, as 360p is in fact a lot worse than the original resolution on H264 (picture at the right below).
This discrepancy in the results may be explained by the natural difference between HEVC and H264 codecs. HEVC is strong in compression of large areas and can provide better quality with higher resolution, even with low bitrate. H264 performs worse in this aspect, and when working with a few bits, after compression, the picture is pixelated. Although the difference seems clear and the inconsistency well explained, we will definitely dig deeper into it in the next blog post.
For now, we take the results of this experiment as a clear demonstration of being right when using a combination of different metrics to get a solid proof for our solutions. We take it for the safe way to go, especially when automating choices like in the per-title or per-short encoding methods. For HEVC, we intend to implement the changes as soon as we assure that the 4.57% increase on the average bitrate won’t increase buffering for our users with low internet speed.
Increasing GOP size
The last change mentioned in the session was about increasing the GOP size from 2s to 5s, which provided a 6% to 8% increase in compression efficiency. The rationale is that by increasing the distance between key frames in a closed GOP, the encoder would have more space to make decisions, adding fewer I-Frames and more P/B-Frames which are much less demanding on bits. A possible drawback may be a potential delay on seeking, as it can only be done on I-Frames.
At Showmax, we were using 2s GOP as well, and we decided to experiment with 5s. Initial results showed a decrease of only -3.32% on average bitrate. We then worked further on the set of videos included in the test, adding the most relevant ones, adjusting the scenecut parameter to be less aggressive in the I-Frame placement. The default value for scenecut is 40 on both H264 and HEVC, so we experimented with 0 (no I-Frames on scene changes) and 10, but pixelated frames could be easily noted until we increased to scenecut=20. The final result is presented in the below table, with a reduction of -4.13% on average bitrate — not as expressive as the improvement experienced by Facebook, but good enough to be rolled out to production.
scenecutBitrate AverageBitrate PeakVMAFSSIMPSNR40-3.57%2.58%0.65%0.19%0.13%20-4.13%0.56%0.52%0.16%0.09%
In summary, lower bitrates and file sizes can positively affect bandwidth consumption, reduce storage, and improve cache hit rate on CDNs. The ultimate goal being a better Quality of Experience for the customer — something that can be measured in a number of ways, and should consider several factors like playback errors, slow start, rebuffering, and low quality.
It is important to note that all these metrics are interconnected. It would be easier to show improvement on buffering by offering only low quality streams, and there would be nearly no playback errors if only high-end devices on top speed connections were supported. This is why any meaningful change affecting QoE can only be achieved as a joint effort from several teams in the organization — and that’s exactly how we approach it.
