How to efficiently serve images in various shapes and formats
Here at Showmax, we use images all over our streaming platform. They can have various forms - background, hero or poster image, carousels, etc. After years of production experience with image management we have a lot to share with you, so here’s a look at the images workflow at Showmax.
The life of every image begins in our Content Management System (CMS). Basically, it’s the heart of our content metadata workflow. Image management is built in to the CMS, where images are stored on GlusterFS volume and served to clients via Image API as shown on the image here:
Image workflow
A content operator usually creates and uploads several pictures for each movie or TV series. Then they prepare carousels, posters and hero images with different aspect ratios, localized per each country where the asset is licensed, and more. Based on the dominant colors of an image, the system automatically computes for font colors for overlaying text and background color gradient. These images are then served to users.
Moreover, we have another type of image - “Image overlays”. They are something like the image snippets and served only as stuck on prepared pictures. Most of them have text marks like “Coming soon”, “Editor’s choice”, “Last chance”, etc.
Subsequently, the images are served to the customers via a microservice called Image API. It’s a place where the actual magic of efficient transformation happens. As stated in the title, we use one origin source image and transform it to different shapes and formats on-the-fly. The Image API knows how to do some transformations, like:
- Nothing, only image quality is set to the default value of 75.
- Resizing just by appending parameters “width=”, “height=” or “800x500” to the URL. Transformation-by-default keeps the same aspect ratio as the original picture, but you could enforce resizing to exact dimensions with a special mark “f”.
- Cropping by appending parameters “crop_width=”, “crop_height=” or “crop=x” to the URL.
- Desaturating by appending boolean switch “desaturate” to the URL.
- Padding by appending parameters “pad=x” and “pad_color=” to URL.
- Sticking pre-defined overlays by appending parameter “overlay=” to the URL. The value is a JSON string encoded in base64 containing overlay attributes like: position, opacity, and scale.
- Setting quality by appending parameter “q=” to the URL.
- Changing the output format by changing the suffix in the URL.
All of those options can be applied on their own, or arbitrarily combined with others. Front end applications then ask the image service for a picture and, after it’s generated, it is cached by a CDN and delivered to the client.
Where the magic(k) happens
Our Image API is built on Falcon framework and GraphicsMagick tool as an image processing engine, hidden behind CDN caches. The main benefit of GraphicsMagick for us was the processing of “webp” format - something other tools such as Pillow or ImageMagick did not have implemented at that time.
Here are some interesting tips and unexpected troubles we had during the implementation.
Streaming the images efficiently
When composing image with overlay (another image), we have to process several images together. The process consists of several steps during which new images are created (as mezzanine) and stored on disk after each call of GraphicMagick. They are then merged back together by another call of GraphicMagick. Every call of GraphicMagick invokes the reading of the image from disk and writing the results to disk - that’s a lot of expensive I/O operations. We needed to find out how to read every image only once and store intermediate and final results in memory.
Luckily, GraphicMagick supports a format called MIFF - it means, “Magick Image File Format“, and is used for storing bitmap images. You could join many images into a single one, or use it as a stream of images like using pipes in Unix systems.
Our more-efficient image workflow:
1. Read one original picture from a filesystem
2. Read overlay image from a filesystem
3. Resize the overlay image based on the size of overlayed image
4. Stick overlay image to the original picture
5. Resize the result and send it to the user
Dominant color
Earlier you read about dominant color of images and where we use it. The problem was that we had to compute the dominant color using GraphicsMagick - which does not have such a function.
The solution is quite simple (with a bit of out-of-the-box thinking). There are steps:
- Transform the input image to the same color space (as other color spaces like CMYK will provide different results than RGB color space).
- Drop all profiles and text attributes from the image.
- Remove the matte channel if the image has one.
undefined - Reduce the colors of the image to the minimum - one.
- Resize the image to 1x1 pixels in MIFF format.
Now we just need to read the color palette containing one color from 1x1 pixel image. We don’t even have to parse the file, just read the last 3 bytes where the color code is always placed. The length and structure of 1x1 MIFF images is always the same.
The GraphicsMagick commands could looks like this:
# Transform to RGB color space with one color - steps from 1 to 4
/usr/bin/env gm convert - -strip +matte -colorspace RGB -colors 1 -compress none miff:-
# Resize image to 1x1 - step 5
/usr/bin/env gm convert miff:- -resize 1!x1 -compress none miff:-GraphicsMagick as a computational engine
Here’s why we use GraphicsMagick as a calculator. We’ve already mentioned that several image transformations can be applied to an image in one request on Image API.
Sometimes a front-end application needs to process more operations in one call. It needs to resize image → crop → stick overlay - everything - while preserving aspect ratio in the resize step, and the app specifies only the required width.
Resize and crop is done in one step, so we need to know what the image’s height would be after resizing in order to adjust crop parameters properly. We can’t compute it in python due to different approach to rounding - we need to use the same kind of rounding as GraphicsMagick. So, we prepare an empty canvas with the original dimensions of the input image, try to resize, and find out the new dimensions.
This trick can be done like this:
# Resize empty canvas
/usr/bin/env gm convert -size ${IMAGE_WIDTH}x${IMAGE_HEIGHT} xc:none -monochrome -resize ${NEW_IMAGE_WIDTH:-}${FORCE}x${NEW_IMAGE_HEIGHT:-} -compress none miff:-
# Get new image properties
/usr/bin/env gm identify -verbose -format %m:%w:%h -Conclusion
The image API helps our front-end teams with basic image transformations without the need to upload the different variants of images to our CMS. Our content teams can mark images with overlays without having to use an image editor to burn in an overlay and then upload a new image.
Image API currently works with an image library containing more than 300,000 images, and growing. Transforming images on-the-fly is quite costly, therefore every variant of a given image is computed only once and then cached on a CDN with an 85% cache hit ratio.
The last (but certainly not least) advantage of our in-house solution is the cost savings. Commercial services providing the same functionalities usually have payment models that consist of a fee for every single transformation. Ours has a semi-fixed cost, as we’re caching the images on the CDN and the network traffic is almost negligible. Finally, it is easy to maintain, since it has small codebase of less than 1000 lines.
