We’ve managed to launch a pretty useful series of talks sharing knowledge and teaching skills some of us have and many more could benefit from. The series has been offered internally for several years now, and for more than a year, we’ve been sharing the talks publicly.

Another edition focuses on JSONB and is led by Angus Dippenaar, DevOps engineer.

The talk offers an explanation of JSON and JSONB, what the difference is between them, what is cool and where their limitations are. You will learn when to use JSONB and when to avoid it. What is interesting is the comparison of JSON, JSONB, and Tables.

We have listed a few points for each of the parts of the talk – check out the marked time within the video to learn more.

Imagine JSON but query it. It is revolutionary.
JSON was introduced in Postgres 9.4 and JSONB in Postgres 9.5
JSON is just stored as text, and is converted to a type when the value is read. JSONB is stored in a binary format.
If we compared JSONB to PostgreSQL types, e.g. a string in JSONB will be stored as text in PostgreSQL, a number will be stored as numeric (NaN and infinity values are not allowed), “null” does not translate because NULL in PostgreSQL is a different concept. Learn more in the talk…

*$ SELECT 'null'::jsonb IS NULL, 'null'::jsonb = 'null';

From PostgreSQL documentation/JSON/Data types
– Simple scalar/primitive value
– Primitive values can be numbers, quoted strings, true, false, or null
SELECT ‘5’::json;

– Array of zero or more elements (elements need not be of same type)
SELECT ‘[1, 2, “foo”, null]’::json;

– Object containing pairs of keys and values
– Note that object keys must always be quoted strings
SELECT ‘{“bar”: “baz”, “balance”: 7.77, “active”: false}’::json;

– Arrays and objects can be nested arbitrarily
SELECT ‘{“foo”: [true, “bar”], “tags”: {“a”: 1, “b”: null}}’::json;

A few cool things about JSONB are that you can filter the data, index specific keys, index the whole thing (jsonb_path_ops), or add constraints. You can interact with JSONB and the normal table columns within the same query.

It is hard to optimize your query as there are no statistics within JSONB or JSON fields, because it’s considered to be just one column.

The concat operator (||) is only top-level
Max row size of 268 435 455 bytes (268.43 MB)
NaN and infinity are not allowed
null IS NOT NULL
\u0000 is not allowed

As for the setup, I installed PostgreSQL 12 on Ubuntu 18.04 and ran Python scripts, each query 12 times, 4 concurrently.

max_connections = 200
shared_buffers = 4GB
effective_cache_size = 12GB
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 5242kB
min_wal_size = 1GB
max_wal_size = 2GB
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8

Figure out how Twitter streaming API works.
Save about 4M Tweets as JSON (Avengers and Game of Thrones).
Save these 4M Tweets into a table, one for Avengers, one for Game of Thrones.
Join tweets_avengers and tweets_got into one table, still in JSON.
Convert to JSONB.
Model single table with all 2nd level JSON objects as JSONB fields.
Model the Tweet fully.
Run experiments.

You can see how the experiment went in the video.

In the real life example sourced by the experiment, you can see a comparison of the data stored in JSON, JSONB, and normal tables. Below is a comparison of table sizes, and a comparison of the time needed for a query, showing which one to use when you need to solve a problem quickly.

Check out the story time at the time 19:40 in the video to think about a possible use case.

Here are some examples of when you can use JSONB:
Got NOSQL, but you need some ACID
Storing document data - like settings, API queries (HAR format?)…
Audit trail
Fast-changing data
Data with many optional columns/keys (Could inherited tables work?)
As an aggregate (e.g. storing JSONB into a table)
When you have JSON data and quickly need to do some queries

Here are some examples of when it could be a bad idea to use JSONB:
When you need to filter the data
When you need indexes
When you need constraints
When you need a key-value store
When you could use a table
When other people need to know what is going on
When you need to store a lot of data in one row

Programming, the same as all other sorts of engineering, is about compromises. There is no absolute answer, maybe JSONB works for you, maybe not. But we need to try it to see what options we have and set benchmarks.

By Showmaxers, for Showmaxers: Is '{JSONB}' a Silver Bullet?

Here at Showmax, we believe that sharing knowledge and teaching each other the skills we have, is truly a way to reach greater heights as a team. In the “Sharing is caring” spirit, we came up with a series of internal talks called, “By Showmaxers, for Showmaxers”, and we recently started making them public. There are already talks available about Networks, Android app building, or PostgreSQL terminology.

Our latest talk focuses on Linux containers and is led by Štefan Šafár (@som_zlo), a Media Engineer at Showmax.

The talk was meant to briefly explain virtualization and containerization history and offer a deep dive into how containers are implemented in the Linux kernel, with how-tos that you can easily follow to set it all up yourself. The second main focus was to show what Docker can actually do and how to debug the most common issues encountered when using Docker.

VMs / virtualization, 1:56
Processes, 3:55
Container, 8:09
Chroots, 10:40
Namespaces, 14:40
cgroups
cgroups namespaces, 43:54
user namespaces, 44:37
container runtimes, 47:19
issues in Docker, 48:30

All the terms are explained in a separate part of the presentation using demos in a terminal to show how they work and how they contribute to creating containers, so it’s best to watch the talk and examples, and possibly use bookmarks to navigate and start learning. Enjoy!

By Showmaxers, for Showmaxers: Linux Containers

Here at Showmax, we enjoy the luxury of being surrounded by highly-experienced colleagues. Somebody somewhere at Showmax can both understand and work with the thing that no one else can — all the time. Maybe it sounds a bit like bragging, but the truth is that Showmax is full of great people who are willing to share their knowledge.

In the “Sharing is caring” spirit, we came up with a series of internal talks called, By Showmaxers, for Showmaxers, and we recently started making them public. There are already talks about Networks, and Android app building, available.

For us, sharing the talks is a fantastic way to look into somebody else’s mind, understand how they think, and see how they solve problems. And, it’s a great way to share and discuss knowledge across Showmax and beyond.

Our latest talk focuses on PostgreSQL Terminology, and is led by Angus Dippenaar. He worked on Showmax projects from South Africa, and moved to work with us in Prague, Czech Republic. Angus was recently introduced in the interview investigating the state of DevOps at Showmax.

The talk was meant to fill some holes in our knowledge of PostgreSQL. So, it guides you through the basic PostgreSQL terminology you need to understand when reading the official documentation and blogs.

You may learn what all these PostgreSQL terms mean

Command, query, local or global object, non-schema local objects, relation, tablespace, database, database cluster, instance and its processes like postmaster or backend; session, connection, heap, file segment, table, TOAST, tuple, view, materialized (view), transaction, commit, rollback, index, write-ahead log, WAL record, WAL file, checkpoint, Multi-version concurrency control (MVCC), dead tuples (dead rows), or transaction exhaustion.

PostgreSQL Terminology from Showmax Engineering

The terminology is followed by a demonstration of transaction exhaustion

Check out the demo here
Code for the demo on GitHub

You can watch a recording of the talk on YouTube

By Showmaxers, for Showmaxers: PostgreSQL Terminology

Recently, we discussed our move to Monitoring with a modern tech stack that revolved around our switch to Prometheus and AlertManager. Since early 2020, we have continuously made changes to both monitoring and alert management systems.

In this interview with Angus Dippenaar, we get into where we stand now, the changes we’ve made, and how we are getting ready for the transition to cloud.
Angus works for MCG and is actively helping the Showmax Engineering team with
the evolution of the monitoring ecosystem.

When working with Showmax DevOps team, you saw a great deal of fine-tuning done to the monitoring and alert management ecosystem built around Prometheus. Take us back to the beginning. What were the main motivations to make the change to Prometheus and AlertManager?
The old way, using Nagios/Incinga and black-box monitoring for our alerting setup, was not ideal. It worked, but there was room for improvement and we all just knew that we could do better. All we had to work with was the current value in the alerting system. We would see “Disk Usage > 90%,” and then we had to investigate.

Using a time series database like Prometheus offers, means that we can see metrics over time. We can see a trend that could have caused the issue, or we can see how a different metric has contributed to the problem that we’re currently experiencing.

Having metrics over time means that we can create an alert based on a trend. For example, we can have a count of all the values above a threshold over a certain period of time and create an alert based on that. You could have an alert based on how many http errors you had over 5 minutes, and if it’s greater than a defined threshold, then you probably know that something is wrong.

What does the alert ecosystem look like now? Is it already 100% based on Prometheus data?
Right now, we are still converting Icinga-based alerts into the new metrics-based approach. It’s not the highest priority, and we still have some Icinga-based alerts. But, we still want to move completely to Prometheus and white-box monitoring, and we are working towards that. Another advantage that might be important in the near future is that Prometheus is cloud-native and we can count on it when moving to the cloud.

Can you share some highlights from your future plans? What is the state you target?
One of the advantages we are seeing with the metric-based alerts is that we can fine tune the alert much more than we could with value-based alerts. I recently found an issue when I was browsing through the metrics that we had not received any alerts yet, but I could see that the metric was trending towards us having an issue. We were able to solve the problem before any alerts had come up. I would guess that this alert would have gone off around 2am, not exactly a time when you want to be solving issues. The state we would like to achieve is to consistently have alerts that something is not right before we see that something has actually gone wrong. This happens over time, where we can review the metrics before something bad happened, and then we can create a trend-based alert using those findings.

That sounds like a very modern approach, very helpful and user friendly. Is this what brought Showmax Engineering to your attention? In February 2020, we introduced the new ecosystem that resolved around Prometheus at the P2d2 conference. You were there and met Roman Fišer, Showmax Head of Infrastructure, who gave the presentation. What happened?
I saw that Showmax was one of the sponsors of the conference, and I knew them as a brand in South Africa (where I’m from). I learned at the conference that Showmax actually had their  engineering operations in Prague, and I found this very interesting. I thought it would be cool to visit the office and see what it’s like. I visited on Friday, and there were very few people at the office. But I got to look around and meet the few people who were there. I wasn’t expecting a job interview, but then we had one.

 And the rest is history, as they say. As actively contributing to the team, you helped to evolve the system. Focusing on alert management, what does the logging pipeline look like currently?
Our logging pipeline has been working alright for a long time and is probably not going to change much in the near future. Logs generated from applications and scripts are sent to RabbitMQ, processed by Logstash into Elasticsearch, and queried by Kibana. We had a very simple proof-of-concept setup for some of our system logs, sending them into Prometheus, the same database as our metrics.
This is quite cool, because we can display the logs on the same screen as the metrics, and we can select a time range and filter to see the logs around that time. This is very useful for debugging, because we can see in the logs the moment that an issue occurred and how it affected the metrics. Getting this logging system is something that we would like to focus on in the future.

You have also improved on-call management a lot recently to cut response time. What have you changed exactly?
In our old on-call system, we put the schedule into a CSV file, and then a few Python scripts managed the informing process for the next person who was on call. An internal web page was available to anyone in the support team where they could access the phone numbers of the person who on-call for the week. Now, with Opsgenie, there’s one phone number that can be saved on the phones of every member of the support team. If they need to phone the person on call, Opsgenie will route the call to the phone number of the person on call.

We also use Opsgenie’s schedules to manage the on-call (this is the only way to do it in Opsgenie). At the end of the month, we import the times from the schedule into our tracking system Sloneek, to correctly compensate everyone who was on-call. If someone is suddenly not available to be on-call, overriding the schedule in Opsgenie is a much easier method than we had before with the CSV file. It’s also something we can do from our phones if we need to.

We have also set up Prometheus Alertmanager to send the alerts to Opsgenie so that Opsgenie can alert the person on call directly.

Because of our planned transition to cloud-managed services, some of our improvements will become obsolete. Monitoring, however, still plays a major role in our future plans, and we will keep the best elements of our current approach.

Join us and build a streaming platform used by millions. 

We’re excited to witness the recently announced partnership between Showmax, Sky and NBCU coming to life here in Prague. Following the recent announcement of our partnership; that will bring some of the world’s best content and technology to streaming customers across sub-Saharan Africa, Sky are hiring developers to make it happen.

You might know Sky for TV. But we’re more than that. We’re a tech business. We’re broadband and broadcasting, streaming and sports, mobile and magic. And we’re proud to be part of the Comcast NBCUniversal family, creating incredible technology and entertainment that connect millions of people to the moments and experiences that matter most. 

We’ve already rolled out NBCUniversal’s streaming platform Peacock to our European audiences. Now you have the opportunity to work on an exciting new venture powered by Peacock’s leading, globally scaled technology. 

You will be part of a dynamic team developing client applications and making sure our technology delivers the best experience for our customers.

Craft high-quality software utilizing agile methodologies

Design clean, efficient, and reliable code

Deliver highly scalable and extremely reliable products (our customers demand 99.999% availability)

Collaborate with fellow developers, exchanging knowledge and insights - prepare to learn a lot!

Passionately introduce top-notch technology and software development practices into the team

Participate in the entire development cycle, from analysis and implementation to testing and operation

Embrace operational excellence and support as integral aspects of the development team's responsibilities

Proficiency in native iOS languages, specifically Swift

Familiarity with Design Patterns and Object-Oriented Design principles

Comprehensive knowledge of key iOS architectural components, including Model-View-Controller (MVC), Model-View-ViewModel (MVVM), and Coordinator patterns

Practical knowledge of network protocols, along with an understanding of latency and reliability challenges

Demonstrated experience with multithreading, request queuing, and caching in code

Familiarity with Continuous Integration/Deployment tools and processes

Excellent teamwork, self-motivation, and eagerness to learn

Creative thinking, strong analytical skills, and attention to detail

 Working in an international environment on an impactful globally scaled technology in cross-functional team with an agile approach 

Open and healthy working culture where we care about people as much as about stability of the code  

Work equipment as per your specifications 

Hybrid work, combining working from our office to be closer to the team and working from home 

Work life balance thanks to flexible working hours

Generous free day scheme providing 25 days of holidays + 5 days of on-demand leave (sick days)

Ergonomic office and kitchen full of tasty snacks 

Contribution to Multisport Card or Benefit Card 

Learning opportunities with access to Udemy, O’Reilly, and our internal library of physical books, as well as a budget for attending conferences, including strong support if you want to present at a conference yourself 

Events, meetups, hackathons in the office or elsewhere

 We’re excited to witness the recently announced partnership between Showmax, Sky and NBCU coming to life here in Prague. Following the recent announcement of our partnership; that will bring some of the world’s best content and technology to streaming customers across sub-Saharan Africa, Sky are hiring developers to make it happen.

We’ve already rolled out NBCUniversal’s streaming platform Peacock to our European audiences. Now you have the opportunity to work on an exciting new venture powered by Peacock’s leading, globally scaled technology.  

Our team develops and supports market-leading video streaming services, underpinned by state-of-the-art engineering principles. We do this at huge scale: for millions of customers globally. No matter the device, the time or the place, we make sure that our diverse audiences can easily find and enjoy whatever they want to watch, choosing from the world’s best entertainment, news and sport.

You will join the Online Video Platform team (OVP) within Global Video Engineering & Apps where you will be positioned within business-critical propositions that support all our global streaming initiatives such as NOW, Sky Go, Peacock and SkyShowtime. 

You will be responsible for automating and streamlining tooling for cloud-based services using a range of scripting languages, automation tools, and cloud-native services. You will work with a variety of development / DevOps teams and business partners to ensure that cloud automation activities are secure, reliable, and efficient. We are on the lookout for motivated and outstanding individuals who want to be involved in ground breaking technology that truly pushes the boundaries and growth of streaming! 

Exploring and implementing new ways to automate systems, across a domain that powers business-critical streaming propositions globally.

Design and implement automation strategies, solutions and policies across a range of web API, network and cloud-based technologies.

Automate aspects of deployment and management of cloud-native applications.

Track and fix automation issues and develop solutions

Develop and maintain scripts to automate tasks and processes.

Collaborating with other scrum teams to help support their technical roadmaps by offloading deployment activities into our team workflow and achieving cross-team and department goals through automation.

Provide technical support and training to other team members.

Create and maintain detailed technical documentation.

What skills do you need to play your role well?  

Experience working with REST APIs and multi-distributed systems e.g. Cassandra

Familiarity with configuration and build tools including Jenkins, Gradle, Maven

An understanding of message driven systems e.g. Kafka, Amazon Kinesis, ActiveMQ

Applied knowledge of agile methodologies and software engineering concepts e.g. XP, Scrum, Pair Programming, TDD, BDD

Ability to communicate well with other teams and consult with developers of all levels of experience

 

Working in an international environment on an impactful globally scaled technology in cross-functional team with an agile approach 

We’re one of Europe’s leading entertainment and communication business. We’re always looking to improve by developing pioneering products and services which give every one of our customers experiences they’ll love. 

Join us as an QA Automation Tester and you'll be part of a dynamic team that makes sure our technology delivers the best experience for our customers.

Help to define and manage our testing processes

Participate in the team's agile planning & delivery process

Responsible for ensuring features meet specified requirements

Definition of test plans, test scripts, test data requirements, etc.

Planning test environments and automation strategies.

Preparation of automation frameworks and test runs for API back ends

Preparation of automation frameworks and test runs for frontend clients

What skills do you need to play your role well? 

Strong at programming in at least one of the following languages: Ruby, Python, Java, C# or JavaScript

Experience with CI process and tools (Jenkins, etc.)

Knowledge of networking and HTTP Protocol

Ability to track down the root cause of a defect using debugging tools

Ability to work well in a team, self-driven and eager to learn

Creative thinking, analytical skills, with attention to detail

Millions of customers use our apps and we have always been looking to improve by pushing the boundaries in our products and services providing every one of our customers experiences they love.

Develop excellent quality software using agile techniques

Deliver highly scalable and highly reliable products (yes, our customers expect 99.999% availability)

Collaborate with other developers, sharing and receiving knowledge

Promote best technology and software development practices into the team with passion and enthusiasm

Contribute to the whole development cycle, analysis, implementation, testing and operation  

Operation excellence and support are part of the development teams’ responsibility

Ensure that our applications are compatible with different versions of Android and different screen sizes.

Development in languages such as Java or Kotlin

Experience with Android SDK, Android Studio, Gradle, and Git.

A passion for Android development and a desire to stay up-to-date with the latest trends and technologies.

Strong problem-solving and debugging skills

We're the Online Video Platforms department. The team behind your favourite Sky products, and the platforms that power them. We make every moment magical, everywhere. Our team is made up of self-motivated, big thinkers who have an interest in solving problems and find new ways to captivate millions of customers by putting them at the heart of everything we do! From Sky Glass, Sky Q, Peacock and NOW to news and sports apps, we make entertainment even better and we can’t wait to get started on what’s next!

Lead the design, development and testing for new and existing Java microservice applications.

Promote the use of container technologies such as Docker and Kubernetes

Utilize pair-programming and help mentor and progress junior developers

Chip in or support training sessions to share knowledge with other members of your team/wider department e.g. training workshops, proof of concept overviews, lightning talks

Maintain software delivery principles and guidelines based on internal Sky standards and industry best-practice

Strive for continuous improvement of our systems and ways of working

Participate in research and technical initiatives to improve department technical capabilities.

Good system development knowledge in building scalable applications within Java

Experience working with REST APIs and noSQL Databases e.g. Cassandra

An understanding of systems e.g. Kafka, Amazon Kinesis, ActiveMQ

Familiarity with configuration and build tools such as Jenkins, Gradle, Maven

Familiarity with standard monitoring and logging tools including Prometheus and Grafana

A willingness to collaborate in a cross-team environment.

Knowledge of design patterns and SOLID principles

Practical experience building and working in a CI/CD environment

 Develop new user-facing features and get direct customer feedback on them.

Help your team solve coding or architecture issues and develop best practices.

Experiment, innovate, and introduce new technologies to the team.

Optimize your code, and your team’s code, for performance.

Work with Business Analysts, Project Managers, and other stakeholders to understand features and provide valuable technical feedback on new initiatives – be part of feature development from start to finish. 

Strong JavaScript/TypeScript knowledge and experience with frontend technologies (React)

Ability to work well not only in your team but also in cross-team environment. 

Creative thinking skills and analytical skills, with attention to detail. 

A plus: Knowledge of WebGL and Lightning TV framework in the context of TV and streaming services (awesome if you have this knowledge, but not mandatory).

We are looking for a strong Java Developer who would like to help develop our latest large scale CMS project for Sky OTT International.  Working in an Agile environment, this role will suit an experienced Java Developer who wants to broaden their skill set in an international multilingual CMS application.

Deliver excellent quality software using agile techniques

Assist in developing application deployment, monitoring and automated control artifacts

Strive for continuous improvement – be able to improve existing practices and techniques from within the team with enthusiasm

Assist in the development the CMS application using Java and Spring

Unit, Functional and Performance test code for robustness including edge cases, usability and general reliability. 

Assist in the maintenance and support of the AWS infrastructure, Postgresql Database and Object Caching layers

Actively help the team improve our automated test systems

Extensive experience creating quality Java based applications

Having worked on live high-volume websites

Proficiency in at least one Continuous Integration tool (ideally Jenkins)

Experience in automated testing frameworks

Ability to use a wide variety of open source technologies and cloud services such as AWS

Strong understanding of Spring and Spring Boot

Extensive experience with Agile Principles

Exposure to Content Management and Content Distribution Networks

Knowledge of Unix/Linux and scripting languages (Bash, etc.)

We are a small team of 5 people who are responsible for running Kubernetes clusters and supporting services like CoreDNS, kube-proxy, Cluster Autoscaler and others. We closely cooperate with the other Platform Engineering team called “Databases” to provide a stable platform based on AWS services with focus on self-service capabilities for other teams.

Besides that we are still improving our services, toolchains and workflows to make our working environment as comfortable as possible. Minimizing the amount of repetitive and dull work is our key for easy going work.

We keep oncalls about the production platform and so the stability of the platform is one of our highest prioritiesy. Every incident is investigated very thoroughly to avoid recurrence in the future.

Our whole platform is a very complex environment that is still very active, like nature in spring. We are constantly improving our services, applying the newest practices and keeping our platform up to date to minimize our technical debt.

A Platform Engineer in our team plays with Terraform/Terragrunt and Tanka/Jsonnet on a daily basis to interact with services deployed on AWS or Kubernetes. Our entire platform is aligned with the IaC standard so each change is properly reviewed before it gets merged into the Git repository. We try to minimize the bus factor so we regularly share knowledge across the team to improve the effectiveness of our effort.

About the team

How we work

Read our blog posts #devops

By Showmaxers, for Showmaxers: Is '{JSONB}' a Silver Bullet?

By Showmaxers, for Showmaxers: Linux Containers

By Showmaxers, for Showmaxers: PostgreSQL Terminology

State of the DevOps ecosystem

By Showmaxers, for Showmaxers: Is '{JSONB}' a Silver Bullet?

By Showmaxers, for Showmaxers: Linux Containers

By Showmaxers, for Showmaxers: PostgreSQL Terminology

State of the DevOps ecosystem

Check out our other teams