Earlier this year, Šimon Koudelka, Showmax Product Manager, presented our approach to continuous research at the WebExpo conference. For a while now, continuous research has been done at Showmax by non-researchers – product managers and designers – who are helping the regular UX research team by collecting some insights on their own. This alternative approach, though it might seem risky or inadequate, helps us develop our apps faster and respond quickly to customers’ needs, act on hiccups in the user journey, and more.

The talk that you can watch in the video below introduces examples of how we collect insights, create opportunity maps, and why we think it’s important to involve developers in the research. You can learn how to adopt our approach at your organization if you find it helpful.

<div id="presentation-embed-38984442"></div>
<script src="https://slideslive.com/embed_presentation.js"></script>
<script>
  embed = new SlidesLiveEmbed("presentation-embed-38984442", {
presentationId: "38984442",
autoPlay: false,
verticalEnabled: true,
});
</script>

Compared to evaluative research, which answers specific problems often related to usability, continuous research belongs to the field of generative research, where you explore broader topics. Generative research helps to better understand users and is great at finding weird patterns; things the users do which you wouldn’t expect. For example, we‘ve learnt that users used the continue watching row as a substitute for the Watchlist. Users simply hit the play button and immediately exit the player to find the film later in the continue watching row. It works, but it’s a bit complicated and it reveals that there is probably something wrong with the Watchlist feature.

At Showmax, we work in a hybrid model, which means even though we have the UX research team, there are more people who can conduct user tests or customer interviews. Each product team talks to two users every week. Interviews are conducted remotely as our users are located in Africa, while product development takes place in Europe. Product teams usually have a stable script, which is related to a business objective they are currently focusing on. Quite often developers attend these interviews as well, as they enjoy watching users tinker with our app. We always do a debrief, where we discuss the learnings and takeaways and discuss the potential next steps.

Thanks to the frequent customer interviews, we’ve discovered the topic of rewatching. Some of our users rewatch the same films or episodes many times. This is particularly true about the kids’ content. For example, one of our young users watched a single episode of Peppa Pig nineteen times within seven hours! We dug deeper and found out that seventeen percent of our users rewatched something in the past month. When you learn such an insight, then you can place it onto the opportunity map (a concept lately championed by Teresa Torres) and start investigating what potential the topic has to help you reach the company goals.

We move fast thanks to the frequency of the customer interviews. Product teams learn something new every week. The credibility of product managers and designers increases as they can now easily show how well they know their users. Furthermore, the involvement of developers helps them get better context for their work.

Our approach takes time as it requires training of product managers and designers, so that they can conduct the interviews independently. However, product managers and designers can help researchers, who wouldn’t be able to cover all the opportunities themselves. With product teams involved in the process, researchers can finally free their hands of the day-to-day operations and focus on the high-value strategic deep dives.


Are you eager to join our Product management team and would you like to keep people engaged on the Showmax web and SmartTV apps? Read more and apply.

Continuous Research: When Non-Researchers Take the Lead

To deliver the best content to the right users we utilize asset metadata - movie or series features like genre, length, stars, etc. - but that has some limitations to it. The set of possible tags for assets is always finite, and adding extra data manually costs time and money.

To improve things further, we use viewing behavior to generate some more features for the assets - at virtually no extra cost. In fact, we created an algorithm that converts assets to vectors based solely on views, which gives us a new set of features to better describe them.

Our algorithm is based on the generic node2vec model that converts nodes in a graph into vectors. The idea is that nodes that are close to each other in the graph should be close to each other in the vector space. In many cases, the shortest path between two nodes could be a good definition of the closeness between them. In cases when the graph is so dense that almost every pair of nodes has an edge, one can use weights for edges - the higher the edge weight, the closer the nodes are.

One of the famous implementations of the node2vec models is the word2vec model presented by Tomáš Mikolov and associated researchers in 2013. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text, and represents each distinct word with a particular vector. The closeness of two words is given by the number of occurrences of the pair in the same context (within n words apart in the same text).

Most neural network models require structured data (e.g. matrices) for training , and graphs do not provide that. One way to overcome this - also, the one we decided to take - is to generate data points using random walk on the graph. One can start at a random node and, from there, sample an edge to get to an adjacent node. If the graph has weighted edges, the weights can be used to calculate transition probabilities.

That way, you generate a pair of origin and target nodes. To be able to train the model, you also need to generate some negative samples - randomly-selected nodes other than the two that have already been selected. You can then mix the target node with the negative samples. Note that this step is completely optional; sometimes we talk about neural networks in the context of artificial intelligence, but this network cannot even figure out that the target node is always the first one :).

The model input is the index of the origin node and indices of the target node together with the negative samples. The ground truth for output is an array with zeros on positions of the negative samples and one on position of the target node.

The inputs and outputs are fed into a neural network with just three layers. First is an embedding layer that is shared between the origin node and the other nodes. It’s literally just a matrix of weights from which the model takes the corresponding row (vector) for a given asset. The next layer calculates dot product between the origin node and all other nodes. The resulting array is fed to a softmax layer that produces numbers between 0 and 1 that are compared with the ground truth. The error is propagated back to the weights in the embedding matrix.

In our case, we don’t search for vectors representing words, we search for vectors representing each of the movies and series in our library. Another difference in our case is that our graph is bipartite. That means that we have two types of nodes - assets and users - with the edges between them representing the user watching the asset. That leads to two possible solutions for how to generate training data for the model.

The first one is to use the bipartite graph as it stands with a two step random walk. We start at a random asset and do a step to a random user that watched the asset. Then we do the second step to another asset that the user also watched (excluding the original asset). This leads to a straightforward generalization of the original idea. But the graph size is proportional to the product of number of users and number of assets which is too big for us to work with.

Another solution is to use a weighted graph with assets only. The edge weights are equal to the number of users that watched both of the assets. Then we can get back to the one-step random walk using probabilities proportional to the edge weights. The graph size is “only” quadratic in number of assets, which is much easier to work with. Both methods lead to very similar results, so we picked the one that fit into memory.

Using this method, we generate one data point. Obviously, we need much more. So we generate dozens or hundreds of data points that we use for multiple training epochs to maximize the utilization of the generated data. Then we can generate another dataset for the next season (that’s what we call epochs on the same training set), and repeat the process. After a few seasons, the embedding matrix contains vectors that represent the assets.

The vectors themselves cannot be used for any content recommendation, and require some model on top of them to be any useful. As we mentioned above, the vectors can be used as a set of extra features that can enhance assets in any existing recommendation engine. Apart from that, we came up with a few other applications:

Calculate vectors for all users by aggregating vectors of assets that they watched — the users’ vectors in this case represent their taste or preferences. Then we can recommend assets to users based on dot products between their vectors and the vectors of the assets. Note that the vectors don’t tell much about the quality or popularity of the asset, so this should be done only with a pre-selected set of assets.

We can use it to assign metadata to assets or find issues with existing metadata. For example, if an asset has neighbors (in terms of dot product between their respective vectors) that are all historical romances, it is expected that the asset itself is also a historical romance.

Make clusters of similar assets that work both for content recommendations and further analyses — it greatly helps us understand the types of content we have on our platform.

We started this as a small and fun project just to see whether something like this was even possible. It turned out to be a great model, and whenever you enter the Showmax app you’ll see content that has been ordered by it. We’re still exploring other ways that it can help us with our daily tasks.

“What to watch recommendation” and our Film2vec model

When software products fail, they do so because they fail to deliver the intended impact. One problem is that teams don’t tackle risks upfront. Will users choose to use the feature? Will users know how to use the feature? Will the feature deliver the expected business results?

Assumption testing helps to answer these questions before the actual development starts. Running these tests might slow things down a bit, but ultimately it saves time — since you only ship validated ideas, there is less throwaway work.

Our team increased trailer playbacks by 44x on the Showmax website app within a few releases. This did not happen by chance. It was based on a deliberate, careful approach of running a series of mini-tests to verify underlying assumptions.

From user feedback, we knew that content discovery — finding a film or series to watch — was a pain point for many of our subscribers. Here’s one example:

Can Showmax show previews or a written summary? Sometimes you would like to know what the movie is all about before watching it. That would be nice.

The information about films and series was too hidden, too buried in the lower levels of the app — users had to do a bit of work to find it. At the same time, we learned from user interviews that trailers played a major role in their deciding what to watch. Moreover, data analysis revealed that there could be a link between trailer playbacks and engaged viewing.

Would more accessible trailers drive engagement?

Our hypothesis was that making trailers easier to watch would drive engagement. We ran a simple A/B test by adding a trailer play button on the homepage that would show on hover. Daily trailer playbacks increased more than 3x.

More importantly, our engagement metrics also showed significant improvement. We were on the right track. The assumption that making trailers more accessible would make users watch more films and series was verified.

When we saw that the new feature delivered the expected positive results, it energised us to come up with a new, upgraded feature — trailer autoplay on hover. The implementation of the trailer button was easy. If it hadn’t worked, we would’ve written off a mere few hours of development. However, there were many user-related risks involved for the more complex autoplay feature.

Would users care about increased data usage?

We assumed that some users might be worried about data usage. In Africa, not everyone has unlimited fibre at home. We ran a questionnaire with a simple question:

Would you have any concerns about a preview playing automatically?

As it turned out, only a small fraction of website users worried about data. Even asking directly about data in the follow up question raised only minor concerns. To minimise data consumption and reduce this usability risk further, we decided to use low resolution for the trailers.

In the older version of the Showmax app, clicking on the poster would take the user to the film detail. Now, the poster turns into the trailer playback window, so we have to offer another way to get to the asset detail information. Our designer proposed two new routes: a dedicated button, and clicking on the poster background. Both of them would show more information about the film.

Not everyone on the team was convinced that users would understand the new navigation. To test the hypothesis, we used Maze, a tool that allows you to ask interactive questions and get answers quickly. The heatmap showing user clicks convinced us that most users would find their way without any difficulties.

Furthermore, the old design made it unclear whether a poster represented a film or series. The obvious solution would be to explicitly indicate the asset type. However, following the classical principle that good design involves as little design as possible, we assumed that showing the asset length would be enough.

“8 seasons” shows implicitly that you’re about to watch a series, whereas “1h 53m” points to a film.

Again, we tested the understanding in the Maze app. With 84% repondees spotting the difference immediately, we gained confidence that the proposed solution would prove sufficient.

Finally, one usability risk remained unanswered. The local internet speeds may not have been fast enough for the autoplay, delaying playback. To address this assumption, we asked engineers for help.

At Showmax, we believe that involving developers in product discovery, before the actual development, is vital. This proved invaluable when tweaking the trailer startup time. While using internet speed throttling tools, we tested numerous resolutions. On top of that, we tinkered with the backend a bit. Every time the app initiates a playback, it also calls a few APIs. Thanks to the engineers on our team, we realized that not all of the API calls were necessary to play trailers. Trimming the calls to the bare minimum further reduced the time it required to launch trailers.

Upon the launch of the trailer autoplay feature, we observed a massive adoption. A few more iterative releases later, we achieved a 44x growth of trailer playbacks. More importantly, engagement metrics grew significantly as well.

Albert Einstein said,”Assumptions are made, and most assumptions are wrong.” And that’s what often happens in software. Don’t throw darts in the dark by developing expensive MVPs and hoping that the A/B test will turn out well — that is often a costly exercise. Test assumptions first, develop later.

Assumption Testing: Don’t Throw Darts in The Dark

Whether you want it or not, the long term use of manual testing with in-house resources usually leads to a point where you are blind to your own bugs. Because of this, we looked for - and found - an opportunity to test our applications from the outside. Tesena and their Klikaton event were perfect. Specifically, Showmax wanted to test our two native mobile applications - Android, written in Kotlin, and iOS, written in Swift.

A clickathon is a public event in which anyone interested in bug hunting in other applications can participate. Usually, these are experienced testers who genuinely enjoy the testing process, but the person who finds the most bugs wins, so there’s a competitive element as well.

Tesena holds events like these often, and they are usually free of charge - the only investments are time and some decent prizes for the winners. In our case, we had gift certificates for Alza.cz (a large electronics e-shop) for the top three testers.

The bug hunts typically have about 15 people testing over the course of two hours. During the event, the development team sits in a different room and evaluates the severity of the discovered bugs in real time. Developers assign points to individual bugs, and the tester with the most points at the end wins.


In total, 13 testers attended the event, so we had to prepare 13 devices apiece with iOS and Android OS. Due to the limited number of devices with iOS, some participants got an iPhone and some an iPad.

Tesena helped us prepare test accounts that gave the testers full access to the application so they could fully-utilize every tool at their disposal. We decided that the participants would test against our testing environment, mainly because it’s easier to set everything up there as opposed to in a production environment.

We prepared accounts, devices, and wifi access, but that was still not enough. The next step was to figure out how to upload the build into devices that did not belong to us. For iOS, we chose to generate a public link from iTunes Connect, shared this link via the notes, and sent it to the devices via AirDrop. From there, we opened the link, installed TestFlight, and then our application.

For Android, we used Beta Crashlytics to distribute the build. All we had to do was forward the email with the invitation to the build and install Beta Crashlytics, which then allows the download of the test build from it.

Because I also planned to take pictures at the event (for this post, in fact), we had our lawyer draw up a consent document for the protection of personal data under GDPR.

After two hours of testing, it was time to evaluate. The bugs had been reported into a shared Google Sheet, which was then analyzed by our designers and developers. Testers did not just report mistakes - they also suggested some improvements some things.

For both applications, the issues were with things like the search bar, where people were expecting different behavior and/or results. The second most common problem was with filters in the Sports section. We discussed both things with our designers and are working on plans to improve them.

In the end, testers found 173 issues in our applications over the course of two hours - 88 on Android and 85 on iOS.

Because our new features were not 100% ready for testing, we didn’t get the results we expected. Instead, they showed us more general shortcomings in the application, and that seeing the app for the first time does not mean one understands how it works - things that are clear to us are not necessarily clear for new users. The lesson is clear and applicable for anyone building anything - it’s good to remember that, even in the clearest of cases, a normal user may not grasp the functionality of what you’ve built. If the testers, who are generally pretty experienced users of tools like this, were unclear on things, it stands to reason that that effect would only be magnified across a population of regular consumers.

The testers’ findings with regards to search was amplified by a mismatch in the search functionality between Android and iOS. On iOS, the search function works after three characters, but on Android it starts immediately after entering just one character. Both applications also display the search results differently, which further confused the testers because they had both platforms and could directly compare them. For example, when they searched for “Intouchables” using the “Int” string, it did not appear in the top position. Instead, there were films with “int” within the name, like “Sintel” or “Match Point”.

Another commonly-reported issue was the use of a filter in the Sports section where, when sorting for events “today”, the results also displayed upcoming events.

No one likes to see issues with something they built, but we were pleased that our external testers did not find any serious bugs that were previously unknown to us. This told us that we have a good team, and that our processes, test plans, and test scenarios are well-designed.

Of course, a huge thanks goes out to our testers. We’re now more aware of how new users can feel about our app and why some may be dissatisfied. Congratulations to the winners and thank you all for your participation and help. A special thanks also goes to Tesena - Without them this never would have happened.

Join us and build a streaming platform used by millions. 

We’re excited to witness the recently announced partnership between Showmax, Sky and NBCU coming to life here in Prague. Following the recent announcement of our partnership; that will bring some of the world’s best content and technology to streaming customers across sub-Saharan Africa, Sky are hiring developers to make it happen.

You might know Sky for TV. But we’re more than that. We’re a tech business. We’re broadband and broadcasting, streaming and sports, mobile and magic. And we’re proud to be part of the Comcast NBCUniversal family, creating incredible technology and entertainment that connect millions of people to the moments and experiences that matter most. 

We’ve already rolled out NBCUniversal’s streaming platform Peacock to our European audiences. Now you have the opportunity to work on an exciting new venture powered by Peacock’s leading, globally scaled technology. 

You will be part of a dynamic team developing client applications and making sure our technology delivers the best experience for our customers.

Craft high-quality software utilizing agile methodologies

Design clean, efficient, and reliable code

Deliver highly scalable and extremely reliable products (our customers demand 99.999% availability)

Collaborate with fellow developers, exchanging knowledge and insights - prepare to learn a lot!

Passionately introduce top-notch technology and software development practices into the team

Participate in the entire development cycle, from analysis and implementation to testing and operation

Embrace operational excellence and support as integral aspects of the development team's responsibilities

Proficiency in native iOS languages, specifically Swift

Familiarity with Design Patterns and Object-Oriented Design principles

Comprehensive knowledge of key iOS architectural components, including Model-View-Controller (MVC), Model-View-ViewModel (MVVM), and Coordinator patterns

Practical knowledge of network protocols, along with an understanding of latency and reliability challenges

Demonstrated experience with multithreading, request queuing, and caching in code

Familiarity with Continuous Integration/Deployment tools and processes

Excellent teamwork, self-motivation, and eagerness to learn

Creative thinking, strong analytical skills, and attention to detail

 Working in an international environment on an impactful globally scaled technology in cross-functional team with an agile approach 

Open and healthy working culture where we care about people as much as about stability of the code  

Work equipment as per your specifications 

Hybrid work, combining working from our office to be closer to the team and working from home 

Work life balance thanks to flexible working hours

Generous free day scheme providing 25 days of holidays + 5 days of on-demand leave (sick days)

Ergonomic office and kitchen full of tasty snacks 

Contribution to Multisport Card or Benefit Card 

Learning opportunities with access to Udemy, O’Reilly, and our internal library of physical books, as well as a budget for attending conferences, including strong support if you want to present at a conference yourself 

Events, meetups, hackathons in the office or elsewhere

 We’re excited to witness the recently announced partnership between Showmax, Sky and NBCU coming to life here in Prague. Following the recent announcement of our partnership; that will bring some of the world’s best content and technology to streaming customers across sub-Saharan Africa, Sky are hiring developers to make it happen.

We’ve already rolled out NBCUniversal’s streaming platform Peacock to our European audiences. Now you have the opportunity to work on an exciting new venture powered by Peacock’s leading, globally scaled technology.  

Our team develops and supports market-leading video streaming services, underpinned by state-of-the-art engineering principles. We do this at huge scale: for millions of customers globally. No matter the device, the time or the place, we make sure that our diverse audiences can easily find and enjoy whatever they want to watch, choosing from the world’s best entertainment, news and sport.

You will join the Online Video Platform team (OVP) within Global Video Engineering & Apps where you will be positioned within business-critical propositions that support all our global streaming initiatives such as NOW, Sky Go, Peacock and SkyShowtime. 

You will be responsible for automating and streamlining tooling for cloud-based services using a range of scripting languages, automation tools, and cloud-native services. You will work with a variety of development / DevOps teams and business partners to ensure that cloud automation activities are secure, reliable, and efficient. We are on the lookout for motivated and outstanding individuals who want to be involved in ground breaking technology that truly pushes the boundaries and growth of streaming! 

Exploring and implementing new ways to automate systems, across a domain that powers business-critical streaming propositions globally.

Design and implement automation strategies, solutions and policies across a range of web API, network and cloud-based technologies.

Automate aspects of deployment and management of cloud-native applications.

Track and fix automation issues and develop solutions

Develop and maintain scripts to automate tasks and processes.

Collaborating with other scrum teams to help support their technical roadmaps by offloading deployment activities into our team workflow and achieving cross-team and department goals through automation.

Provide technical support and training to other team members.

Create and maintain detailed technical documentation.

What skills do you need to play your role well?  

Experience working with REST APIs and multi-distributed systems e.g. Cassandra

Familiarity with configuration and build tools including Jenkins, Gradle, Maven

An understanding of message driven systems e.g. Kafka, Amazon Kinesis, ActiveMQ

Applied knowledge of agile methodologies and software engineering concepts e.g. XP, Scrum, Pair Programming, TDD, BDD

Ability to communicate well with other teams and consult with developers of all levels of experience

 

Working in an international environment on an impactful globally scaled technology in cross-functional team with an agile approach 

We’re one of Europe’s leading entertainment and communication business. We’re always looking to improve by developing pioneering products and services which give every one of our customers experiences they’ll love. 

Join us as an QA Automation Tester and you'll be part of a dynamic team that makes sure our technology delivers the best experience for our customers.

Help to define and manage our testing processes

Participate in the team's agile planning & delivery process

Responsible for ensuring features meet specified requirements

Definition of test plans, test scripts, test data requirements, etc.

Planning test environments and automation strategies.

Preparation of automation frameworks and test runs for API back ends

Preparation of automation frameworks and test runs for frontend clients

What skills do you need to play your role well? 

Strong at programming in at least one of the following languages: Ruby, Python, Java, C# or JavaScript

Experience with CI process and tools (Jenkins, etc.)

Knowledge of networking and HTTP Protocol

Ability to track down the root cause of a defect using debugging tools

Ability to work well in a team, self-driven and eager to learn

Creative thinking, analytical skills, with attention to detail

Millions of customers use our apps and we have always been looking to improve by pushing the boundaries in our products and services providing every one of our customers experiences they love.

Develop excellent quality software using agile techniques

Deliver highly scalable and highly reliable products (yes, our customers expect 99.999% availability)

Collaborate with other developers, sharing and receiving knowledge

Promote best technology and software development practices into the team with passion and enthusiasm

Contribute to the whole development cycle, analysis, implementation, testing and operation  

Operation excellence and support are part of the development teams’ responsibility

Ensure that our applications are compatible with different versions of Android and different screen sizes.

Development in languages such as Java or Kotlin

Experience with Android SDK, Android Studio, Gradle, and Git.

A passion for Android development and a desire to stay up-to-date with the latest trends and technologies.

Strong problem-solving and debugging skills

We're the Online Video Platforms department. The team behind your favourite Sky products, and the platforms that power them. We make every moment magical, everywhere. Our team is made up of self-motivated, big thinkers who have an interest in solving problems and find new ways to captivate millions of customers by putting them at the heart of everything we do! From Sky Glass, Sky Q, Peacock and NOW to news and sports apps, we make entertainment even better and we can’t wait to get started on what’s next!

Lead the design, development and testing for new and existing Java microservice applications.

Promote the use of container technologies such as Docker and Kubernetes

Utilize pair-programming and help mentor and progress junior developers

Chip in or support training sessions to share knowledge with other members of your team/wider department e.g. training workshops, proof of concept overviews, lightning talks

Maintain software delivery principles and guidelines based on internal Sky standards and industry best-practice

Strive for continuous improvement of our systems and ways of working

Participate in research and technical initiatives to improve department technical capabilities.

Good system development knowledge in building scalable applications within Java

Experience working with REST APIs and noSQL Databases e.g. Cassandra

An understanding of systems e.g. Kafka, Amazon Kinesis, ActiveMQ

Familiarity with configuration and build tools such as Jenkins, Gradle, Maven

Familiarity with standard monitoring and logging tools including Prometheus and Grafana

A willingness to collaborate in a cross-team environment.

Knowledge of design patterns and SOLID principles

Practical experience building and working in a CI/CD environment

 Develop new user-facing features and get direct customer feedback on them.

Help your team solve coding or architecture issues and develop best practices.

Experiment, innovate, and introduce new technologies to the team.

Optimize your code, and your team’s code, for performance.

Work with Business Analysts, Project Managers, and other stakeholders to understand features and provide valuable technical feedback on new initiatives – be part of feature development from start to finish. 

Strong JavaScript/TypeScript knowledge and experience with frontend technologies (React)

Ability to work well not only in your team but also in cross-team environment. 

Creative thinking skills and analytical skills, with attention to detail. 

A plus: Knowledge of WebGL and Lightning TV framework in the context of TV and streaming services (awesome if you have this knowledge, but not mandatory).

We are looking for a strong Java Developer who would like to help develop our latest large scale CMS project for Sky OTT International.  Working in an Agile environment, this role will suit an experienced Java Developer who wants to broaden their skill set in an international multilingual CMS application.

Deliver excellent quality software using agile techniques

Assist in developing application deployment, monitoring and automated control artifacts

Strive for continuous improvement – be able to improve existing practices and techniques from within the team with enthusiasm

Assist in the development the CMS application using Java and Spring

Unit, Functional and Performance test code for robustness including edge cases, usability and general reliability. 

Assist in the maintenance and support of the AWS infrastructure, Postgresql Database and Object Caching layers

Actively help the team improve our automated test systems

Extensive experience creating quality Java based applications

Having worked on live high-volume websites

Proficiency in at least one Continuous Integration tool (ideally Jenkins)

Experience in automated testing frameworks

Ability to use a wide variety of open source technologies and cloud services such as AWS

Strong understanding of Spring and Spring Boot

Extensive experience with Agile Principles

Exposure to Content Management and Content Distribution Networks

Knowledge of Unix/Linux and scripting languages (Bash, etc.)

We’re an international mix of product managers with different backgrounds and skill sets, working together with our engineering and design teammates. Each PM has their own area of responsibility and together with their team they work on improving that space.

On the front end of things, we have PMs responsible for the apps (mobile, web, various TVs), some who are responsible for a specific part of the experience (Sports, personalized recommendations), and others who focus on the signing up and payments of new customers.

We enjoy reasonable freedom to identify problems and validate solutions through product discovery, but being part of a multinational corporation means that every now and then we need to align with the rest of the group and prioritize together.

3298427078.mp4

The PMs are tightly aligned with their counterparts in their respective teams (primarily engineering and design, but there are other key roles as well). This is their “family”, this is where they spend most of their time. 

Product discovery is key. We consider all our ideas to be assumptions that need to be validated.

We interview our customers on a regular basis, the front end teams do so weekly.

We’re making sure we’re best friends with the analytics team.

Most changes are shipped as an A/B test or at least incrementally rolled out.

We plan quarterly and in the form of topics, ideally with measurable results. Specific features are what comes out of discovery, not planning. There are exceptions to this and we’re still finding ways to engrain this into the whole company.

About the team

How we work

Read our blog posts #product

Continuous Research: When Non-Researchers Take the Lead

“What to watch recommendation” and our Film2vec model

Assumption Testing: Don’t Throw Darts in The Dark

External Testing with Tesena

Continuous Research: When Non-Researchers Take the Lead

“What to watch recommendation” and our Film2vec model

Assumption Testing: Don’t Throw Darts in The Dark

External Testing with Tesena

Check out our other teams