Every minute, hundreds of hours of movies are being uploaded to different media platforms. For a recommender system to match the right user with the right movie, a certain amount of data is required, e.g. in the form of explicit or implicit ratings provided by users. But what if there is little or no such data available? How to increase the chance of a movie that lacks sufficient information to appear in the right recommendation feed of the right user? The answer may lay in “stylistic” visual features that can be extracted automatically and utilized effectively to give personalized recommendations based on state-of-the-art machine learning algorithms!
Introduced in the mid-1990s, recommender systems have over the years attracted considerable attention in both industry and academic research communities. AI algorithms with the ability to offer personalized recommendations have been used to support users’ decision-making processes on online platforms, and to assist them in discovering relevant media content (e.g. news articles, movie items). While becoming a critical part of our digital lives, and a valuable tool for many businesses, the use of recommendation systems, however, comes with its fair share of challenges.
Common Issue for Media Platforms
There has been a substantial advancement in the research field of recommender systems over the past years, albeit considerable challenges identified by research communities in academia and the media industry are yet to be solved. As an example, when a new user registers to a streaming platform, or a new movie item is introduced to the movie catalogue, the recommendation engine may encounter problems in recommending the new item to both new and existing users. The phenomenon is commonly referred to as cold start, and it has long captured the attention of researchers and students who attempt to provide solutions to overcome it.
Among the latter is David Kvasnes Olsen, a final year master’s student at the Department of Information and Media Studies (InfoMedia) at the University of Bergen (UiB). Under the supervision of MediaFutures´ WP2 leader Mehdi Elahi and industry co-leader Lars Skjærven from TV 2, Olsen has investigated whether the use of visual analysis of movies can aid in finding visual similarities and using them in generating recommendations. Such type of data can be representative of the movie style and can be automatically analysed and extracted from the movie files. In addition to mitigating the “new item” problem, visual analysis of movies could potentially lead to more diverse movie recommendations insofar that it provides the users of media platforms with new ways of accessing a broader spectrum of media content, e.g., by finding lesser known, but equally relevant movies that might be difficult to discover otherwise. It can better address the needs of platform users, hence increase their satisfaction. This is an important goal of recommendation systems, and one of the MediaFutures´ WP2´s objectives.
Potential Solution based on Visual Analysis
“There are two main techniques used to recommend movies to users: Collaborative Filtering and Content-based Filtering. The former revolves around the similarity between user preferences, while the latter focuses on similarities between movies. The common denominator for both of these techniques is the data requirement regarding the users or the movies. Nowadays loads of movies that are being uploaded to platforms don’t have any user interaction, or suffer from a lack of item metadata such as a description or a genre tag”, explains Olsen.
In his master’s thesis, Olsen aims to provide a solution to such difficult recommendation scenarios by building a movie recommender system that generates recommendations based on automatically extracted “stylistic” visual features. The master’s student further explains:
“These features can be low level features such as colorfulness or brightness, as well as higher level features like object and emotion detection. It is important to specify that while content features (e.g. tags and genre) used in recommender systems require a person to generate them, visual features can be extracted automatically. In addition to being cheaper, an automated approach may alleviate the cold start problem that arises when other content features are missing or are insufficient”.
A Successful Collaboration with TV 2
TV 2 plays a crucial role in Olsen’s master´s thesis project. The Norwegian broadcaster supplied the InfoMedia student with movie posters and trailers from their streaming service TV 2 Play for analysis. In collaboration with TV 2, Olsen integrated his analysis tool into the production piplines at TV 2, facilitating prototype testing of the algorithm and exposing the results to end-users of TV 2 Play.
“The fact that we get the chance to test the recommender system we developed based on the extraction of visual features on a platform belonging to a national commercial actor is very exciting, and something that has never been done before!” adds Olsen.
The master project also yields auspicious results. Olsen elaborates:
“After running offline evaluations comparing pure collaborative filtering to hybrid recommender systems using visual features we can see promising results in accuracy metrics. The hybrid recommender using visual features also seems to outperform collaborative filtering in terms of novelty (recommending less popular movies) in cold start situations.”
The master thesis and the code will be available for access through the communication channels of the MediaFutures center.
Photo – Left to Right: Master student David Kvasnes Olsen, Prof. Mehdi Elahi, TV 2 industry co-leader Lars Skjærven. Phot credit: Anna Pacholczyk
Movie posters illustration: TV 2 Play