2 Anatomy of a recommender system
Recommender systems are best understood as more than a single black box that spits out a ranked list. They are systems built around users and items, where the “items” can be posts, songs, videos, ads, or even other users. The chapter distinguishes recommendation from search by noting that search responds to an explicit query, while recommendation responds to an implicit one: what seems relevant to a person right now. It also emphasizes that platforms make decisions throughout the process, so the content people see is shaped by design choices as well as algorithms.
The recommendation pipeline is broken into several stages: candidate generation, filtering, ranking, post-processing, delivery, and recording feedback. Candidate generation narrows a huge universe of possible content down to a manageable pool so the system can respond quickly; filtering removes blocked, unsafe, illegal, or otherwise unwanted content; and ranking uses machine learning to order the remaining items by predicted value or engagement. Different engagement goals can be weighted differently, which means the ranking formula reflects the platform’s priorities. Post-processing can smooth out awkward results, such as too many items from the same creator in a row, and delivery plus behavioral logging closes the loop by turning user responses into future training data.
The chapter then shifts to how the ranking model itself is trained and represented. Early heuristic or content-based approaches depended on manual labels and did not scale well, especially for user-generated content. Data-driven methods, especially collaborative filtering and deep learning, solve this by learning from interactions such as clicks, likes, dwell time, and watch completion. Central to modern systems are embeddings: compact numerical representations of users and items that let the platform compare them in a shared space and recommend the closest matches. This approach is powerful and scalable, but it is also opaque, resource-intensive, and shaped by the platform’s values, business goals, and governance decisions.
A bird’s-eye overview of a recommender system. Recommendations are not the output of a single operation; rather, they are the results of multiple interlinked operations. Candidates are generated from the universe of all possible content—for example, by picking posts from users that are followed. Later, candidates are filtered to restrict the pool to a more manageable number. At this stage, the content gets actively sorted by the Heavy Ranker in accordance with a predefined metric. Such a ranked list can be adjusted, if needed, before being delivered to the users, whose engagement is recorded. Platform policies influence the filtering and post-processing steps, for example, by limiting the amount of content from the same author.
Two different kinds of engagement are predicted: A first model predicts the probability of a Like, and another one, the probability of Reply. Each item is thus assigned two different scores.
Two items that have been scored by two different models: one to predict the probability of liking the post and the other predicting the probability of replying. The two scores must be coalesced together before a unique list can be created.
The two items from before are combined into a single list by assigning a high coefficient (10) to the Like score and a lower one (1) to the Reply score.
The same two items from before, ranked using a different formula. In this case, more weight is given to the probability of replying, obtained by multiplying the predicted score by a high number.
A third ranking of the two original items. In this case, to obtain the final scoring, a negative coefficient is assigned to the Liking score, and a positive one to the Reply score. The resulting list will penalize items with a higher predicted probability of being liked.
An example of a user-item matrix for collaborative filtering. Each cell records the rating of each user for the corresponding item. Note that, in general, users don’t interact with all the items.
An example of a Spam classifier. Each message is examined to determine whether it’s legitimate or spam.
The dependency graph between raw and combined scores can be non-linear and complicated. In this case, we can see how Combined Score B is dependent on both Raw scores C and D, whereas Combined Score A is dependent on Raw scores A, B, and C. Note how Raw score A influenced Combined Score C in two different ways: directly and indirectly via Combined Score B.
Derived score A is used both in Model A and Heavy Ranking. Note that Heavy Ranking is also dependent on Model A.
When representing users and items with a single number, they can be thought of as points on a line. The recommendations will be ordered by their distance on the line. In this case, the User is represented with a 0.8; therefore, the items will be ordered as Item 3, Item 2, Item 1.
Two-dimensional representation of users’ and items' embeddings. We represent users and items as two numbers: one for cat affinity and one for dog affinity. Such numbers can also be used as coordinates to represent them on the plane. User A has a high affinity for both animals, so they are in the top-right corner, while user B is neutral and close to the center. The items recommended for them are those that minimize the distance to their respective embeddings.
Summary
- Recommender systems help users find the right items among a predefined list.
- While we are used to thinking about RecSys as a monolith, they are comprised of different steps.
- The candidate-generation and filtering steps reduce the pool of potential items to rank to a few hundred.
- The heavy ranking is the crucial step, where this abridged list gets ordered by order of value to the users—usually, the predicted probability of engagement.
- Before delivering the list to the users—and registering their interaction with it—a post-processing step might be desirable, for minor reordering, including increasing the diversity of content.
- Recommender systems that use machine-learning methods are more powerful than rule-based ones, since they require little manual intervention and can adapt to changes, such as users’ shifting tastes. At the same time, they are more complex and less explainable.
- The design of all items of the platforms contributes to algorithmic amplification, not just the main feed.
- Under the hood, recommendations are created by selecting the items that are the closest to a specific user, by comparing their mathematical representations.
FAQ
```htmlWhat is a recommender system in simple terms?
A recommender system takes a user’s request and returns a ranked list of items that are likely to be relevant or useful to that user.
How is a recommender system different from a search engine?
Search engines respond to an explicit query typed by the user, while recommender systems work without an explicit query and instead infer what might be relevant at that moment.
What do “user” and “item” mean in recommender systems?
The user is the person asking for recommendations, and the item is any piece of content or entity being ranked, such as posts, songs, movies, or even other users.
What are the main steps in a recommender system?
The main steps are candidate generation, filtering, heavy ranking, post-processing, delivering the results, and recording user behavior for feedback.
Why is candidate generation necessary?
Candidate generation narrows a huge pool of possible items down to a manageable number so the system can respond quickly enough for users, usually within a few hundred milliseconds.
What kinds of content can be filtered out before ranking?
Filtering can remove blocked or unwanted content, illegal material, NSFW content, and other items that are not appropriate or not worth ranking.
What is heavy ranking?
Heavy ranking is the step where the candidate list is ordered by predicted value or engagement, often using machine learning models that estimate actions like likes or replies.
How do platforms decide what engagement matters most?
Platforms assign weights to different predicted actions, such as likes or replies, and those weights reflect the platform’s priorities or value model.
Why do recommender systems use machine learning instead of manual rules?
Machine learning is used because manual rules do not scale well, while historical interaction data can help the system discover patterns and predict what users are likely to value.
What are embeddings in a recommender system?
Embeddings are vector representations of users and items that place them in the same mathematical space, allowing the system to recommend items that are closest or most similar to a user.
Hidden Influences ebook for free
