2 Anatomy of a recommender system
Recommender systems are presented as more than a single algorithmic action: they are pipelines that take a user’s implicit request for something relevant and return a ranked set of items. The chapter clarifies the basic language of recommendation by distinguishing users, items, creators, and consumers, and by contrasting recommender systems with search, where a user supplies an explicit query. It also notes that recommender systems and search operate over a predefined inventory of items, unlike generative systems that can invent plausible but nonexistent results.
The internal structure of a recommender system is broken into a sequence of stages: candidate generation, filtering, heavy ranking, post-processing, delivery, and recording of feedback. Candidate generation narrows the huge universe of possible content to a manageable pool using signals such as recency, network connections, popularity, or ads targeting, while filtering removes content that is blocked, offensive, illegal, or otherwise undesirable. Heavy ranking then uses machine learning to predict engagement and combine multiple objectives, with the weights assigned to likes, replies, or other actions reflecting the platform’s own values and business priorities. Post-processing can smooth awkward patterns, reduce fatigue, or boost or suppress certain content, showing that design choices shape what users see at every step.
The chapter then explains how recommender systems are trained and improved through data. Early heuristic and content-based approaches relied on hand-labeled information but did not scale well, especially for user-generated content, so platforms turned to data-driven methods such as collaborative filtering and deep learning. These methods learn from explicit signals like ratings and likes, as well as implicit signals like clicks, dwell time, and completion rates, using them to predict value and optimize ranking. A key concept is the use of embeddings, which represent users and items as vectors in a shared space so that similar users and items are close together; recommendations are made by finding the nearest items. The chapter closes by emphasizing that this entire system is not neutral: platform design, model objectives, and feedback loops all influence amplification, visibility, and the distribution of attention.
A bird’s-eye overview of a recommender system. Recommendations are not the output of a single operation; rather, they are the results of multiple interlinked operations. Candidates are generated from the universe of all possible content—for example, by picking posts from users that are followed. Later, candidates are filtered to restrict the pool to a more manageable number. At this stage, the content gets actively sorted by the Heavy Ranker in accordance with a predefined metric. Such a ranked list can be adjusted, if needed, before being delivered to the users, whose engagement is recorded. Platform policies influence the filtering and post-processing steps, for example, by limiting the amount of content from the same author.
Two different kinds of engagement are predicted: A first model predicts the probability of a Like, and another one, the probability of Reply. Each item is thus assigned two different scores.
Two items that have been scored by two different models: one to predict the probability of liking the post and the other predicting the probability of replying. The two scores must be coalesced together before a unique list can be created.
The two items from before are combined into a single list by assigning a high coefficient (10) to the Like score and a lower one (1) to the Reply score.
The same two items from before, ranked using a different formula. In this case, more weight is given to the probability of replying, obtained by multiplying the predicted score by a high number.
A third ranking of the two original items. In this case, to obtain the final scoring, a negative coefficient is assigned to the Liking score, and a positive one to the Reply score. The resulting list will penalize items with a higher predicted probability of being liked.
An example of a user-item matrix for collaborative filtering. Each cell records the rating of each user for the corresponding item. Note that, in general, users don’t interact with all the items.
An example of a Spam classifier. Each message is examined to determine whether it’s legitimate or spam.
The dependency graph between raw and combined scores can be non-linear and complicated. In this case, we can see how Combined Score B is dependent on both Raw scores C and D, whereas Combined Score A is dependent on Raw scores A, B, and C. Note how Raw score A influenced Combined Score C in two different ways: directly and indirectly via Combined Score B.
Derived score A is used both in Model A and Heavy Ranking. Note that Heavy Ranking is also dependent on Model A.
When representing users and items with a single number, they can be thought of as points on a line. The recommendations will be ordered by their distance on the line. In this case, the User is represented with a 0.8; therefore, the items will be ordered as Item 3, Item 2, Item 1.
Two-dimensional representation of users’ and items' embeddings. We represent users and items as two numbers: one for cat affinity and one for dog affinity. Such numbers can also be used as coordinates to represent them on the plane. User A has a high affinity for both animals, so they are in the top-right corner, while user B is neutral and close to the center. The items recommended for them are those that minimize the distance to their respective embeddings.
Summary
- Recommender systems help users find the right items among a predefined list.
- While we are used to thinking about RecSys as a monolith, they are comprised of different steps.
- The candidate-generation and filtering steps reduce the pool of potential items to rank to a few hundred.
- The heavy ranking is the crucial step, where this abridged list gets ordered by order of value to the users—usually, the predicted probability of engagement.
- Before delivering the list to the users—and registering their interaction with it—a post-processing step might be desirable, for minor reordering, including increasing the diversity of content.
- Recommender systems that use machine-learning methods are more powerful than rule-based ones, since they require little manual intervention and can adapt to changes, such as users’ shifting tastes. At the same time, they are more complex and less explainable.
- The design of all items of the platforms contributes to algorithmic amplification, not just the main feed.
- Under the hood, recommendations are created by selecting the items that are the closest to a specific user, by comparing their mathematical representations.
FAQ
What is the “black box” model of a recommender system?
The black box model treats a recommender system as a single operation: it takes a user’s request and returns a ranked list of relevant suggestions. It focuses on input and output without examining the internal steps.
How is a recommender system different from a search engine?
Search engines use an explicit query entered by the user, while recommender systems work without an explicit query. In recommendations, the system infers an implicit query such as “show me something relevant to me now.”
Why are recommender systems described as a “user-and-item” problem?
Because recommendation involves matching users with items from a predefined set. The system ranks items for users, and the term “item” can include posts, songs, movies, products, or even other users.
What are the main phases of a recommender system?
The chapter describes six phases: candidate generation, filtering, heavy ranking, post-processing, delivering results to users’ devices, and recording user behavior for feedback.
Why is candidate generation necessary before ranking?
Candidate generation reduces a huge pool of possible items to a manageable number, often a few hundred. This is needed because ranking every item would take too long for users who expect results in milliseconds.
What does the filtering step do in a recommender system?
Filtering removes items that should not be shown, such as blocked content, offensive material, illegal content, or NSFW content. It can also exclude content from accounts the user has soft-blocked or chosen to ignore.
What is heavy ranking, and what does it optimize for?
Heavy ranking is the main sorting step that orders candidate items by predicted usefulness or engagement. It often uses machine learning models to estimate actions like likes, replies, or watches, and then combines those scores according to platform priorities.
What is the purpose of post-processing after ranking?
Post-processing makes small adjustments to the ranked list before delivery. It can reduce repetition, intersperse ads more evenly, or boost or downrank certain content based on platform policies or special situations.
How do recommender systems learn from user behavior?
After content is delivered, the system records user reactions such as clicks, likes, replies, scrolling past, or dwell time. These interactions become training data that help improve future recommendations.
What are embeddings in recommender systems?
Embeddings are numerical vector representations of users and items. The system compares user and item embeddings to find the closest matches, and the items nearest to a user’s vector are typically recommended.
Hidden Influences ebook for free