8 The unreasonable effectiveness of engagement
The chapter explains why engagement-based recommender systems are so effective by showing that their power comes from a mix of human psychology, platform design, and historical conditions. Rather than treating technology as neutral, it argues that social media and recommendation engines work because they tap into slow-evolving instincts: our tendency to react quickly, seek novelty, and care deeply about belonging. These systems succeed by turning human attention into a feedback loop, where user interactions train the algorithms and the algorithms then shape what users see next.
A major part of the chapter focuses on the user side of that loop. It describes how fast, intuitive thinking makes people prone to snap reactions, emotional posting, and habit-driven scrolling, while our need for group membership pushes us toward conformity, social comparison, gossip, and moral outrage. Because online groups can be defined arbitrarily and content is often curated as highlight reels, people are constantly comparing themselves to skewed samples of others’ lives. This can distort perceptions, intensify polarization, and normalize more extreme behavior, especially when public shaming and viral outrage are rewarded with visibility.
The chapter also shows how recommender systems amplify these tendencies through their technical structure. They are content-agnostic, scale easily, and optimize for engagement, which makes them able to surface communities, activate fringe audiences, and push users toward increasingly extreme or harmful material through out-of-network recommendations, feedback loops, and rabbit holes. Because the systems react at machine speed while oversight and mitigation move much more slowly, harmful content can spread before it is recognized. The chapter closes by placing all of this in a broader historical setting: a time of political polarization, weakened shared institutions, media fragmentation, and social change that made these technologies especially influential and especially dangerous.
An example of how the shape of the network, i.e., its connections, influences the flow of information. In diagram A, the circle nodes are better connected than the square ones. Because of that, the square nodes would receive more information from the circular ones, giving the impression that their opinion is the majority. In contrast, a different network shape, shown on the right, can lead to a very different spread of information where the square nodes receive the majority of their information from other square nodes.
The Ash conformity experiment, conducted by the psychologist Solomon Ash, to understand whether people would conform to the majority group. Each subject was asked to indicate, by speaking after everyone else, which line in the right group matches in length the line on the left. When everyone in the subject’s group gave a clearly wrong answer, they also conformed with the opinion of the majority.
An example of how interacting with a new user might lead to unexpected content. Assume, like in Figure A above, that the user is exposed to the content and interacts with someone who’s tightly connected to a group. If the two users keep interacting, the circle user will become increasingly exposed to the group's content, as shown in Figure B. If the group shares harmful content, that can make its way to the original user, even if the original connection was benign.
Summary
- Recommender systems’ impact, as socio-technical systems, is determined by a combination of both users’ and technology's peculiarities, and both need to be understood.
- To understand users’ reactions, we need to remember how important it was to our survival to be in a group. That results in both stronger identification with the group, and less trust in members of other groups.
- We also have faster, more primitive ways of making decisions that are easily invoked by posts with strong emotional factors.
- Recommendation systems are content-agnostic. While this makes them extremely valuable and useful for ever-growing collections—including social media posts—they also can’t discern between helpful and harmful recommendations. The same problem is reflected in the community of users they create.
- The fact that we can be exposed to content from people we don’t explicitly follow is one reason content goes viral. Because more engaging content is recommended more often, and more extreme content tends to elicit a stronger emotional response, it tends to travel further.
- The societal environment in which such systems emerged was characterized by a more fragmented media industry, increased polarization, and a decline in in-person connection. Recommender systems didn’t create any of those problems, but contributed to accelerating them.
Hidden Influences ebook for free
