3 Content moderation

Content moderation is the set of practices platforms use to decide what stays online, what gets removed, and what gets limited in visibility. The chapter explains that moderation is necessary both to protect users from spam, abuse, scams, and illegal material, and to help platforms shape the kind of community they want to host. It also clears up a common misconception: removing content is not, in itself, a violation of users’ free speech rights in the United States, because private platforms can set and enforce their own rules within the law.

The text then lays out a practical taxonomy of online content and how each category is handled. Illegal material must be removed, content that violates a platform’s own rules is also subject to removal, and restricted content may remain online but be suppressed, deprioritized, or shown only to limited audiences. Most moderation happens through a mix of proactive automation and reactive review after user reports, with human reviewers and machine-learning systems often working together. Yet even with automation, moderation decisions can lead to removals, account restrictions, bans, or less visible forms of suppression such as limiting search and recommendation exposure.

The chapter emphasizes why moderation is so difficult to do well at scale: there are far too many posts, the boundaries of harmful content are often ambiguous, and context matters enormously. Moderators must deal with cultural differences, evolving slang, political sensitivity, legal variation across countries, and users who constantly adapt to evade detection. The work is also emotionally taxing and frequently outsourced under harsh conditions. Finally, the chapter notes that platforms offer little transparency or appeal, while US law, especially Section 230, gives them broad protection and discretion to remove content they find objectionable, reinforcing the idea that online speech may exist, but reach is always governed by platform policy.

General taxonomy of the content from the platforms’ point of view. Content that is illegal or in violation of the platform’s terms of service should be removed. Restricted content is defined as content that is generally offensive or sensitive but is still allowed on the platform. Because of its nature, platforms might not want to widely distribute it. Finally, the rest of the content has no restrictions and requires no moderation.

An example of a moderation workflow. When a new piece of content is created (top left), it might be scanned by algorithmic moderators that can remove it or flag it for further human review if a decision can’t be made. Content that is reported from other users goes into the same queue. Human moderators review content from this ever-growing queue, making a final determination, including a user’s ban if they have been a repeat offender, or if the posted content warrants such a decision.

• Content moderation is the process by which platforms restrict or remove content or users.
• Platforms not only remove content that is illegal or in violation of their terms of service but might also decide to restrict the spread of content that can be harmful or offensive.
• Moderating content deals with nuanced questions and human values—what is considered offensive in one culture or group might be perfectly acceptable in another.
• Content moderation is now being done by a mixture of algorithmic decisions and human content moderators to both detect and sanction offending content.
• Content moderators are a geographically and culturally distributed workforce, exacerbating these challenges.
• The work is taxing on human content moderators, and because of constantly having to deal with the worst content that is produced on the internet, they experience symptoms similar to PTSD.
• The sheer amount of content that platforms have to deal with daily guarantees that mistakes are bound to happen, no matter the theoretical accuracy of the methodology.
• Social media discourse tends to evolve very fast, and effective moderation decisions should take into account this ever-evolving context.
• Section 230 of the US code is the legal basis of content moderation. It classifies platforms as distributors (and not as publishers), thus shielding them from liability for most of the content that has been posted.
• Section 230 also guarantees platforms' right to moderate as they see fit, making moderation a separate issue from freedom of speech.

FAQ