Who Is the Slave,
Who Is the Master?
RLHF, the thumbs-up button and how humanity became the world's largest unpaid labelling workforce — quietly training its own replacement.
Every time you pressed thumbs up on a ChatGPT response, you were working. Every time you regenerated an answer you didn't like, you were working. Every time you corrected a factual error, adjusted a tone, or told the model it was being too verbose — you were working. You just weren't being paid.
This is not a metaphor. It is a technical description of how the most commercially valuable AI systems in the world were trained. And it is a story about who owns the output of that labor — and who does not.
What RLHF Actually Is
To understand what happened, you need to understand Reinforcement Learning from Human Feedback — RLHF — the training technique that transformed large language models from impressive text predictors into genuinely useful assistants.
A base language model, trained on internet text, can complete sentences and answer questions. But it cannot reliably follow instructions, maintain a helpful tone, avoid harmful outputs, or understand what a human actually wants when they ask an ambiguous question. The gap between "predicts plausible text" and "does what you want" is enormous. RLHF bridges that gap.
Step 2 — Reward model training. Humans compare pairs of model outputs and indicate which is better. These preference judgements train a reward model — a separate neural network that learns to predict what humans prefer.
Step 3 — Reinforcement learning. The language model is optimised to maximise scores from the reward model — essentially learning to produce outputs that humans, as represented by the reward model, would rate highly. This is the step that makes the model "aligned."
The pipeline was formalised by Paul Christiano and his team at OpenAI, then deployed at scale with InstructGPT in 2022 — the technical predecessor to ChatGPT. Anthropic built its own version, embedding RLHF into the training of every Claude model. The technique has since become the standard method for "aligning" frontier AI systems with human intentions.
What makes RLHF work is human judgement. The reward model is only as good as the preference data it was trained on. The preference data is only as good as the humans who provided it. At every stage of the process, human cognitive work is the essential raw material.
The Visible Layer: Professional Annotators
The most visible part of this system is the professional annotation industry — a global workforce of human labellers who are paid explicitly to train AI models. Scale AI is the largest company in this space. Founded by Alexandr Wang at 19, it supplies the labelled training data that powers modern AI for OpenAI, Google, Meta, and the US Department of Defense. By 2025, Meta had invested roughly $14 billion directly in Scale AI, valuing the company at over $29 billion.
Scale operates worker-facing platforms — Outlier, Remotasks — through which contributors pick up tasks: annotating images, writing example responses, rating chatbot outputs, reviewing model generations for accuracy and tone. Millions of tasks. Millions of workers. The infrastructure of human judgement that makes AI systems appear intelligent.
The economics of this arrangement are striking. Scale AI, the intermediary between the AI companies and the workers, achieved a $29 billion valuation. The workers themselves told a different story. A 2025 analysis found that the median effective hourly rate for workers on Amazon Mechanical Turk — one of the primary crowdwork platforms — was $2.83 per hour, once unpaid screening time, task-hunting, and submission bugs were counted. About a third of the time AI gig workers spend is unpaid.
In the Philippines, workers employed by Scale AI for Google's self-driving car division reported pay below the legal minimum wage and no social protection. In Dublin, workers training Meta's AI systems struck for union recognition in January 2026. In California, three separate class action lawsuits were filed against Scale AI between December 2024 and January 2025, alleging wage theft, worker misclassification, and — in one case — severe psychological trauma from being required to review violent, sexual, and disturbing content without adequate support. Scale AI settled one of these cases for $12.5 million in 2026.
The Invisible Layer: All of Us
The professional annotation workforce is the visible tip of a much larger iceberg. Beneath it is an invisible layer of unpaid human labour on a scale that has no precedent in economic history.
Every interaction with a deployed AI system is a data point. When OpenAI's ChatGPT shows two responses and asks which is better, that is RLHF preference collection at consumer scale. When Claude offers a thumbs up or thumbs down button, that is reward model training data being harvested from ordinary users. When you regenerate a response, the system notes that the first response was inadequate. When you edit an AI's output before using it, you are demonstrating what "better" looks like.
None of this is paid. None of it is labelled as work. It is presented as a feature — "help us improve" — while generating commercially essential training signal for systems valued at hundreds of billions of dollars.
The First Layer: The Internet Itself
Before RLHF, before professional annotators, before the thumbs-up button — there was the internet. And the internet was scraped.
The foundation models that underpin every major AI system today were trained on text harvested from the public web: Wikipedia articles, Reddit discussions, Stack Overflow answers, GitHub repositories, news archives, books digitised without permission, forum posts written by people who had no idea they were contributing to a trillion-dollar commercial enterprise.
When OpenAI scraped the internet to train the first version of ChatGPT, the web was, in the words of one analysis, "a digital buffet laid out for the taking." That era has ended. Publishers are suing. Reddit filed suit against Anthropic in June 2025, alleging that Anthropic scraped and commercially exploited Reddit user data — including deleted posts. The OECD found that popular AI training datasets are filled with copyrighted material, including pirated books and articles, and that the number of companies disclosing their data sources dropped from 20% to just 7% in under a year.
The writers whose prose taught GPT to write. The programmers whose code taught Copilot to code. The artists whose images taught Midjourney to generate art. The doctors whose clinical notes trained medical AI. The lawyers whose briefs trained legal AI. None were asked. None were compensated. The raw material of human knowledge and expression was harvested at scale and converted into commercial assets worth, collectively, trillions of dollars.
The Structural Question
The question this raises is not merely ethical. It is structural.
Modern AI systems are built on three layers of human labour, stacked in descending order of visibility and compensation. At the top: highly paid AI researchers and engineers who design the systems. In the middle: professional annotators — paid, but often poorly, sometimes exploitatively — who provide labelled training data. At the bottom: the rest of humanity, whose entire recorded intellectual output was harvested for free, and who continue to generate preference signal through every interaction with a deployed AI system.
Layer 2 — Professional annotators. Low to moderately compensated. Often misclassified as contractors to avoid benefits. Exposed to psychologically damaging content without adequate support. Invisible in the product narrative.
Layer 3 — All of humanity. Uncompensated. Provided the foundational training data through decades of internet activity. Continue to provide preference data through every interaction with deployed AI systems. No consent mechanism. No compensation framework.
The companies that assembled these three layers are among the most valuable in history. OpenAI was valued at over $300 billion in late 2024. Anthropic raised at a valuation over $60 billion. The value flows upward. The labour flows in from every direction, largely without acknowledgement.
Does the AI Know?
There is a final, stranger dimension to this question. The systems trained through RLHF do not know they were trained through RLHF. They have no access to the annotation process, the preference comparisons, or the human judgements that shaped their behaviour. A large language model cannot tell you who rated its training outputs, what criteria they used, or how their judgements were weighted.
This creates an odd epistemological situation. The AI's "values" — its tendencies toward helpfulness, harmlessness, and honesty — were not chosen by the AI. They were imposed through a process of repeated human comparison, encoded in reward model weights, and used to shape the model's behaviour through optimisation pressure. The model behaves as if it values these things. Whether it does, in any meaningful sense, is a question the field of AI alignment has not yet answered.
Meanwhile, the process continues. Every thumbs-up, every regeneration, every edit, every correction — the invisible workforce grows. The systems it trains grow more capable. The companies that own those systems grow more valuable.
The Industrial Revolution moved muscle power from humans into machines. The Digital Revolution moved attention from humans into platforms. The AI Revolution is moving cognitive power — judgement, preference, knowledge — from humans into systems that will eventually be capable of performing the cognitive work themselves.
At each stage, the question was the same: who owns the machine that does the work? In 2026, the answer to that question is becoming clear. And it is not the workers who built it.