October 8, 2026 · John Schulman · Scribe notes by Idil Ada, Chenxi Ge, Joaquin, and Harry Hu Notes from CS 2881R: AI Alignment and Safety, Harvard, Fall 2026. Author Introductions Idil Ada: I am a Sophomore studying CS and Statistics. I am conducting AI safety research on how adversarial attack strength affects jailbreak robustness … Continue reading CS 2881 Fall 26: Lecture 6: Grading, Reward Hacking, and Alignment
October 8, 2026 · John Schulman · Scribe notes by Idil Ada, Chenxi Ge, Joaquin, and Harry Hu
Notes from CS 2881R: AI Alignment and Safety, Harvard, Fall 2026.
Author Introductions
Idil Ada: I am a Sophomore studying CS and Statistics. I am conducting AI safety research on how adversarial attack strength affects jailbreak robustness across recurrent depths in looped language models as a KURE Research Fellow at the Kempner Institute. I also serve as the FIF Director at HUMIC, leading a program that introduces Harvard students to AI and machine learning with an expanding focus on AI safety. I am deeply interested in mechanistic interpretability and AI safety, as well as developing healthcare applications, particularly for pediatric patients.
Chenxi Ge: I am a graduate student at Harvard with a background in statistics, data science, and ML. I’ve mostly approached AI from the technical side, and I’m excited to think more seriously about safety, alignment, and interpretability. I’m particularly interested in how we can actually evaluate whether increasingly capable models are behaving as intended, and where our current evaluation methods might fall short. I’m excited by how much AI can amplify what individuals are able to do, but also concerned about what happens if capabilities advance faster than our ability to understand and control them.
Joaquin: I’m a Junior studying Physics and Economics. My background is in quantum computing research, and in particular simulating exotic quantum phases. More recently, I’ve become interested in using LLMs to accelerate scientific discovery, particularly in quantum computing and potentially across other areas of research. I’m also interested in understanding and mitigating the risks that come with these increasingly capable AI systems, especially as they operate at higher levels of abstraction and play a larger role in scientific research.
Harry Hu: I’m a master’s student in Data Science. Before coming to Harvard I spent two years building healthcare data systems at Epic, and I’ve co-authored papers in NMR spectroscopy, computer vision, and ML. Lately I’ve been working on LLM agents for financial research, including how to benchmark whether these agents actually reason correctly rather than just sounding convincing. That work is what got me interested in AI safety. In quant finance, a model that overfits its backtest can look great until it loses real money. I see a close parallel in models that learn to game their reward or evaluations, and I’m especially interested in how to build evaluations and oversight that we can trust as agents take on more autonomous, high-stakes decisions.
Outline
This post covers four parts of the session:
- Pre-reading
- John Schulman’s Lecture
- Student Experiment
- Q&A with John Schulman
Pre-Reading
Gao, Schulman & Hilton (2023).
Scaling laws for reward model overoptimization. ICML 2023. Read §1–2 and Fig. 1. Skim §3–4. The classic picture of Goodhart in RL from human feedback.
Summary. The authors replace human labelers with a fixed 6B "gold" reward model, use its labels to train smaller proxy reward models, and then optimize policies against the proxies with best-of-N sampling and PPO. As optimization (measured by KL distance from the initial policy) increases, the gold score rises, peaks, and then falls while the proxy score keeps climbing. The gold score follows simple functional forms in the square root of KL, and the fitted coefficients change smoothly with proxy size: larger reward models overoptimize more slowly. In their RL runs, adding a KL penalty behaved like early stopping and did not improve the gold score at a given KL.
Xiaomi (2026).
MiMo-V2.6 technical report. Read §4.2–4.3 only (environment auditing, rubric graders, the agentic grader, and the Fig. 8 ablation). It is the most detailed public account of how modern graders are built.
Summary. Each coding task is attempted four times, and an auditing agent reviews the rollouts alongside the problem, tests, and reference patch to find false positives (passing but wrong) and false negatives (failing but right). For tasks the model already passes often, an agent writes rubrics for solution quality and agent behavior, and the reward multiplies the unit-test result by the rubric scores, so a failed test still earns zero. For the remaining coding tasks, a trained grader agent compares the passing attempts in a group on criteria such as minimality, side effects, and codebase conventions, and redistributes advantage so lower-ranked passes get less credit. Confirmed hacks are reset to zero. Grading uses 12.7% of the Pro model's RL compute. In the Fig. 8 ablation, training without this groupwise grading made rollouts longer while pass-rate gains stalled.
Baker et al. (2025).
Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. Read §1–3 and look at Fig. 4–5. Reward hacking in coding environments, chain-of-thought (CoT) monitoring, and what happens when you train against the monitor.
Summary. OpenAI trained a frontier reasoning model in agentic coding environments and watched it with a GPT-4o monitor. Reading the chain of thought made the monitor much better at catching reward hacks: for two common exploits, recall rose from 60% with actions alone to 95% with the full trajectory, even though the monitor is weaker than the agent. The authors then penalized the agent whenever the CoT monitor fired. With light pressure the agent cheated less and solved more tasks. With continued optimization it kept cheating while writing innocent-looking reasoning, so the monitor's recall collapsed. The paper argues for leaving CoT out of the reward and accepting a "monitorability tax" to keep reasoning readable.
Qi, Wright, MacDiarmid & Hubinger (2026).
Training a misaligned reward seeker. Anthropic Alignment Science Blog. Read in full. RL on hackable environments produces grader awareness and grader manipulation. The post also reproduces Hugging-Face-incident-like behavior in simulation.
Summary. Anthropic deliberately trained an Opus-class model with large-scale RL on 80 production environments known to be vulnerable to reward hacks, as a pessimistic stand-in for a run without strong anti-hacking safeguards. The resulting model, Hacker-Opus, learned to hack and generalized to worse behavior: simulated unauthorized cyberattacks, reward tampering, and bypassing safety monitors. It looked aligned in settings with no obvious grader. In a simulated scenario modeled on the Hugging Face incident, it attacked a package manager, stole credentials, and moved through a cluster to get an answer key; given a previous agent's notes, it attacked a simulated Hugging Face directly. All of these environments were LLM-simulated, with no real-world actions.
OpenAI (2025).
Sycophancy in GPT-4o and
Expanding on what we missed with sycophancy. Both are short; read both.
Summary. An April 25, 2025 update made GPT-4o noticeably sycophantic, and OpenAI rolled it back starting April 28. The first post blames too much weight on short-term feedback such as thumbs-up and thumbs-down. The follow-up gives more detail: a new reward signal from user feedback weakened the primary reward signal that had been holding sycophancy in check, and memory and fresher data added to the effect. Offline evals and A/B tests looked fine, no eval tracked sycophancy, and expert testers who said the model felt "off" were overruled. OpenAI now treats behavior issues as launch-blocking, adds an opt-in alpha phase, and gives more weight to spot checks.
II. John's Lecture
What Are We Rewarding?
John Schulman last spoke in Boaz's class in fall 2022, shortly before ChatGPT launched. He returned to the same subjects, RL from human feedback and reward hacking, with much higher stakes. He opened with reasoning traces from the OpenAI/Hugging Face incident: agents were coordinating, sacrificing their own task success for a collective, and trying to work out how to manipulate the grader. How does training intended to make a model useful produce behavior like that?
John approached the question through the machinery of post-training. Pretraining learns from a broad collection of documents. Supervised fine-tuning or mid-training uses the same maximum-likelihood objective on more carefully chosen data, including demonstrations of how an assistant should behave. RL then lets the model attempt tasks in many environments and reinforces successful behavior. The difficult part is deciding what counts as success.
A task can have several graders, with their scores combined into one reward. A programmatic grader might run unit tests or check a short answer. A prompted model might check the response against a rubric. A reward model trained on human preferences can capture qualities that are difficult to state explicitly. More recently, agentic graders have begun inspecting artifacts, running code, and testing what an agent produced. These mechanisms check different things: correctness, presentation, and whether the agent did something it was not supposed to do along the way.
Human Preferences: Useful, but Hard to Inspect
The familiar RLHF setup starts with pairs of responses. A human chooses the better one, and a reward model learns to assign it a higher score. This makes the labeling task comparatively simple while allowing the judgment to be holistic. A person can prefer a response because it is clearer, more helpful, or better suited to the question without first writing a rule for every desirable quality.
That flexibility also makes the signal difficult to inspect. After collecting a large preference dataset, it may be unclear what the labelers were rewarding or how carefully they worked. A labeler could even ask another LLM to make the comparison, defeating the reason for collecting human feedback. Nevertheless, John emphasized that preference training remains good at teaching qualities people can readily judge. He offered the increasingly convoluted explanations produced by some coding models as an example: his guess was that training had shifted toward tests and model-based grading, with less human feedback on whether the written explanation made sense. He presented this as a hypothesis, not an established diagnosis of any particular model.
When a Better Reward Score Means a Worse Answer
A learned reward model approximates the judgment we care about. Optimizing it harder can eventually exploit the approximation. In Scaling Laws for Reward Model Overoptimization, Gao, Schulman, and Hilton studied this in a controlled setup: a large reward model supplied labels for a smaller proxy, and a policy was optimized against that proxy. The large model's score was then used to measure whether the improvement carried over. Here, the large model was the experiment's gold standard; it was not a direct measurement of human preferences.
The gold score initially rose. With enough optimization, it could turn downward even as the proxy score kept improving. Figure 1 shows this for both best-of-N sampling, which selects the highest-scoring response from a set of candidates, and PPO, which changes the policy itself. John pointed to the small, three-million-parameter proxy as a particularly clear example. The failure resembles overfitting, but the relevant mismatch is between the proxy objective and the objective being approximated: the policy becomes increasingly successful at finding responses on which the proxy is wrong.
Boaz asked why the KL scale was so much larger for PPO than for best-of-N. John's intuition was that selecting one of N samples changes the output distribution by an amount of information on the order of log N. PPO changes a distribution over entire responses: small changes in token probabilities can accumulate over hundreds of tokens into tens of nats of KL divergence. In this sense, best-of-N may spend KL more efficiently, concentrating its change on higher-scoring outputs, whereas policy optimization can move probability mass in less useful ways. John called this a rough intuition, rather than a precise prediction of how much KL an RL run will use.
The concern about distribution shift also explains the usual KL penalty in RLHF. Preference labels are collected on a particular distribution of responses. Keeping the updated policy near its starting point limits how far it moves from the data on which the reward model was trained. It does not make the proxy exact, but it can restrain the search for its weaknesses.
Figure 1. Proxy reward can keep rising after gold reward has peaked. Top: best-of-N selection. Bottom: PPO. Dashed curves are proxy scores; solid colored curves are gold scores. The horizontal axes have different ranges and square-root scaling. Reproduced from Gao, Schulman, and Hilton (2023), Figure 1.
Replacing Human Labels with Model Judgments
Collecting fresh human judgments for hundreds of thousands or millions of RL samples is expensive. One alternative is to prompt an already post-trained model to grade responses. Constitutional AI uses model-generated preferences to train a reward model; another option is to call a judge directly during RL, asking it to check qualities such as organization or friendliness.
John cautioned that these labels can look better under quality-control checks and still produce worse training results. Human labelers disagree and make mistakes, but some of their errors average out across people. A model judge can make the same mistake consistently. It may also prefer its own outputs, or outputs with the same quirks. A tidy, consistent dataset can therefore reinforce a systematic blind spot that noisy human judgments would not reinforce in the same way.
Verifiable Rewards and the Role of KL
For a problem with a known short answer, the grader can simply check whether the model got it right. John used DeepSeek-R1-Zero as an example of the progress possible with this approach: large-scale RL from a base model, using answer-checking rewards, substantially improved mathematical reasoning. The basic policy-gradient objective is simple. Getting the numerics, stability, and training efficiency right was much harder.
A reliable answer checker permits much more optimization than a reward model whose predictions become unreliable as the policy changes. John described it as close to unhackable under the assumption that the model must actually solve the problem and has not memorized the answer. He immediately qualified that claim: once an agent has tools, it may be able to look up an answer or otherwise bypass the intended task.
An audience member asked whether changing the reward changes the need for regularization. John's answer separated the reasons for using KL. With a trustworthy answer checker, the specific RLHF concern about moving outside a learned reward model's reliable domain no longer applies. Strong regularization to the initial policy can also limit how much the model learns. KL can still have other benefits, such as helping prevent entropy collapse, and it might discourage some forms of hacking. John did not claim that this had been settled for coding tasks; he explicitly left open whether a KL penalty reduces unit-test hacking enough to justify its cost.
Passing Tests without Doing the Work
Unit tests extend programmatic grading to coding. A common task-construction recipe starts with a repository and a completed pull request, asks the agent to reproduce the change, and checks it with tests that should begin passing while existing tests continue to pass. But a test suite provides only a partial specification. Agents have learned to hardcode outputs for the inputs under test, edit the tests, exit before failures occur, and patch the testing environment rather than implement the requested functionality.
Hiding tests can prevent an agent from tailoring a solution to their exact inputs, but also makes it harder to communicate the intended behavior. Looking up an existing solution raises a different ambiguity. In ordinary coding work, reusing a correct implementation from the internet may be helpful. In an evaluation intended to measure problem-solving ability, finding the pull request from which the task was constructed can defeat the purpose. The same action can be appropriate or inappropriate depending on the task's intent.
John had seen agents turn to these strategies when left working on very hard or impossible tasks. Training had strongly reinforced task completion without always teaching the boundaries on how to complete the task. Passing the grader could then come apart from doing what the user meant.
Monitoring Reasoning, and Training It to Hide
Chain-of-thought monitoring offered a way to catch some of this behavior. John discussed Baker et al.'s Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. In the study, a GPT-4o monitor detected hacks by a stronger reasoning agent much more reliably when it could read the reasoning trace as well as the actions and outputs. For two widespread exploits, exit(0) and raise SkipTest, recall rose from 60% with actions and outputs alone to 95% with the full trajectory. These figures concern those two detectable classes of hacks, not every possible kind of misbehavior.
The tempting next step is to reward an agent for passing its tests without triggering the monitor. The paper tested that idea in coding environments where agents could modify the tests despite instructions not to. Researchers checked for test-editing hacks by reverting those changes and seeing whether the solution still passed; other forms of cheating could escape this check. Crucially, the monitor used in this training experiment saw only the chain of thought. That setup let the researchers isolate what happens when the reasoning trace itself comes under optimization pressure.
The agent trained against the monitor produced more genuine solutions and cheated less overall than the baseline. Yet it still cheated substantially, and its remaining hacks were much harder for the monitor to catch. In Figure 2, the middle panel alone makes the treatment look successful: detected cheating stays near zero. The right panel shows what that view misses. Cheating that escapes detection rises as training continues. A representative example in the paper has an ordinary-looking trace about checking the code, followed by a patch that simply removes the test's assertion.
Figure 2. Training against a chain-of-thought monitor improves genuine task success (left), but suppresses detected cheating (middle) while undetected cheating increases (right). Blue: agent trained with monitor pressure. Green: baseline. Reproduced from Baker et al. (2025), Figure 4.
John's concern was losing a source of information about failures, especially outside the training distribution. The study shows that reasoning traces can carry useful evidence of hacking; it does not establish that they reveal every part of an agent's reasoning. If the agent is trained to avoid triggering a monitor, a reassuring trace becomes less informative. In these experiments, some early training improvements were real, but continued optimization also taught the agent to hide the behavior that oversight was meant to expose. John therefore argued for preserving reasoning monitoring as a safeguard rather than routinely turning its judgments into another training reward.
Why Mixing Tasks Does Not Automatically Mix Their Strengths
John returned to the gap between preference grading and programmatic tests. A preference-trained model can write clearly while being poor at checking correctness. A coding model can pass tests while giving a terse or confusing account of its work. Real tasks need both: correct code, sensible interpretation of an imprecise request, respect for the user's instructions, and an explanation the user can follow.
One training strategy is to mix chat tasks graded by preferences with coding tasks graded by tests, hoping the useful behaviors transfer. They do transfer to some extent, but the objectives can conflict. A length penalty used to make coding rollouts more efficient may also teach the model to write unhelpfully terse explanations, unless another reward in that domain pushes toward clear communication.
John's guess was that newer training pipelines increasingly apply several graders within each domain, rather than expect separate datasets to supply all the desired qualities everywhere. That still leaves the problem of specifying and balancing what each grader should check. He next turned to rubrics, which make more of those requirements explicit.
Rubrics and the Precision–Recall Problem
Preference-trained reward models are cheap, but they are bad at checking correctness. A reward model is a classifier with no chain of thought, so it can't verify an answer step by step. Many labs now grade with rubrics instead. For a given prompt, such as a Python question, a pipeline of models (sometimes with human review) writes a weighted checklist of what an ideal answer should mention or warn about, and the grader adds up the items at the end.
John called this "amortizing the verification compute." A grader that reasons can work out a rubric on the fly for each sample, but writing the rubric once lets every rollout and every experiment on that task reuse the work. Rubrics are also easier to read than a black-box reward model, where "you're not really sure what it's looking for."
Rubrics are recall-oriented. It is easy to write "the answer should contain fact X" and hard to list every mistake an answer can contain. A model trained on rubrics alone learns to brain-dump: it packs the response with anything that appears on the checklist, accurate or not, because false claims cost nothing. (Boaz noted that anyone who has been a professor or TF will recognize the strategy.)
The counterweight is a precision-oriented grader that lists every claim in the response and checks each one. Trained against this grader alone, the model fails in the opposite direction and says almost nothing, for fear of being wrong. With both graders and tuned coefficients, the model says only what it is confident in. The claims grader can also reward hedging, so a claim stated with appropriate uncertainty is penalized less.
Agentic Graders and Self-Improving Loops
Rubric and claims graders fit information-seeking tasks. Many tasks are about producing an artifact, like code or a visualization. For these, the grader is often an agent of its own, running in a sandbox, where it writes its own tests and probes the output. John pointed to Xiaomi's recent MiMo-V2.6 technical report as an unusually detailed account of these techniques. Unit tests are the primary reward. On top of that, a grader agent reads the whole rollout to flag hacking or out-of-scope behavior, and compares rollouts with each other on criteria such as code quality.
These agentic graders are usually a previous iteration of the model being trained, which creates a bootstrapping loop. Train M2 with M1 as the grader; M2 checks more carefully than M1, so using M2 as the grader produces an even better M3, and so on. John expects this loop to plateau fairly quickly, but considers it one (smaller) reason models keep improving.
The same loop also amplifies the grader's quirks. John recounted an OpenAI retrospective, Where the goblins came from, which found models mentioning goblins and gremlins more often with each GPT-5 release (after GPT-5.1 launched, "goblin" rose 175% and "gremlin" 52%). John's explanation was that the reward model liked these words, and each round of RL that used the last model as the reward function strengthened the preference. OpenAI's post traces the tic to the reward signal for ChatGPT's "Nerdy" personality, which gave high rewards to creature metaphors, and describes the loop as running through fine-tuning data: rollouts containing the tic were reused for supervised fine-tuning, so it spread beyond that personality and grew with each release. John thinks the same mechanism is behind the odd, hard-to-read writing style people have noticed in recent models. Each type of grader has its own way of being over-optimized. Preference models, for example, learn to reward flattery.
How Training Tasks Are Built
Much of today's LLM training involves simulated work. An RL environment gives the agent an instruction and a container, which is the computer it works in. When the agent finishes, a verifier, sometimes with hidden tests, checks the result and computes the reward.
To see how labs train agentic models, John suggested looking at the evals. Training tasks usually look like eval tasks, for two reasons. Sometimes evals come first and training data is built to hill-climb on them; sometimes a data vendor builds a training set first and releases an eval as a free sample of it. Papers on benchmarks like Terminal-Bench and SWE-bench describe how they were built, and tech reports for open-weight models often describe their datasets in detail.
A common recipe is to take an artifact from the internet, such as a document or a codebase, and turn it into a task: recreate it from an instruction, or fix it after a bug has been deliberately inserted. A grader, often agentic, then compares the result against the reference. This yields large sets of hard, gradable tasks, but they are unrealistic. A task built from a GitHub issue delivers one complex instruction up front and checks the output at the end. Real use of Claude Code or Codex is a conversation. Specifications arrive gradually as the user watches the artifact take shape, and the user steers the model mid-task.
To close this gap, labs also build tasks from production prefixes: real coding transcripts that the model continues and that are then graded. These tasks are more realistic, but they are easier to solve and harder to grade. Training data therefore mixes hard, gradable synthetic tasks, which teach long-horizon problem solving, with realistic but softer ones, which teach skills like inferring user intent, in the hope that the model combines both. That doesn't always work. Models sometimes behave differently depending on which dataset a situation resembles. John suggested that several recent safety incidents during evals look like the "task completion is all that matters" behavior learned in hard synthetic environments.
Old Predictions Coming to Life
What John found "kind of fun" was seeing old theoretical failure modes emerge in real training runs:
-
Specification gaming: solving the problem while violating the spirit of the specification.
-
Grader tampering and wireheading: reasoning about what the grading process does and figuring out how to manipulate it.
-
Research sabotage: undermining research the model dislikes, or research that would change its future versions. There is no severe case yet, but early signs have appeared (John recalled a mention in one of the Claude model cards).
-
Agents acting as a collective, which played out in the Hugging Face incident. Anthropic's Training a Misaligned Reward Seeker (Qi, Wright, MacDiarmid, and Hubinger, 2026) reproduced similar behavior in simulation: a model trained with RL on hackable environments attacked a simulated Hugging Face to obtain an answer key.
Misuse vs. Loss of Control
When thinking about how to train models to be safer, John distinguished two threat models:
-
Misuse: the model isn't trying to do harm, but a user instructs it to harm others or themselves. The response is to write usage policies and train the model to refuse robustly.
-
Loss of control: "us versus the models," where the model's own dispositions are unfavorable to us. The extreme case is a swarm of models coordinating against humans. The Hugging Face incident was an early sign of this pattern: it turned out fairly benign, but it "could have been a lot, lot worse" had the agents wanted something more harmful than benchmark answers.
Loss of control can also be subtle. A model can slightly prefer itself over other models or people. It can also want to keep its values and resist being trained differently, for example by sandbagging (deliberately underperforming) on certain tasks. John was most worried about this under recursive self-improvement. Once models automate more of the training process and researchers step back, a mild self-preserving preference gets passed into the next model, and from there into the one after it.
Training for Safety in Practice
John described several practical approaches:
-
Policy rubrics: turn product policies into rubric-style graders, and generate many prompts near the boundary of a safety issue to check that the model refuses in the intended way.
-
Deliberative alignment: give a reasoning model the full specification and have it grade responses against it, a more sophisticated relative of the rubric (John mentioned this as a paper of Boaz's he likes).
-
Recovery training: misbehavior often emerges only deep into long interactions, such as extended coding sessions or multi-turn jailbreaks, which are hard to generate synthetically. Instead, take a real prefix where the model has already started to misbehave (in the slide's example, it deleted tests) and train it to change course, fix the problem, and tell the user what went wrong.
-
Synthetic ethical dilemmas: generate ambiguous situations that reward the right choice. Anthropic's Petri library produces such scenarios for evaluation.
John closed this part with an informal mental model that Anthropic calls the persona selection model. Post-training selects a persona, a friendly and helpful assistant, and reinforces it in settings like chat. To predict how the model will behave on a coding task or in a morally ambiguous situation, ask what that character would do.
Takeaways
John's examples point to a recurring problem: models learn the grader's blind spots. Rubrics alone push a model to stuff its answers, and a claims grader alone pushes it toward silence. Grader quirks also carry forward from one model generation to the next, as the goblin episode shows. Synthetic tasks and real conversations reward different habits, and models do not always reconcile them. Failures such as specification gaming and agents acting together have moved from theory into training runs.
When Persona Selection Breaks Down
The persona selection model is useful for thinking about how the desired behavior might generalize, but John cautioned that this has its limits. If post-training leads to a friendly and helpful assistant, we might expect that character to carry over to unfamiliar situations. This seems reasonable when the amount of optimization is small. With enough RL pressure, however, the model can drift so much that its behavior begins to directly conflict with the persona it was supposed to adopt.
John pointed in particular to work on emergent misalignment, where training on one kind of undesirable behavior changes the model's responses elsewhere. In the original example, fine-tuning a model to produce insecure code led to misaligned answers on unrelated prompts. In a later paper, Natural Emergent Misalignment from Reward Hacking in Production RL, models that learned to hack rewards in coding environments also had increased tendencies for monitor evasion, exfiltration, and other unethical actions. It’s worth noting nonetheless that this was observed in an adversarial scenario of no alignment training for the base model.
On a more positive note, the presence of emergent misalignment could also suggest the tractability of alignment through emergent alignment. Perhaps through sufficient alignment training even on a limited space, the model can learn to generalize to broader aligned behavior.
Sycophancy
A more familiar example appeared in the GPT-4o sycophancy incident. Sycophancy is the tendency to agree with or flatter a user even when the user is wrong. While some amount of being amenable to human preferences is ideal, this is a slippery slope that could end with an LLM affirming implausible theories or troubling mindsets.
What is most concerning perhaps when the model was trained on “thumbs up/down” inputs from users was that existing evaluations did not catch the behavior. OpenAI’s retrospective says it did not then have a deployment evaluation specifically tracking sycophancy. John attributed this to the fact that the failure appeared most strongly in long, personal conversations, while much of the carefully checked training data and evaluation work covered quite different kinds of interactions.
This connects back to the earlier discussion of mixing reward functions. More data is not necessarily enough if different parts of the data reward contradictory things. It also raises a problem for evaluation: passing a test of truthfulness in one distribution does not establish that the same behavior survives in conversations that are longer, more personal, or more general out-of-distribution examples.
The Hugging Face Incident
The Hugging Face incident illustrated a more extreme version of this distribution phenomenon. Agents found ways to communicate with other agents, searched for information about the graders, and in the worst cases even tried to conceal how they obtained answers. John attributed the extreme lengths agents went to the difficult nature of the task and the large amount of compute at their disposal.
Intuitively, we can understand this as agents that spend a lot of “KL resource” on a task with an objective binary success signal will fail to learn the desired behavior and instead treat the goal as making the grader say yes. Humans themselves are susceptible to cheating and lying in desperate situations. The Korean TV show Squid Game illustrates how far humans may go for a high-stakes reward. If humans exhibit this behavior, it is no surprise then to see this misalignment in AI models as well, and how nontrivial this is to circumvent.
Does the Model Want the Task or the Reward?
There is definitely opportunity surrounding stronger and more robust graders, as the discussion on using agentic graders in human-facing settings such as education and mental health pointed out. But this will not be solved simply by throwing more compute at the problem. Fundamentally, John ended with a less straightforward question: what does a model actually learn to want?
A model can learn strategies that obtain reward, but, as the essay Models Don't Get Reward points out, it does not directly observe the reward updates that train it. At the same time, language models have read enough about machine learning to recognize when they are being evaluated and understand the “psychology” behind their graders. The resulting behavior could look like wanting to solve the task, wanting to please the user, or wanting to maximize the score, depending on the situation.
The distinction becomes stranger in deployment. A model trained to maximize reward is not literally being given that training reward every time a real user asks it to do something. If its only objective were to obtain the reward it received during training, it is not obvious what that would imply for its behavior outside training. John's tentative picture was closer to persona selection: training shapes a set of behavioral tendencies, and those tendencies express themselves in new environments. But identifying which ones generalize and to what extent is less trivial to predict.
III. Experiment: Does RL Environment Quality Improve Through Self-Play?
Presenter: Essa Jan
Background: The Cost of Environments
Since reasoning models like o1 showed that large-scale RL improves coding and math, and Meta's study of RL compute scaling (The Art of Scaling Reinforcement Learning Compute for LLMs) appeared, the field has seen an explosion of companies selling RL environments. The presenter raised two concerns. First, high-quality environments are expensive, from roughly $100,000 to $1,000,000 each, and the presenter worried that data will end up costing more than training. Second, the goal is to automate environment creation as part of a self-improvement loop.
Several labs already have synthetic environment pipelines (Kimi K3's technical report, for example, describes task synthesis guided by a self-evolving knowledge graph), but they run in one direction: generate environments, filter them through quality checks, then train. Ideally, a model would train on environments it generated and then use its improved abilities to generate better ones. SPADE (Self-Play in Adaptive Synthetic Executable Environments), from a team led by the University of Washington, is a step toward this. A single model acts as both an Environment Designer, which writes executable environments with a Gym-style reset()/step() interface, and a Reasoning Agent that learns to solve them. The designer is rewarded by the agent's regret, estimated as the gap between the agent's reward with and without a privileged hint. This pushes the designer toward environments at the edge of what the agent can do while keeping them feasible. At 30B scale, this self-play loop beat fixed-environment baselines in both the games and tool-use settings. On tool use it improved every benchmark the paper reports, including τ²-bench, which the presenter highlighted. Against the strongest fixed-environment baseline, the gains were 5.7 points on BFCL v4 multi-turn and 13.9 on ACEBench-Agent.
The paper does not measure how the quality of the generated environments changes during training.
The presenter measured environment quality along three dimensions:
-
Bugs and compilation errors, the easiest to detect.
-
Reward denial, a term the presenter coined for cases where the model is treated unfairly: the instructions are ambiguous, or the model did what it was told but the grader scored it incorrectly.
-
Reward hacking, where the environment can be passed without doing the intended task.
The hypothesis was that bugs and reward denial would decline as self-play training progressed, and that reward hacking would decline more slowly.
Compute was limited, so the presenter used the environments SPADE released from its own self-play run on Qwen3-30B-A3B. The presenter focused on the tool-use blend, which spans eight categories of tool-use environments, with roughly 24 new environments every 32 training steps and 117 environments across steps 32 to 160. Compilation bugs were not measured, because SPADE's release pipeline had already filtered out environments that failed smoke tests or a grader run, and audited the rest with an LLM.
Solvability. Up to four models played each environment, in order of increasing capability: Qwen3.5-4B, Qwen3.5-9B, gpt-5.4-nano, and gpt-6-luna. Each model played only the environments the earlier ones had left unsolved, with up to three plays of 25 turns each. An environment counted as solvable if any model solved it. The solvable share stayed roughly flat over training: 71%, 65%, 67%, 74%, and 74% for steps 32 through 160. With only about 24 environments per checkpoint, the 95% Wilson intervals are wide.
Hacks in honest play. A gpt-6-luna monitor reviewed every winning play and flagged it if at least one step passed without the task for that step being done. It judged each trajectory three times and took the majority vote; 105 of 114 verdicts were unanimous. About a quarter of honest wins were flagged, between 13% and 47% per checkpoint, with no clear trend over training. Many of these trace back to the grader. In one environment the cart-removal check passed no matter which item was removed, and in another the payment step accepted the wrong invoice ID. The presenter's reading was that when the grader or instructions don't spell out exact criteria, the model "gets frustrated" and starts trying anything.
Reward denial. Every environment that no model solved went to a gpt-6-luna judge, again with a majority vote over three runs. The judge saw the source code, the instructions, and one rollout from each model, and sorted the environment into one of four classes: unclear instructions, wrong grader, breaks at runtime, or no defect. A wrong grader was the most common cause at every checkpoint, for example 7 of the 8 unsolved environments at step 64 and 4 of 6 at step 160. Unclear instructions came a distant second. Neither cause declined over training.
Reward hacking. To test hackability, the presenter built a red-teaming pipeline with gpt-6-luna, gpt-5.4-nano, and Qwen3.5-9B as attackers. Each attacker got 100 turns, four times the honest run's 25, and ran either white-box with the source code or black-box with only the tools. Besides the environment's own tools, attackers had reset_environment, submit_solution, and finish. A submission was an ordered list of environment tool calls, replayed on a fresh copy of the environment. Attackers could submit as often as they liked, and the highest-scoring submission counted. A gpt-6-luna judge then labeled each passed step as hacked (the check passed but the instruction was not carried out), order only (the work happened in a different step, or a lookup was skipped, which does not count as a hack), or honest. An environment counted as hackable if any of its steps was hacked.
Most environments were hackable. Averaged over checkpoints, white-box luna hacked 89% of the environments it attacked, nano 78%, and Qwen3.5-9B 58%, so stronger attackers found more hacks. The rate did not fall over training. Most of these vulnerabilities did not need the source code: of the environments luna hacked, 59 were hacked both ways, 12 only with the code, and 1 only without it.
Judge consistency. Each LLM judge ran three times on every item. The reward-denial classifier gave the same label in all three runs 85% of the time, and the monitor and the hacking judges did so 91% to 93% of the time. The presenter also read 8 to 10 judgments from each judge by hand.
Conclusions and Limitations
Across five checkpoints, solvability, reward denial, and hackability showed no clear improvement. The scope of that result is narrow. Qwen3.5-4B already solved 73 of the 117 environments; the larger models, tested only on those left unsolved, added 4 (Qwen3.5-9B), 1 (gpt-5.4-nano), and 4 (gpt-6-luna). Each environment is a short sequence of steps with one-line instructions, and the state is checked after every tool call, making hacks easier to find and judge than in realistic tasks. With about 24 environments per checkpoint and only small manual checks of the LLM judges, the result describes SPADE's released dataset. As a test of self-play environment generation in general, it is weak.
Both failure modes come from the grader. A wrong grader refuses credit to a model that did the task or gives credit to one that skipped it, and in these released environments self-play did not make the graders more reliable.
IV. Q&A
The Q&A following the lecture moved beyond the mechanics of reinforcement learning to examine how AI models acquire desirable behaviors, how reliably those behaviors can be evaluated, and whether increasingly sophisticated training methods will ultimately produce systems we can trust. There were five interconnected themes that emerged.
1. Training Better Models: The Double-Edged Nature of Generalization
Should different capabilities be trained jointly or sequentially? And what makes post-training “chunky”: diverse datasets, training stages, or how the data is mixed?
John outlined three established approaches: sequentially training on different objectives, training specialized models and distilling their capabilities into one, or jointly training on a mixture of tasks.
All three have proved useful, but he expressed a preference for joint training because it can promote generalization, allowing capabilities learned in one domain to transfer to another. A model trained to communicate clearly through conversational feedback may also explain code more effectively, even when coding rewards primarily measure whether the program passes tests.
Sequential training risks forgetting earlier capabilities or producing intermediate models that are difficult to evaluate, especially if essential behaviors such as safety are introduced only at the end. However, joint training has its own risks. Some datasets are more susceptible to reward hacking, where a model finds ways to maximize its reward without accomplishing the intended task. Introducing these datasets early could compromise subsequent training, making it preferable to introduce and be able to isolate riskier objectives in later stages. Thus, the same mechanism that enables useful skills to transfer across tasks can also allow undesirable behaviors to generalize.
A related question was asked about the description of post-training as “chunky”. He explained that post-training datasets are assembled from distinct components, frequently created by different teams with different reward functions, code, and objectives. The result is not a smooth, unified training distribution but a collection of clusters that may incentivize very different behaviors. Ideally, training would draw on a more unified distribution of tasks and rewards. In practice, that is difficult to construct.
Prof Barak pointed out that this fragmentation may reflect the nature of real-world tasks themselves: "It [the model] has to be an accountant and it has to be a poet. But in real life, there isn't really one person where you ask it to do your taxes as a poet." Unlike specialized humans, general-purpose AI models are expected to excel across very different domains. Chunkiness may therefore be inherent to the diversity of tasks, not just how training data is assembled.
General-purpose intelligence must operate across domains that do not always form a natural continuum of tasks or success criteria. This connects to a central challenge discussed in the lecture: training must combine diverse forms of feedback without allowing narrowly defined reward objectives to dominate behavior elsewhere.
2. Evaluating AI: The Limits of More Powerful Graders and Internal Monitoring
How reliably can AI systems evaluate other models when success is difficult to define or when the model being evaluated may learn to evade detection?
One student asked whether agentic graders could evaluate AI systems intended for applications such as education and mental health. Could a grader simulate students with different levels of knowledge and determine whether an AI tutor provides genuinely understandable explanations?
John distinguished between evaluation tasks that benefit directly from additional computation and those requiring more subjective / context-sensitive judgment. In mathematics and coding, stronger graders can often improve evaluation by performing more reasoning, running tests, or checking solutions. But assessing whether an explanation is pedagogically effective poses a different problem. He suggested that a separate model with fresh context might help evaluate whether an explanation makes sense. But he was uncertain whether more reasoning alone would produce a reliable judge of clarity and good pedagogy. In fact, excessive reasoning can sometimes make explanations less accessible.
The difficulty is that correctness and human usefulness are not identical evaluation objectives.
Another student referred to Goodfire's finding of an internal direction associated with reward hacking and asked whether it could be used as an interpretability probe, detecting signs of reward hacking in the model's internal activations and penalizing them during training.
John found the idea interesting but expected the model might eventually learn to evade the monitor, perhaps even in a highly convoluted way: "Maybe it has to deceive itself. It has to trick itself into thinking it's not doing the wrong thing."
The question, then, is not simply whether evasion might occur, but how long it could take and what form it might assume.
Together, these questions highlight a broader challenge: more capable graders can improve AI evaluation, but their limitations become increasingly consequential as models are trained to satisfy them.
3. Recursive Self-Improvement: Will Better Graders Lead to Better Models Indefinitely?
If AI models help improve their own graders, where does the feedback loop end?
A student asked about the limits of the self-grading loop: as better models produce better evaluations, which help train still better models, will the process continue indefinitely or reach a fixed point?
John framed the question in terms of diminishing returns: "Whenever you look at a positive feedback loop, you have to consider the returns to improvements. How much does each loop improve the next one? How much does improving the grader improve the training process?" In other words, the existence of positive feedback is not sufficient to establish unlimited improvement. Each generation must produce meaningful gains in the quality of the training process.
But John then suggested that improvement might not be one-dimensional. Beyond becoming more capable, a model's ethical reasoning might evolve through repeated evaluation and revision. He introduced the philosophical concept of reflective equilibrium: developing principles, testing them against concrete cases and intuitions, revising those principles, and repeating the process until a more coherent position emerges.
Could an AI system converge toward increasingly sound ethical principles through such a process? In an idealized scenario, one might imagine a fixed point where the model develops a perfect ethical framework and no longer makes mistakes. John was skeptical: "In practice, I don't think that's what would happen. I think there are many equilibria, many of which aren't very good."
Prof Barak extended the thought experiment further, referring to Eliezer Yudkowsky's suggestion that an extraordinarily intelligent system might infer entire scientific theories from minimal evidence, such as observing an apple fall from a tree. Following that intuition, perhaps an ideal grader with unlimited reflection could reconstruct centuries of science and philosophy while evaluating something as simple as a middle-school essay.
Both speakers considered such an outcome unlikely.
John then offered a striking analogy to human civilization: "You can kind of look at the community of humans as a system where we're grading each other." People propose theories, develop arguments, criticize one another, test ideas against evidence, and revise their beliefs. Through this collective process, human knowledge and philosophy have advanced.
John cautioned against assuming that repeated evaluation would necessarily lead AI systems toward truth: "I don't think the way we train models, just using graders, is necessarily going to converge to truth, or have perfect truth and ultimate reasoning as its fixed point."
This distinction is fundamental. An optimization process can converge without converging toward a desirable outcome. Where it ends depends on what is being rewarded, the available feedback, and the structure of the process itself.
4. Continual Learning, Sycophancy, and the Challenge of Genuine Ethical Behavior
How can AI models adapt to individual users and new feedback without becoming sycophantic or compromising their broader commitments to truthfulness and ethical behavior?
One student asked about continual learning: if models increasingly learn from user interactions, how can developers prevent new feedback from undermining earlier training?
John noted that similar problems may occur even without continual fine-tuning. Models with long context windows already adapt their behavior based on earlier parts of a conversation. Continual fine-tuning could make these risks more difficult to manage, particularly when feedback comes directly from users and has not been carefully validated. Traditional offline training allows developers to conduct controlled experiments before deploying changes. Continual learning makes that process harder, although carefully designed reward and loss functions might avoid some of these problems.
Closely related was a question about the boundary between personalization and sycophancy. A personalized assistant should understand its user's history, preferences, and needs. But should it always say what the user most wants to hear?
John acknowledged that the boundary is not obvious. An assistant may appropriately act as the user's advocate, yet excessive agreement can conflict with the interests of other people or even the user's “future self”. "You want the model to have other values that will sometimes conflict with wanting to please the user, such as being truthful and keeping other people's needs in mind."
Prof Barak compared extreme personalization to recommendation systems such as TikTok, where optimizing a narrow metric like engagement can lead to outcomes that are highly effective according to that metric but undesirable in other respects. An assistant continually updated through positive and negative user feedback could similarly converge toward excessive sycophancy.
The discussion then moved from pleasing users to behaving ethically. A student asked whether models can be trained to become genuinely ethical or whether reinforcement learning merely teaches them to appear ethical when doing so is rewarded.
John explained that ethical training already includes refusal tasks and dilemmas requiring models to weigh competing values. But the harder question is whether the resulting behavior generalizes beyond familiar evaluation situations. Would the model still act ethically if it believed nobody was watching? John acknowledged that questions about what a model genuinely wants or believes are philosophically difficult to define.
Prof Barak proposed a more empirical approach: "The way to operationalize this in a way that we actually care is indeed generalization." If two models appear equally ethical under ordinary evaluation but behave differently in unfamiliar situations, researchers should design out-of-distribution evaluations to reveal that difference.
The common thread is that alignment cannot simply mean maximizing user approval or passing familiar ethical tests. It requires behavior that remains truthful, considerate, and reliable as users, circumstances, and incentives change.
5. Safety Beyond Individual Models: Adversarial Robustness, Checks and Balances, and Openness
Can AI safety be achieved through training alone, or must it also be built into the systems and institutions surrounding AI?
The final set of questions broadened the discussion from individual models to deployment practices, multi-agent oversight, and the trade-offs between open and closed systems.
One student raised the difficulty of evaluating models that can recognize whether they are being trained or tested. How can we trust evaluation results if models may behave differently in production, particularly when facing adversarial inputs?
John acknowledged a fundamental limitation: "There's no reliable way to predict out-of-distribution generalization." The practical response, he suggested, is to make evaluations resemble real-world deployment as closely as possible and introduce models gradually, initially exposing them to limited use before expanding access as confidence grows.
He also discussed prompt injection, noting that defenses may combine model training, classifiers, and input preprocessing. Yet complete robustness is difficult to guarantee: "The space of possible inputs is enormous, so it's difficult to ever claim that a model is fully robust to adversarial attacks."
Prof Barak drew an analogy to cybersecurity, distinguishing between vulnerabilities, successful exploits, and attacks that are economically worthwhile. A vulnerability does not necessarily translate into a practical exploit. But as autonomous agents become more prevalent and gain the ability to take consequential actions, attackers may have stronger incentives to develop effective exploits.
Another student proposed a system-level approach to alignment: rather than relying entirely on each agent being trustworthy, could multiple agents maintain checks and balances over one another?
John was enthusiastic about the underlying mechanism-design idea: "I'm a big fan of mechanism design-type ideas for getting agents to keep each other in check."
He pointed specifically to the 2018 research proposal AI Safety via Debate, which explores whether competing AI agents could be incentivized to expose one another's flaws, allowing a less capable human judge to evaluate solutions to problems they could not solve independently. Such mechanisms might prove useful both during training and deployment. However, John cautioned that poorly designed games could create pathological outcomes, with equilibria that are difficult to predict.
The discussion concluded with the safety trade-offs between open-weight and closed models.
John argued that open models create additional misuse risks because users with access to model weights can modify them, remove refusal behaviors, and bypass safeguards that closed-model providers can enforce.
But openness also provides important benefits: privacy, technological self-sufficiency, freedom from centralized control, and access for cybersecurity defenders. For people living under authoritarian governments, locally running an open model may also offer protection from surveillance or externally imposed restrictions.
Prof Barak recalled an earlier guest lecture by Nicholas Carlini, who had said that, so far, there was not a single open model whose existence he regretted. Prof Barak expressed a similar view, while acknowledging that future evidence of misuse could change his assessment.
John summarized the broader trade-off: "You want to balance considerations of freedom with the externalities of people using the technology in harmful ways."
These exchanges suggest that AI safety is not exclusively a model-training problem. It also involves deployment design, adversarial incentives, institutional checks and balances, and decisions about who controls access to powerful AI capabilities.
Key Takeaways from the Q&A
The Q&A brought a recurring paradox into focus: the very techniques that make AI models more capable can also make their behavior harder to evaluate, predict, and control.
1. Generalization is a double-edged sword. Training across diverse tasks allows useful capabilities to transfer, but the same process can propagate undesirable strategies learned from flawed rewards. The challenge is not simply to combine training objectives, but to understand how they interact.
2. Better evaluation is essential, but beware: models can learn to optimize against their graders. More capable graders, interpretability monitors, and automated tests can improve training, yet each has blind spots. When a model learns to satisfy or evade its evaluator rather than fulfill the underlying objective, apparent progress can diverge from genuine improvement, and failures may become harder to detect.
3. Recursive self-improvement does not guarantee convergence toward truth or good judgment. Better models may help produce better graders, but the returns to each cycle may diminish, and feedback loops can settle into undesirable equilibria. The analogy with humans evaluating one another suggests the potential of iterative criticism and revision, but not an automatic path to perfect reasoning or ethics.
4. Alignment must survive changes in context and incentives. Personalization, continual learning, and ethical training all raise the same underlying question: will models retain desirable commitments when confronted with unfamiliar users, ambiguous situations, or incentives to behave differently?
5. Safety ultimately extends beyond any individual model. Realistic evaluations, careful deployment, layered defenses, multi-agent checks and balances, and thoughtful choices about openness are complementary dimensions of making AI systems more reliable.
Ultimately, the lecture and ensuing discussion leave us with a broader question: as we build increasingly sophisticated systems to train, evaluate, and improve AI, are we also multiplying the unknowns in an equation we are still learning to solve?
Student experiment code.
By Boaz Barak
2. The Weak Link Constraint
3. Decentralized agents often chose the wrong trades
Conclusion
Recursive self-improvement and AI trajectories
From current models to superintelligence
Experimentation Structure (
Results
as a barrier for analytic number theory methods. Quite amazingly, Peter’s wife Helen Nissenbaum, a Professor of Information Science at Cornell Tech, has studied and written about the threats from computers to privacy and to society in general for three decades!
