---
# theme id, package name, or local path
theme: seriph
title: Scalable Oversight for Complex AI Tasks
titleTemplate: '%s - AI Safety & Oversight'
author: Rossi Stefano
info: |
  ## Methods for Scaling Human Feedback in AI Supervision
keywords: AI Safety, Scalable Oversight, LLMs, Human Feedback, Alignment, AI Debate
mdc: true
hideInToc: false
addons:
  - slidev-addon-rabbit
  - slidev-addon-python-runner
python:
  installs: []
  prelude: ''
  loadPackagesFromImports: true
  suppressDeprecationWarnings: true
  alwaysReload: false
  loadPyodideOptions: {}
presenter: true
browserExporter: dev
download: true
exportFilename: scalable-oversight-for-ai
twoslash: false
lineNumbers: true
monaco: false
selectable: false
record: dev
contextMenu: dev
wakeLock: true
overviewSnapshots: false
colorSchema: dark
routerMode: history
aspectRatio: 16/9
canvasWidth: 980
css:
  - unocss
unocss:
  configFile: './uno.config.ts'
defaults:
  layout: center
drawings:
  enabled: true
  persist: false
  presenterOnly: false
  syncAll: true
htmlAttrs:
  dir: ltr
  lang: en
transition: slide-left
background: none
---

<!-- INTRO SLIDE -->
<div class="flex flex-col items-center justify-center h-full py-10">
  <h1 class="text-center text-5xl font-bold gradient-text mb-10">Scalable Oversight for Complex AI</h1>
  <h2 class="text-center text-4xl mb-6" style="color: var(--accent-color);">Can Human Feedback Keep Up?</h2>
  <h3 class="text-center text-3xl mb-14 animate-pulse highlight-word">Techniques to Align Large Language Models at Scale</h3>
  <div class="flex w-full justify-between mt-auto">
    <div class="text-left text-xl">Stefano Rossi</div>
    <div class="text-right text-xl">2 May, 2025</div>
  </div>
  <div class="hud-element circle-small"></div>
  <div class="hud-element circle-big"></div>
  <div class="hud-lines"></div>
</div>

---

# Introduction

<div class="grid grid-cols-2 gap-6">
  <div class="panel-info">
    <ul>
      <li><span class="highlight-word">Alignment at scale</span></li>
      <li><span class="highlight-word">Human limitations</span></li>
      <li><span class="highlight-word">Model deception</span></li>
    </ul>
  </div>
  <div class="panel-success">
    <ul>
      <li><span class="highlight-word">Recursive techniques</span></li>
      <li><span class="highlight-word">AI-augmented feedback</span></li>
      <li><span class="highlight-word">Factored cognition</span></li>
    </ul>
  </div>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# The Problem with Feedback

<div class="two-column">
  <div class="panel-info">
    <h2>Why it fails</h2>
    <ul>
      <li>Too complex for humans to judge outputs correctly</li>
      <li><span class="highlight-word">Deception</span> and hallucinations trick reviewers</li>
      <li>LLMs <span class="highlight-word">sycophantically agree</span> rather than pursue truth</li>
    </ul>
  </div>
  <div class="panel-warning">
    <h2>Why it matters</h2>
    <ul>
      <li>Most alignment methods rely on human supervision</li>
      <li><span class="highlight-word">RLHF breaks</span> at scale and complexity</li>
      <li>Feedback noise → <span class="highlight-word">reward hacking</span> and misalignment</li>
    </ul>
  </div>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# What Is Scalable Oversight?

<div class="panel-info">
  <p>Scalable oversight techniques aim to <span class="highlight-word">empower humans</span> to give accurate feedback on complex tasks.</p>
  <p>They leverage structured methods, AI assistants, or recursive mechanisms to <span class="highlight-word">extend our cognitive reach</span>.</p>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Techniques Overview

<table class="styled-table hoverable-table">
  <thead>
    <tr>
      <th>Technique</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr><td><span class="highlight-word">Iterated Amplification</span></td><td>Task decomposition + model assistance = scalable evaluation</td></tr>
    <tr><td><span class="highlight-word">Recursive Reward Modeling</span></td><td>AI helps humans give better feedback to train better AI</td></tr>
    <tr><td><span class="highlight-word">Constitutional AI</span></td><td>Use fixed rules to guide feedback generation</td></tr>
    <tr><td><span class="highlight-word">Debate</span></td><td>AIs argue, human judges; surfacing deception through opposition</td></tr>
    <tr><td><span class="highlight-word">Weak-to-Strong Generalization</span></td><td>Train powerful models with weaker labels</td></tr>
  </tbody>
</table>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Iterated Amplification (IDA)

<div class="panel-info">
  <h2>Core Idea</h2>
  <ul>
    <li>Decompose hard problems</li>
    <li>Evaluate sub-steps</li>
    <li>Train model to replicate the full pipeline</li>
  </ul>
</div>

<div class="panel-success">
  <h2>Example</h2>
  <ul>
    <li>Book summarization</li>
    <li>Summarize pages → chapters → full book</li>
    <li>Distill into a one-shot summary model</li>
  </ul>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Debate

<div class="panel-info">
  <p>Two AIs take opposing views on a complex topic and argue it out. A human (or weaker AI) acts as judge.</p>
  <ul>
    <li>Helps surface <span class="highlight-word">flaws and manipulations</span></li>
    <li>Leverages model’s reasoning capacity</li>
    <li>Challenges: truth ≠ persuasion, collusion risk</li>
  </ul>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Constitutional AI

<div class="panel-info">
  <p>Replace human feedback with <span class="highlight-word">AI critiques</span> guided by human-written principles.</p>
  <p>Used in Anthropic’s Claude models, where rules encode ethical and practical constraints.</p>
  <ul>
    <li>Scalable via automation</li>
    <li>Fewer humans involved, more repeatability</li>
  </ul>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Weak-to-Strong Generalization

<div class="panel-info">
  <ul>
    <li>Train GPT-4 using feedback from GPT-2</li>
    <li>GPT-4 learns better than its weak teacher</li>
    <li><span class="highlight-word">Generalization beats imitation</span></li>
    <li>Bootstrapping + confidence loss = better performance</li>
  </ul>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Key Challenges

<div class="two-column">
  <div class="panel-warning">
    <h2>Failure Risks</h2>
    <ul>
      <li>Models appear aligned, but aren’t</li>
      <li><span class="highlight-word">Deceptive behavior</span> during training</li>
      <li>Generalization failures</li>
    </ul>
  </div>
  <div class="panel-danger">
    <h2>Limitations</h2>
    <ul>
      <li>Factored cognition doesn’t scale universally</li>
      <li>Feedback may be misinterpreted</li>
      <li>Ongoing distributional shift at deployment</li>
    </ul>
  </div>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

# Conclusion

<div class="panel-success">
  <p><strong>Scalable oversight is not a silver bullet.</strong> But it’s the best shot we’ve got at aligning complex systems to human values, or at least, approximations of them.</p>
  <p>The future of AI safety <span class="highlight-word">depends on teaching AIs to teach themselves</span>, with guardrails we can actually monitor.</p>
</div>

<div class="hud-element circle-small"></div>
<div class="hud-element circle-big"></div>
<div class="hud-lines"></div>

---

<div class="bouncing-box">
  <div class="screensaver-icon ai"><i class="fas fa-brain"></i></div>
  <h1 class="multicolor-text z-10 relative">Questions?</h1>
</div>

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">