283 lines
8.2 KiB
Markdown
283 lines
8.2 KiB
Markdown
---
|
||
# theme id, package name, or local path
|
||
theme: seriph
|
||
title: Scalable Oversight for Complex AI Tasks
|
||
titleTemplate: '%s - AI Safety & Oversight'
|
||
author: Rossi Stefano
|
||
info: |
|
||
## Methods for Scaling Human Feedback in AI Supervision
|
||
keywords: AI Safety, Scalable Oversight, LLMs, Human Feedback, Alignment, AI Debate
|
||
mdc: true
|
||
hideInToc: false
|
||
addons:
|
||
- slidev-addon-rabbit
|
||
- slidev-addon-python-runner
|
||
python:
|
||
installs: []
|
||
prelude: ''
|
||
loadPackagesFromImports: true
|
||
suppressDeprecationWarnings: true
|
||
alwaysReload: false
|
||
loadPyodideOptions: {}
|
||
presenter: true
|
||
browserExporter: dev
|
||
download: true
|
||
exportFilename: scalable-oversight-for-ai
|
||
twoslash: false
|
||
lineNumbers: true
|
||
monaco: false
|
||
selectable: false
|
||
record: dev
|
||
contextMenu: dev
|
||
wakeLock: true
|
||
overviewSnapshots: false
|
||
colorSchema: dark
|
||
routerMode: history
|
||
aspectRatio: 16/9
|
||
canvasWidth: 980
|
||
css:
|
||
- unocss
|
||
unocss:
|
||
configFile: './uno.config.ts'
|
||
defaults:
|
||
layout: center
|
||
drawings:
|
||
enabled: true
|
||
persist: false
|
||
presenterOnly: false
|
||
syncAll: true
|
||
htmlAttrs:
|
||
dir: ltr
|
||
lang: en
|
||
transition: slide-left
|
||
background: none
|
||
---
|
||
|
||
<!-- INTRO SLIDE -->
|
||
<div class="flex flex-col items-center justify-center h-full py-10">
|
||
<h1 class="text-center text-5xl font-bold gradient-text mb-10">Scalable Oversight for Complex AI</h1>
|
||
<h2 class="text-center text-4xl mb-6" style="color: var(--accent-color);">Can Human Feedback Keep Up?</h2>
|
||
<h3 class="text-center text-3xl mb-14 animate-pulse highlight-word">Techniques to Align Large Language Models at Scale</h3>
|
||
<div class="flex w-full justify-between mt-auto">
|
||
<div class="text-left text-xl">Stefano Rossi</div>
|
||
<div class="text-right text-xl">2 May, 2025</div>
|
||
</div>
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
</div>
|
||
|
||
---
|
||
|
||
# Introduction
|
||
|
||
<div class="grid grid-cols-2 gap-6">
|
||
<div class="panel-info">
|
||
<ul>
|
||
<li><span class="highlight-word">Alignment at scale</span></li>
|
||
<li><span class="highlight-word">Human limitations</span></li>
|
||
<li><span class="highlight-word">Model deception</span></li>
|
||
</ul>
|
||
</div>
|
||
<div class="panel-success">
|
||
<ul>
|
||
<li><span class="highlight-word">Recursive techniques</span></li>
|
||
<li><span class="highlight-word">AI-augmented feedback</span></li>
|
||
<li><span class="highlight-word">Factored cognition</span></li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# The Problem with Feedback
|
||
|
||
<div class="two-column">
|
||
<div class="panel-info">
|
||
<h2>Why it fails</h2>
|
||
<ul>
|
||
<li>Too complex for humans to judge outputs correctly</li>
|
||
<li><span class="highlight-word">Deception</span> and hallucinations trick reviewers</li>
|
||
<li>LLMs <span class="highlight-word">sycophantically agree</span> rather than pursue truth</li>
|
||
</ul>
|
||
</div>
|
||
<div class="panel-warning">
|
||
<h2>Why it matters</h2>
|
||
<ul>
|
||
<li>Most alignment methods rely on human supervision</li>
|
||
<li><span class="highlight-word">RLHF breaks</span> at scale and complexity</li>
|
||
<li>Feedback noise → <span class="highlight-word">reward hacking</span> and misalignment</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# What Is Scalable Oversight?
|
||
|
||
<div class="panel-info">
|
||
<p>Scalable oversight techniques aim to <span class="highlight-word">empower humans</span> to give accurate feedback on complex tasks.</p>
|
||
<p>They leverage structured methods, AI assistants, or recursive mechanisms to <span class="highlight-word">extend our cognitive reach</span>.</p>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Techniques Overview
|
||
|
||
<table class="styled-table hoverable-table">
|
||
<thead>
|
||
<tr>
|
||
<th>Technique</th>
|
||
<th>Description</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr><td><span class="highlight-word">Iterated Amplification</span></td><td>Task decomposition + model assistance = scalable evaluation</td></tr>
|
||
<tr><td><span class="highlight-word">Recursive Reward Modeling</span></td><td>AI helps humans give better feedback to train better AI</td></tr>
|
||
<tr><td><span class="highlight-word">Constitutional AI</span></td><td>Use fixed rules to guide feedback generation</td></tr>
|
||
<tr><td><span class="highlight-word">Debate</span></td><td>AIs argue, human judges; surfacing deception through opposition</td></tr>
|
||
<tr><td><span class="highlight-word">Weak-to-Strong Generalization</span></td><td>Train powerful models with weaker labels</td></tr>
|
||
</tbody>
|
||
</table>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Iterated Amplification (IDA)
|
||
|
||
<div class="panel-info">
|
||
<h2>Core Idea</h2>
|
||
<ul>
|
||
<li>Decompose hard problems</li>
|
||
<li>Evaluate sub-steps</li>
|
||
<li>Train model to replicate the full pipeline</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="panel-success">
|
||
<h2>Example</h2>
|
||
<ul>
|
||
<li>Book summarization</li>
|
||
<li>Summarize pages → chapters → full book</li>
|
||
<li>Distill into a one-shot summary model</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Debate
|
||
|
||
<div class="panel-info">
|
||
<p>Two AIs take opposing views on a complex topic and argue it out. A human (or weaker AI) acts as judge.</p>
|
||
<ul>
|
||
<li>Helps surface <span class="highlight-word">flaws and manipulations</span></li>
|
||
<li>Leverages model’s reasoning capacity</li>
|
||
<li>Challenges: truth ≠ persuasion, collusion risk</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Constitutional AI
|
||
|
||
<div class="panel-info">
|
||
<p>Replace human feedback with <span class="highlight-word">AI critiques</span> guided by human-written principles.</p>
|
||
<p>Used in Anthropic’s Claude models, where rules encode ethical and practical constraints.</p>
|
||
<ul>
|
||
<li>Scalable via automation</li>
|
||
<li>Fewer humans involved, more repeatability</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Weak-to-Strong Generalization
|
||
|
||
<div class="panel-info">
|
||
<ul>
|
||
<li>Train GPT-4 using feedback from GPT-2</li>
|
||
<li>GPT-4 learns better than its weak teacher</li>
|
||
<li><span class="highlight-word">Generalization beats imitation</span></li>
|
||
<li>Bootstrapping + confidence loss = better performance</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Key Challenges
|
||
|
||
<div class="two-column">
|
||
<div class="panel-warning">
|
||
<h2>Failure Risks</h2>
|
||
<ul>
|
||
<li>Models appear aligned, but aren’t</li>
|
||
<li><span class="highlight-word">Deceptive behavior</span> during training</li>
|
||
<li>Generalization failures</li>
|
||
</ul>
|
||
</div>
|
||
<div class="panel-danger">
|
||
<h2>Limitations</h2>
|
||
<ul>
|
||
<li>Factored cognition doesn’t scale universally</li>
|
||
<li>Feedback may be misinterpreted</li>
|
||
<li>Ongoing distributional shift at deployment</li>
|
||
</ul>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
# Conclusion
|
||
|
||
<div class="panel-success">
|
||
<p><strong>Scalable oversight is not a silver bullet.</strong> But it’s the best shot we’ve got at aligning complex systems to human values, or at least, approximations of them.</p>
|
||
<p>The future of AI safety <span class="highlight-word">depends on teaching AIs to teach themselves</span>, with guardrails we can actually monitor.</p>
|
||
</div>
|
||
|
||
<div class="hud-element circle-small"></div>
|
||
<div class="hud-element circle-big"></div>
|
||
<div class="hud-lines"></div>
|
||
|
||
---
|
||
|
||
<div class="bouncing-box">
|
||
<div class="screensaver-icon ai"><i class="fas fa-brain"></i></div>
|
||
<h1 class="multicolor-text z-10 relative">Questions?</h1>
|
||
</div>
|
||
|
||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">
|