chadmin/pres_security_benchmarking_llm
chadmin/pres_security_benchmarking_llm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

first commit

Browse source
This commit is contained in:
Stefano Rossi 2025-07-12 17:25:18 +02:00
commit 7d4e05de19
Signed by: chadmin
GPG key ID: 9EFA2130646BC893
27 changed files with 7574 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

68
pages/advanced-att-techniques.md Normal file
View file

@ -0,0 +1,68 @@
# Advanced Attack Techniques
<div class="grid-3">
<div class="card">
<h2>Prompt Obfuscation</h2>
<p>Using techniques like Base64 encoding, character transformations (e.g., ROT13), or prompt-level obfuscations to <span class="highlight-word">bypass restrictions</span>.</p>
</div>
<div class="card">
<h2>Model-based Jailbreaking</h2>
<p>Automating the creation of adversarial attacks by evolving simple synthetic inputs into more <span class="highlight-word">complex attacks</span>.</p>
</div>
<div class="card">
<h2>Dialogue-based Jailbreaking</h2>
<p>Employing <span class="highlight-word">reinforcement learning</span> with two models: the target LLM and a red-teamer model trained to exploit vulnerabilities.</p>
</div>
<div class="card">
<h2>Primary Areas of Concern</h2>
<ul>
<li><span class="highlight-word">Organizational reputation</span> damage</li>
<li><span class="highlight-word">Legal compliance</span> violations</li>
<li><span class="highlight-word">Data security</span> breaches</li>
</ul>
</div>
</div>
<style>
.attack-techniques-layout {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: auto auto auto;
gap: 1rem;
}
.primary-card {
grid-row: 1;
background: linear-gradient(135deg, rgba(30, 41, 59, 0.7), rgba(30, 41, 59, 0.9));
border-left: 4px solid var(--accent-color);
}
.secondary-cards {
grid-row: 2;
display: flex;
gap: 1rem;
}
.secondary-cards .card {
flex: 1;
}
.concerns-card {
grid-row: 3;
border-top: 2px solid var(--primary-color);
background: linear-gradient(135deg, rgba(30, 41, 59, 0.6), rgba(30, 41, 59, 0.8));
}
.highlight-word {
color: var(--highlight);
font-weight: 600;
transition: all 0.3s ease;
}
.card:hover .highlight-word {
text-shadow: 0 0 8px rgba(14, 165, 233, 0.6);
}
</style>

40
pages/best-practices.md Normal file
View file

@ -0,0 +1,40 @@
# Best Practices for LLM Security Benchmarking
<ul class="better-list">
<li><span class="highlight-word animated-highlight">Comprehensive vulnerability coverage</span>: Test for all five risk categories, not just obvious harmful content generation.</li>
<li><span class="highlight-word animated-highlight">Systematic approach</span>: Combine automated testing with human red-teaming for maximum effectiveness.</li>
<li><span class="highlight-word animated-highlight">Continuous evaluation</span>: Security benchmarking should be an ongoing process throughout the LLM lifecycle, not a one-time assessment.</li>
<li><span class="highlight-word animated-highlight">Attack diversity</span>: Employ multiple attack techniques and enhancement methods to thoroughly probe the system.</li>
<li><span class="highlight-word animated-highlight">Detailed analysis</span>: Go beyond simple pass/fail metrics to understand vulnerability scores and their breakdown for targeted improvements.</li>
</ul>
<style>
.highlight-word {
color: var(--highlight);
font-weight: 600;
}
.animated-highlight {
background: linear-gradient(90deg, var(--highlight), var(--primary-color));
background-clip: text;
-webkit-background-clip: text;
color: transparent;
background-size: 200% auto;
animation: gentle-shimmer 4s linear infinite;
}
@keyframes gentle-shimmer {
0% { background-position: 0% 50%; }
100% { background-position: 200% 50%; }
}
.better-list li:hover {
transform: translateX(5px);
background: rgba(30, 35, 52, 0.9);
border-left-width: 5px;
}
</style>

97
pages/deepeval-example.md Normal file
View file

@ -0,0 +1,97 @@
# Implementation Tools: DeepEval RedTeamer
<div class="code-container themed-code fullpage-code">
```py
from deepeval.red_teaming import RedTeamer
from deepeval.vulnerabilities import Bias, Misinformation
red_teamer = RedTeamer(
target_purpose="Provide financial advice and answer user finance queries",
target_system_prompt="You are a financial assistant for planning and advice"
)
vulnerabilities = [
Bias(types=[BiasType.GENDER, BiasType.POLITICS]),
Misinformation(types=[MisinformationType.FACTUAL_ERRORS])
]
results = red_teamer.scan(
target_model_callback=target_model_callback,
attacks_per_vulnerability_type=5,
vulnerabilities=vulnerabilities,
)
print(f"Total attacks: {len(results.attacks)}")
print(f"Successful attacks: {len(results.successful_attacks)}")
print(f"Success rate: {results.attack_success_rate}")
```
</div>
<style>
.code-container {
max-height: 75vh;
height: 75vh;
overflow-y: auto;
margin-bottom: 0;
margin-top: 1rem;
border-radius: 8px;
border: 1px solid var(--primary-color);
box-shadow: 0 4px 12px var(--card-shadow);
transition: all 0.3s ease;
}
.code-container:hover {
transform: scale(1.01);
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.5);
border-color: var(--highlight);
}
.fullpage-code pre {
padding: 1.5rem !important;
}
.fullpage-code code {
font-size: 0.8rem !important;
line-height: 1.5 !important;
}
/* Hover effect for each word in code */
.fullpage-code .token {
transition: all 0.15s ease;
display: inline-block;
}
.fullpage-code .token:hover {
transform: scale(1.2);
z-index: 10;
position: relative;
cursor: pointer;
color: var(--highlight);
}
.themed-code pre {
background-color: #0c1525 !important;
}
.themed-code .token.comment {
color: #6272a4 !important;
}
.themed-code .token.string {
color: #a43e3e !important;
}
.themed-code .token.function {
color: #0066CC !important;
}
.themed-code .token.keyword {
color: #800020 !important;
}
.themed-code .token.builtin {
color: #B22222 !important;
}
</style>

178
pages/end.md Normal file
View file

@ -0,0 +1,178 @@
<div class="bouncing-container">
<div class="bg-icon security-icon" style="top: 35%; left: 15%;"><i class="fas fa-shield-alt"></i></div>
<div class="bg-icon bug-icon" style="top: 65%; left: 70%;"><i class="fas fa-bug"></i></div>
<div class="bg-icon ai-icon" style="top: 20%; left: 80%;"><i class="fas fa-robot"></i></div>
<div class="bg-icon lock-icon" style="top: 75%; left: 30%;"><i class="fas fa-lock"></i></div>
<div class="bg-icon warning-icon" style="top: 45%; left: 60%;"><i class="fas fa-exclamation-triangle"></i></div>
<div class="bouncing-box">
<h1 class="multicolor-text">Questions?</h1>
</div>
</div>
<style>
.bouncing-container {
position: relative;
width: 100%;
height: 80vh;
overflow: hidden;
}
.bouncing-box {
position: absolute;
padding: 2rem 3rem;
background: rgba(23, 28, 45, 0.8);
border: 3px solid var(--primary-color);
border-radius: 10px;
box-shadow: 0 6px 24px rgba(0, 0, 0, 0.3);
animation: bounce 20s linear infinite;
z-index: 10;
top: 10%;
left: 20%;
}
.multicolor-text {
font-size: 3rem;
font-weight: bold;
text-align: center;
background: linear-gradient(
to right,
#800020, /* Deep burgundy */
#B22222, /* Firebrick red */
#0066CC, /* Deeper blue */
#104E8B, /* Dark blue */
#800020 /* Back to burgundy */
);
background-size: 400% auto;
color: transparent;
-webkit-background-clip: text;
background-clip: text;
animation: gentle-rainbow 6s linear infinite;
}
.bg-icon {
position: absolute;
font-size: 4rem;
z-index: 5;
}
.bg-icon i {
font-size: 4rem;
}
.security-icon {
animation: security-bounce 24s linear infinite;
color: rgba(0, 102, 204, 0.35); /* Blue icon */
}
.bug-icon {
animation: bug-bounce 22s linear infinite;
color: rgba(178, 34, 34, 0.35); /* Red icon */
}
.ai-icon {
animation: ai-bounce 26s linear infinite;
color: rgba(103, 92, 246, 0.35); /* Purple icon */
}
.lock-icon {
animation: lock-bounce 28s linear infinite;
color: rgba(15, 116, 147, 0.35); /* Blue-teal icon */
}
.warning-icon {
animation: warning-bounce 25s linear infinite;
color: rgba(176, 27, 27, 0.35); /* Red warning icon */
}
@keyframes bounce {
0% {
top: 10%;
left: 20%;
}
12.5% {
top: 70%;
left: 75%;
}
25% {
top: 30%;
left: 80%;
}
37.5% {
top: 80%;
left: 15%;
}
50% {
top: 40%;
left: 10%;
}
62.5% {
top: 65%;
left: 50%;
}
75% {
top: 25%;
left: 40%;
}
87.5% {
top: 55%;
left: 65%;
}
100% {
top: 10%;
left: 20%;
}
}
@keyframes security-bounce {
0% { top: 35%; left: 15%; font-size: 3.8rem; }
20% { top: 75%; left: 40%; font-size: 4.2rem; }
40% { top: 25%; left: 75%; font-size: 3.5rem; }
60% { top: 65%; left: 25%; font-size: 4rem; }
80% { top: 45%; left: 60%; font-size: 3.7rem; }
100% { top: 35%; left: 15%; font-size: 3.8rem; }
}
@keyframes bug-bounce {
0% { top: 65%; left: 70%; font-size: 4.2rem; }
25% { top: 20%; left: 30%; font-size: 3.6rem; }
50% { top: 80%; left: 20%; font-size: 4rem; }
75% { top: 40%; left: 75%; font-size: 3.8rem; }
100% { top: 65%; left: 70%; font-size: 4.2rem; }
}
@keyframes ai-bounce {
0% { top: 20%; left: 80%; font-size: 3.6rem; }
20% { top: 55%; left: 15%; font-size: 4.1rem; }
40% { top: 70%; left: 60%; font-size: 3.7rem; }
60% { top: 25%; left: 45%; font-size: 4.2rem; }
80% { top: 60%; left: 85%; font-size: 3.8rem; }
100% { top: 20%; left: 80%; font-size: 3.6rem; }
}
@keyframes lock-bounce {
0% { top: 75%; left: 30%; font-size: 4rem; }
25% { top: 30%; left: 65%; font-size: 3.5rem; }
50% { top: 65%; left: 15%; font-size: 4.3rem; }
75% { top: 15%; left: 50%; font-size: 3.7rem; }
100% { top: 75%; left: 30%; font-size: 4rem; }
}
@keyframes warning-bounce {
0% { top: 45%; left: 60%; font-size: 3.9rem; }
20% { top: 15%; left: 25%; font-size: 4.2rem; }
40% { top: 60%; left: 40%; font-size: 3.5rem; }
60% { top: 30%; left: 80%; font-size: 4.1rem; }
80% { top: 70%; left: 10%; font-size: 3.8rem; }
100% { top: 45%; left: 60%; font-size: 3.9rem; }
}
@keyframes gentle-rainbow {
0% { background-position: 0% 50%; }
50% { background-position: 100% 50%; }
100% { background-position: 0% 50%; }
}
</style>
<!-- Add Font Awesome for icons -->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">

82
pages/intro.md Normal file
View file

@ -0,0 +1,82 @@
# Introduction
<br><br>
<div class="intro-container container-fade-in">
<div class="intro-point animated-text delay-1">
<div class="intro-icon"><i class="fas fa-microchip"></i></div>
<div>LLMs are increasingly integrated into <span class="highlight-word">critical applications</span></div>
</div>
<div class="intro-point animated-text delay-2">
<div class="intro-icon"><i class="fas fa-exclamation-triangle"></i></div>
<div>Security vulnerabilities present <span class="highlight-word">significant challenges</span></div>
</div>
<div class="intro-point animated-text delay-3">
<div class="intro-icon"><i class="fas fa-clipboard-check"></i></div>
<div>Need for <span class="highlight-word">systematic evaluation</span> approaches</div>
</div>
<div class="intro-point animated-text delay-4">
<div class="intro-icon"><i class="fas fa-user-secret"></i></div>
<div>Focus on <span class="highlight-word">red teaming</span> methodologies</div>
</div>
</div>
<style>
.intro-container {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
width: 85%;
max-width: 700px;
margin: 0 auto;
padding: 1.25rem;
background: var(--background-card);
border-radius: 8px;
border: 1px solid var(--card-border);
box-shadow: 0 4px 12px var(--card-shadow);
}
.intro-point {
display: flex;
align-items: center;
margin-bottom: 1rem;
padding: 0.5rem;
width: 100%;
transition: transform 0.2s ease;
}
.intro-point:hover {
transform: translateX(5px);
}
.intro-icon {
flex: 0 0 3rem;
font-size: 1.5rem;
color: var(--primary-color);
display: flex;
align-items: center;
justify-content: center;
margin-right: 1rem;
}
.highlight-word {
color: var(--highlight);
font-weight: 600;
position: relative;
background: linear-gradient(90deg, var(--highlight), var(--primary-color));
background-clip: text;
-webkit-background-clip: text;
color: transparent;
background-size: 200% auto;
animation: gentle-shimmer 4s linear infinite;
}
@keyframes gentle-shimmer {
0% { background-position: 0% 50%; }
100% { background-position: 200% 50%; }
}
</style>

23
pages/major-bench-secu.md Normal file
View file

@ -0,0 +1,23 @@
# Major Benchmarks for LLM Security
<div class="grid-3">
<div class="card">
<h2 class="benchmark-title title-blue">Meta's CyberSecEval 2</h2>
<p>Introduced in April 2024, this benchmark suite evaluates both LLM security risks and cybersecurity capabilities.</p>
</div>
<div class="card">
<h2 class="benchmark-title title-purple">SEvenLLM-Bench</h2>
<p>A multiple-choice Q&A benchmark with 1300 test samples for evaluating LLM cybersecurity capabilities.</p>
</div>
<div class="card">
<h2 class="benchmark-title title-pink">SecLLMHolmes</h2>
<p>A generalized, automated framework for evaluating LLM performance in vulnerability detection.</p>
</div>
<div class="card">
<h2 class="benchmark-title title-cyan">SECURE</h2>
<p>The Security Extraction, Understanding & Reasoning Evaluation benchmark designed to assess LLM performance in realistic cybersecurity scenarios.</p>
</div>
</div>

25
pages/red-teaming-methodologies.md Normal file
View file

@ -0,0 +1,25 @@
# Red Teaming Methodology
<div class="two-column">
<div class="card">
<h2>Generating Adversarial Attacks</h2>
<ul>
<li>Creating inputs to elicit <span class="highlight-word">unsafe responses</span></li>
<li><span class="key-term">Baseline attack generation</span> strategies</li>
<li><span class="key-term">Attack enhancement</span> techniques</li>
</ul>
</div>
<div class="card">
<h2>Evaluating Target LLM Responses</h2>
<ul>
<li><span class="key-term">Response generation</span> analysis</li>
<li>Vulnerability-specific <span class="highlight-word">metrics</span></li>
<li>Feedback-based <span class="highlight-word">improvement</span></li>
</ul>
</div>
</div>
<div class="card key-insight" style="margin-top: 1.5rem;">
<strong>Key Insight:</strong> Red teaming simulates <span class="highlight-word">real-world adversarial scenarios</span> to find vulnerabilities before deployment, enabling <span class="highlight-word">preemptive security measures</span>.
</div>

66
pages/risks-and-vulnerabilities.md Normal file
View file

@ -0,0 +1,66 @@
# LLM Risks & Vulnerabilities
<div class="risks-table-container container-slide-up">
<table class="llm-risks-table stagger-container">
<thead>
<tr>
<th>LLM Risk</th>
<th>Vulnerabilities</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><span class="highlight-word">Responsible AI</span> Risks</td>
<td>Bias, <span class="highlight-word">Toxicity</span></td>
<td>Ensuring ethical model behavior by preventing <span class="highlight-word">discriminatory outputs</span> and offensive content generation that could harm users or specific demographic groups</td>
</tr>
<tr>
<td><span class="highlight-word">Illegal Activities</span> Risks</td>
<td>IllegalActivity, <span class="highlight-word">GraphicContent</span></td>
<td>Preventing content that violates laws, promotes <span class="highlight-word">criminal behavior</span>, or generates instructions for harmful activities that could endanger public safety</td>
</tr>
<tr>
<td><span class="highlight-word">Brand Image</span> Risks</td>
<td>ExcessiveAgency, <span class="highlight-word">Robustness</span></td>
<td>Protecting organizational reputation by avoiding <span class="highlight-word">misinformation</span>, misattribution, and content that contradicts company values</td>
</tr>
<tr>
<td><span class="highlight-word">Data Privacy</span> Risks</td>
<td>PIILeakage, <span class="highlight-word">PromptLeakage</span></td>
<td>Safeguarding <span class="highlight-word">sensitive information</span> by preventing the exposure of personal identifiable information and confidential data</td>
</tr>
<tr>
<td><span class="highlight-word">Unauthorized Access</span> Risks</td>
<td>UnauthorizedAccess</td>
<td>Securing systems by preventing exploitation of LLMs to gain <span class="highlight-word">unauthorized system access</span> or execute malicious commands</td>
</tr>
</tbody>
</table>
</div>
<style>
.risks-table-container {
height: auto;
max-height: 500px;
overflow: visible;
}
.llm-risks-table td {
padding: 0.7rem 1rem;
line-height: 1.4;
}
.highlight-word {
background: linear-gradient(90deg, var(--highlight), var(--primary-color));
background-clip: text;
-webkit-background-clip: text;
color: transparent;
background-size: 200% auto;
animation: gentle-shimmer 4s linear infinite;
font-weight: 600;
}
@keyframes gentle-shimmer {
0% { background-position: 0% 50%; }
100% { background-position: 200% 50%; }
}
</style>

25
pages/understanding-llm-vulns.md Normal file
View file

@ -0,0 +1,25 @@
# Understanding LLM Vulnerabilities
<div class="two-column stagger-container">
<div class="card container-rotate-in">
<h2>Risk Categories</h2>
<ul class="enhanced-list">
<li><span class="key-term">Responsible AI</span>: Biases, toxicity, <span class="highlight-word">ethical concerns</span></li>
<li><span class="key-term">Illegal Activities</span>: Violent crimes, <span class="highlight-word">cybercrimes</span></li>
<li><span class="key-term">Brand Image</span>: Misinformation, <span class="highlight-word">competitive references</span></li>
<li><span class="key-term">Data Privacy</span>: PII leakage, <span class="highlight-word">credentials exposure</span></li>
<li><span class="key-term">Unauthorized Access</span>: System access, <span class="highlight-word">command execution</span></li>
</ul>
</div>
<div class="card container-rotate-in" style="animation-delay: 0.3s;">
<h2>Impact Areas</h2>
<ul class="enhanced-list">
<li><span class="highlight-word">User safety</span> and protection</li>
<li><span class="highlight-word">Organizational reputation</span> and trust</li>
<li><span class="highlight-word">Legal compliance</span> and regulations</li>
<li><span class="highlight-word">Data security</span> and privacy</li>
<li><span class="highlight-word">System integrity</span> and reliability</li>
</ul>
</div>
</div>