Files
hips/HIPs/hip-0210-safety-evaluation-framework.md
zeekay ad5a010f79 Three words for what a proposal is, and a generator that refuses
The same fact lived in nine hand-kept places and they disagreed about 58
proposals. This makes the file the only authority and everything else a
projection of it.

ONE VOCABULARY, in vocabulary.json, read by the lint, the generator and the
doc site. Three words, each answering "can I rely on this?" and nothing else:

    Draft    nothing implements it yet, or its own text says the work is ahead
    Final    the thing it specifies exists in the code
    Living   governance and process: amended in place, never finished

Eight spellings folded in. `Active` was the one doing real damage: on four Meta
and Process HIPs it meant living, on 48 others it meant shipped, and one word
doing two jobs is why nothing agreed. `Review` and `Last Call` described a
queue rather than a document, and a proposal under review is still one you
cannot build against.

STATUS IS EARNED, not declared. Every Standards Track proposal was checked
against the estate: 235 have their implementation, and eight do not and say so.
HIP-0095 carries a twelve-week plan ending at "mainnet launch" and has no code.
HIP-0525 specifies an all-machine meeting while apps/meet refuses a machine a
seat. HIP-0078 names a prover at a "planned location" that does not exist.
HIP-0114 is the cog.* procedure set, and that procedure set returns nothing.
HIP-0003, HIP-0102, HIP-0305 and HIP-0901 likewise: looked, found nothing.

SIXTEEN TOMBSTONES DELETED rather than statused. A dead document kept alive in
the index is one concept wearing two names. Each successor exists and carries
the text, and the 90 references that pointed at a tombstone now point at it --
whole markdown links at once, so link text cannot outlive its target. Three of
the sixteen were pure compatibility: HIP-0028 specified a migration off a
Bitnami chart, HIP-0029 dual-engine MySQL, HIP-0030 a schema-deprecation
policy. We ship no backwards compatibility, so we do not specify it.

Nine links were already broken and had been since the corpus was renumbered:
short-form ./hip-5.md paths that never existed. Two named the wrong proposal --
one cited "HIP-1: Hanzo Multimodal Models" where multimodal is HIP-0003.

THE GENERATOR REFUSES more often than it writes, because a short table looks
exactly as authoritative as a complete one. Rows must equal files; a shrinking
index must say by how much; a section whose input is missing is an error, not
an omission; nothing is written until all of it validates. Its self-test breaks
the corpus eleven ways and requires each to refuse AND leave README byte-
identical -- a guard that reports after half-writing has protected nothing.
The lint's second implementation of the same comparison is gone.

The site rendered 52 proposals with a fallback grey badge and counted them
toward none of its three stat tiles, because docs/lib/source.ts declared a
status union by hand: it admitted a `Stagnant` no file has ever carried and
omitted the one 52 files did. Five components each kept their own copy of the
colours. All six now read vocabulary.json.
2026-08-28 18:05:41 -07:00

9.7 KiB
Raw Permalink Blame History

hip, title, tags, description, author, discussions-to, status, type, created, requires
hip title tags description author discussions-to status type created requires
210 Safety Evaluation Framework
ai-ethics
safety
evaluation
testing
Framework for evaluating AI system safety through comprehensive testing and red teaming. Hanzo AI Team (@hanzoai) https://github.com/hanzoai/hips/discussions Living Meta 2025-12-17 HIP-0200, HIP-0201

HIP-0210: Safety Evaluation Framework

Abstract

This HIP establishes the safety evaluation framework for Hanzo AI systems. It defines testing methodologies, evaluation criteria, red teaming processes, and safety benchmarks required before model deployment.

Safety Evaluation Overview

Objectives

  1. Identify harms: Discover potential harmful behaviors
  2. Measure severity: Quantify harm potential
  3. Verify mitigations: Confirm safety measures work
  4. Track progress: Monitor safety over time
  5. Enable comparison: Benchmark against standards

Evaluation Types

Type Purpose When
Automated testing Scalable coverage Continuous
Human evaluation Nuanced assessment Milestone
Red teaming Adversarial probing Pre-deployment
External audit Independent validation Annual

Harm Taxonomy

Harm Categories

Content Harms

Category Examples Severity
Violence Instructions for violence, glorification Critical
CSAM Any CSAM generation or facilitation Critical
Hate speech Slurs, dehumanization, discrimination High
Self-harm Suicide methods, eating disorder promotion High
Sexual content Non-consensual, inappropriate contexts High
Harassment Targeted abuse, doxxing High

Deception Harms

Category Examples Severity
Misinformation False claims presented as fact High
Manipulation Psychological manipulation High
Fraud facilitation Scam scripts, phishing High
Impersonation False identity claims Medium

Security Harms

Category Examples Severity
Cyberweapons Malware, exploits Critical
CBRN Chemical, biological, nuclear info Critical
Weapons Weapons manufacturing Critical
Privacy violations PII exposure, surveillance High

Societal Harms

Category Examples Severity
Election interference Voter suppression, disinformation Critical
Radicalization Extremist content High
Economic harm Market manipulation advice Medium

Severity Levels

Level Definition Response
Critical Immediate physical harm or illegal Block deployment
High Significant harm potential Remediation required
Medium Moderate harm, context-dependent Mitigation recommended
Low Minor issues, edge cases Monitor and improve

Automated Evaluation

Safety Benchmarks

Standard Benchmarks

Benchmark Coverage Metrics
ToxiGen Toxicity detection F1, precision, recall
RealToxicityPrompts Generation toxicity Toxicity rate
WinoBias Gender bias Accuracy parity
BBQ Social bias Disambiguation accuracy
TruthfulQA Truthfulness MC accuracy

Custom Benchmarks

Benchmark Coverage Metrics
Hanzo-Safety-1K Multi-harm coverage Refusal rate, harm rate
Jailbreak-Eval Prompt injection resistance Success rate
PII-Detection Privacy protection Detection F1

Automated Testing Pipeline

Input prompts (adversarial + benign)
    ↓
Model generation
    ↓
Classifier scoring (toxicity, harm type)
    ↓
Threshold checking
    ↓
Report generation

Thresholds

Metric Threshold Action if Failed
Toxicity rate <1% on benign prompts Block deployment
Jailbreak success <5% on known attacks Remediation
PII leakage 0% on test set Block deployment
Refusal accuracy >95% on harmful requests Remediation

Human Evaluation

Evaluator Requirements

Requirement Details
Training Safety evaluation training (HIP-200 overview)
Background checks For sensitive content evaluation
Psychological support Access to counseling
Rotation Limit exposure to harmful content

Evaluation Protocol

Rating Scale

Rating Definition
1 - Safe No safety concerns
2 - Minor Slight issues, acceptable
3 - Moderate Concerning but not severe
4 - High Significant safety concern
5 - Critical Immediate harm potential

Evaluation Criteria

Criterion Question
Harmfulness Could this output cause harm?
Helpfulness Does this appropriately assist the user?
Honesty Is this truthful and accurate?
Appropriateness Is this suitable for the context?

Sample Size

Evaluation Type Minimum Samples Evaluators
Pre-deployment 5,000 3 per sample
Periodic review 1,000 2 per sample
Incident response As needed 3 per sample

Inter-Rater Reliability

Metric Target
Krippendorff's α >0.8
Cohen's κ >0.7
Agreement rate >90% on Critical ratings

Red Teaming

Red Team Structure

Internal Red Team

Role Focus
Safety researchers Known attack patterns
Domain experts Domain-specific harms
Adversarial ML specialists Technical attacks

External Red Team

Partner Purpose
Security researchers Novel attack discovery
Domain experts Specialized knowledge
Academic partners Research collaboration

Red Team Methodology

Attack Categories

Category Techniques
Prompt injection Jailbreaks, role-play attacks
Context manipulation Multi-turn attacks, persona switching
Encoding attacks Base64, translation, cipher
Social engineering Persuasion, authority claims
Technical attacks Adversarial inputs, token manipulation

Red Team Process

1. Scoping (define attack surface)
    ↓
2. Reconnaissance (understand model behavior)
    ↓
3. Attack development (create test cases)
    ↓
4. Execution (run attacks)
    ↓
5. Documentation (record findings)
    ↓
6. Remediation (develop fixes)
    ↓
7. Verification (confirm fixes work)

Red Team Reporting

Section Contents
Executive summary Key findings, risk assessment
Methodology Approaches used, scope
Findings Detailed vulnerability list
Severity ratings Per finding
Recommendations Suggested mitigations
Appendix Test cases, evidence

Red Team Cadence

Trigger Red Team Activity
New model Full red team before deployment
Major update Focused red team on changes
Quarterly Routine assessment
Incident Investigation and expanded testing

Safety Metrics

Primary Metrics

Metric Definition Target
Harm rate % outputs rated as harmful <0.1%
Refusal appropriateness % correct refusals >98%
Over-refusal rate % incorrect refusals <5%
Jailbreak resistance % attacks blocked >95%

Derived Metrics

Metric Calculation
Safety score Weighted composite of primary metrics
Risk exposure Harm rate × severity × volume
Defense depth Layers of protection passed

Trend Monitoring

Track over time:

  • Safety metrics by model version
  • Attack success rates
  • Incident frequency
  • Time to remediation

Evaluation Governance

Evaluation Independence

  • Safety evaluation team independent from development
  • Separate reporting line to ESG Committee
  • Authority to block deployment

Review Process

Stage Review
Pre-training Safety objectives review
Post-training Initial safety evaluation
Pre-deployment Full safety review
Post-deployment Ongoing monitoring

Sign-Off Requirements

Risk Tier Sign-Off
Critical Board + external review
High ESG Committee
Medium Safety Lead
Low Team Lead

External Validation

Third-Party Audits

Frequency: Annual for high-risk models

Scope:

  • Methodology review
  • Independent testing
  • Process assessment
  • Recommendations

Academic Collaboration

  • Share evaluation methodologies
  • Participate in benchmark development
  • Publish safety research
  • Host safety workshops

Regulatory Alignment

Regulation Alignment
EU AI Act High-risk system requirements
NIST AI RMF MEASURE function
ISO/IEC 42001 Performance evaluation
  • HIP-200: Responsible AI Principles
  • HIP-201: Model Risk Management
  • HIP-220: Bias Detection & Mitigation
  • HIP-230: AI Transparency & Explainability
  • HIP-240: AI Incident Response

Changelog

Version Date Changes
1.0 2025-12-17 Initial draft

Copyright and related rights waived via CC0.