// category · 10 tools · 7 open source
AI Red Teaming & Safety Tools
Open frameworks, red teaming kits, guardrails and benchmarks for measuring what frontier models can do and keeping deployed systems within safe bounds.
How AI safety testing works
Safety and evaluation tools answer two questions: how capable is a model, and how easily can it be pushed into harmful behavior. The tools here cover both, from benchmarks to red teaming kits to runtime guardrails.
- Capability benchmarks such as Humanity's Last Exam and the ARC-AGI tasks behind ARC Prize measure how far models have come on hard, expert or novel reasoning problems. Scores on these are often cited in claims about AGI, so check which version and date a score uses.
- Evaluation frameworks such as Inspect, from the UK AI Security Institute, let researchers build repeatable tests. Government institutes and groups like METR use frameworks like this before and after model releases.
- Red teaming tools such as garak, PyRIT and HarmBench probe models for jailbreaks, prompt injection and data leakage.
- Guardrails such as NeMo Guardrails, Guardrails AI, Llama Guard from Purple Llama, and Lakera check inputs and outputs in live apps.
Labs tie their own testing to published thresholds. Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework define capability levels that require stronger safeguards, and Anthropic says Claude Mythos 5.1's chemical and biological capabilities sit below its next risk tier, per its Fable 5.1 announcement.
Read Preparedness Framework vs Responsible Scaling Policy, has AGI been achieved, and the safety section of Claude Fable vs Opus vs Mythos for how these tests shape real releases.
Reading a safety claim
When a lab or startup says a model is safe, ask three questions. Who ran the test: the lab itself, a government institute, or an independent group such as METR or Apollo Research? What exactly was measured: refusals on a fixed list of harmful prompts, success at a realistic task, or resistance to a skilled red team? And when: a score on an earlier snapshot may not hold for the model you can use today. System cards usually answer all three, and the UK AI Security Institute publishes its own findings. A benchmark score with no date, version or method is a marketing number, not evidence.
ARC Prizearcprize.orgNonprofit competition built around the ARC-AGI benchmarks, which test fluid reasoning on puzzles that are easy for people and hard for AI.Safety & EvalsFree★ 746
Inspectinspect.aisi.org.ukEvaluation framework from the UK AI Security Institute for building rigorous, reproducible LLM and agent evaluations.Safety & EvalsOpen source★ 3.1k
Guardrails AIguardrailsai.comFramework and hub of validators that check LLM inputs and outputs for risks and enforce structured responses.Safety & EvalsOpen source★ 7.6k
garakgarak.aiLLM vulnerability scanner that probes models for jailbreaks, prompt injection, data leakage and other failure modes.Safety & EvalsOpen source★ 9.7k
NeMo Guardrailsgithub.comToolkit for adding programmable rails to LLM apps: topic control, jailbreak detection, fact checking and safe dialog flows.Safety & EvalsOpen source★ 7.4kPurple Llamagithub.comMeta's umbrella project for open trust and safety tools, including Llama Guard classifiers and CyberSecEval benchmarks.Safety & EvalsOpen source★ 4.4k
HarmBenchharmbench.orgStandardised evaluation framework for automated red teaming and measuring model refusal robustness.Safety & EvalsOpen source★ 1.1kPyRITgithub.comMicrosoft's Python Risk Identification Toolkit for automating red teaming of generative AI systems.Safety & EvalsOpen source★ 121
Humanity's Last Examlastexam.aiExpert written benchmark of very hard questions across dozens of subjects, built to measure progress at the frontier of knowledge.Safety & EvalsFreeHN 1
Lakeralakera.aiRuntime security for generative AI apps, detecting prompt injection, data leakage and harmful content in real time.Safety & EvalsSee siteHN 1// guides
Guides to AI red teaming & safety tools
UK AI Security Institute: What It Does in 2026
The UK AI Security Institute explained: who runs it, how it tests frontier models before release, what the 2026 Mythos 5 incident showed and how to use it.
AGI vs LLM: What's the Difference in 2026?
AGI vs LLM explained: an LLM is a kind of AI model that predicts text, while AGI is a goal for human level general ability. See where today's chatbots sit.
What Is AI Safety? Risks, Research and Who Does It
What is AI safety? A clear 2026 explainer: the three kinds of AI risk, how safety differs from alignment and security, the main research areas and who does it.
Mistral Large 4: Release, Price and Benchmarks
Mistral Large 4 explained: the 1 trillion parameter preview, the October 31 open weights date, current API prices, benchmarks and how it compares to GPT-6.
Has AGI Been Achieved? GPT-6 Astra and the Debate
Has AGI been achieved? What OpenAI really said about GPT-6 Astra, how AGI is defined, the evidence on each side, and what a formal declaration would change.