Inspect AI

Inspect AI is an open-source framework developed by the UK AI Safety Institute for evaluating large language models and AI agents. It helps developers and researchers test model capabilities, safety, reasoning, and tool-use through structured evaluation workflows.

The framework provides datasets, solvers, scorers, benchmarks, and sandboxing tools for running repeatable AI evaluations. It can be used to test both individual models and more complex agentic systems across different model providers.

How Inspect AI Works

• Create Datasets: Add testing samples containing inputs and expected targets or answers.
• Define Solvers: Set the logic for how models approach tasks, including prompts and multi-step tool interactions.
• Apply Scorers: Use rule-based or model-based scoring methods to evaluate model outputs.
• Build Tasks: Combine datasets, solvers, and scorers into complete evaluation tasks.
• Run Evaluations: Execute tasks across models and review detailed logs, tool calls, and evaluation results.

How We Rated Inspect AI

We considered Inspect AI's LLM evaluation capabilities, agent testing, benchmarks, datasets, scorers, sandboxing, model-provider support, customization, and developer tooling.