Glossary · AI & Data

AI Benchmark

An AI benchmark is a standardized test set and scoring method used to compare models on a task, such as reasoning, coding, or question answering.

Updated October 6, 2026

A benchmark is a fixed collection of tasks with a defined scoring rule, used to measure and compare AI models. Examples cover multiple-choice knowledge tests, code-generation problems, math word problems, and image recognition.

Why it matters

Benchmarks give a shared yardstick for progress and help shortlist models. However, scores may not reflect your own use case, and published results can be inflated if test items leaked into training data. Treat a benchmark as one signal among several, not a verdict.

Example

A coding benchmark might run model-written functions against hidden unit tests:

164 problems, 1 attempt each
solved: 123 -> pass@1 = 75%

The figure is only meaningful alongside the exact prompt format, settings, and benchmark version used.

Common mistakes

  • Choosing a model from a leaderboard rank without testing on your own examples.
  • Comparing scores produced with different prompts or settings.
  • Ignoring cost, speed, and reliability, which benchmarks rarely capture.

Get notified when this term updates

Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.

Keep reading

Glossary · AI & DataA/B TestingA/B testing is a controlled experiment that shows two versions of a page or feature to different users to learn which performs better on a chosen metric.Glossary · AI & DataAGI (Artificial General Intelligence)AGI is a hypothetical form of artificial intelligence able to learn and perform any intellectual task a human can, rather than excelling only at narrow tasks.Glossary · AI & DataAI AgentAn AI agent is a system built around a language model that plans steps, calls tools or APIs, and acts toward a goal with limited human direction.

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.