# Continuous AI Evaluation on Every Change

> Test language models automatically against benchmark datasets. Compare models side-by-side and detect prompt regressions before production.

Canonical: https://evals.do/

## In code

```ts
import { Evals } from 'evals.do'

// Block deployments on quality regressions
Evals.every.deploy(async $ => {
  const results = await $.Evals.runSuite({
    models: ['gpt-5', 'claude-sonnet-4.5', 'gemini-pro'],
    dataset: 'customer-support-qa'
  })
  if (results.accuracy < 0.85 || results.latency > 2000) {
    throw new Error(`Quality regression: ${results.summary}`)
  }
})
```

## Frequently asked questions

### What evaluation metrics does evals.do support?

100+ built-in metrics including accuracy (exact match, semantic similarity, BLEU, ROUGE), latency (p50/p95/p99 response times), cost (tokens/request, $/1M tokens), safety (toxicity, PII detection, harmful content), factuality (hallucination detection, citation accuracy), consistency (temperature=0 repeatability), and custom metrics via TypeScript functions. Track metrics over time to detect regressions and optimize tradeoffs.

### Can I evaluate models before switching in production?

Yes. Shadow mode runs new models alongside production models, comparing outputs without affecting users. Evaluate GPT-4 vs Claude vs Gemini on real production traffic, measure quality/speed/cost differences with statistical significance, and switch models with confidence. Automatic rollback if quality drops below thresholds.

### How do I create custom evaluation datasets?

Import existing datasets (CSV, JSON, Hugging Face), generate synthetic test cases using AI, capture production examples automatically, or write test cases manually. Datasets support input/output pairs, multi-turn conversations, and multi-modal content (text, images, audio). Version control tracks dataset changes over time with Git integration.

### What about comparing prompts and finding optimal variations?

Automatic prompt optimization tests hundreds of variations (temperature, system prompts, few-shot examples, formatting) to find the best performer for your metrics. Genetic algorithms explore the prompt space intelligently. A/B testing with statistical significance ensures changes actually improve quality. Built-in prompt versioning with rollback for every production prompt.

### How fast are evaluations and what's the cost?

Parallel execution across 300+ edge locations evaluates 1000s of test cases in <30 seconds. Caching skips unchanged prompts/models for instant reruns. Cost: first 10K evaluations/month free, then $0.001 per evaluation. Model API costs (OpenAI, Anthropic, Google) are pass-through at your negotiated rates. Save 40-70% by identifying cheaper models with equivalent quality.

## Part of Platform.do

evals.do is one of the primitives of Platform.do, in the Insight layer (measure, test and trace). Every .do shares one grammar, Business-as-Code, and one SDK.

- What is Business-as-Code? https://platform.do/business-as-code
- Platform.do: https://platform.do/
- Docs: https://docs.platform.do