evals.do
Continuous AI Evaluation on Every Change
Test language models automatically against benchmark datasets. Compare models side-by-side and detect prompt regressions before production.
import { Evals } from 'evals.do' // Block deployments on quality regressionsEvals.every.deploy(async $ => { const results = await $.Evals.runSuite({ models: ['gpt-5', 'claude-sonnet-4.5', 'gemini-pro'], dataset: 'customer-support-qa' }) if (results.accuracy < 0.85 || results.latency > 2000) { throw new Error(`Quality regression: ${results.summary}`) }})
One of 48 primitives
evals.do sits in Insight: measure, test and trace. Every .do shares one grammar, Business-as-Code, and one SDK.
Data
Records, vocabulary and search
Insight
Measure, test and trace
- Analytics
- Experiments
- Evals
- Benchmarks
- Trace
- Models
Frequently asked questions
100+ built-in metrics including accuracy (exact match, semantic similarity, BLEU, ROUGE), latency (p50/p95/p99 response times), cost (tokens/request, $/1M tokens), safety (toxicity, PII detection, harmful content), factuality (hallucination detection, citation accuracy), consistency (temperature=0 repeatability), and custom metrics via TypeScript functions. Track metrics over time to detect regressions and optimize tradeoffs.
Yes. Shadow mode runs new models alongside production models, comparing outputs without affecting users. Evaluate GPT-4 vs Claude vs Gemini on real production traffic, measure quality/speed/cost differences with statistical significance, and switch models with confidence. Automatic rollback if quality drops below thresholds.
Import existing datasets (CSV, JSON, Hugging Face), generate synthetic test cases using AI, capture production examples automatically, or write test cases manually. Datasets support input/output pairs, multi-turn conversations, and multi-modal content (text, images, audio). Version control tracks dataset changes over time with Git integration.
Automatic prompt optimization tests hundreds of variations (temperature, system prompts, few-shot examples, formatting) to find the best performer for your metrics. Genetic algorithms explore the prompt space intelligently. A/B testing with statistical significance ensures changes actually improve quality. Built-in prompt versioning with rollback for every production prompt.
Parallel execution across 300+ edge locations evaluates 1000s of test cases in <30 seconds. Caching skips unchanged prompts/models for instant reruns. Cost: first 10K evaluations/month free, then $0.001 per evaluation. Model API costs (OpenAI, Anthropic, Google) are pass-through at your negotiated rates. Save 40-70% by identifying cheaper models with equivalent quality.