The Ateve TeamSep 2, 2026

Ateve Eval Methodology: How We Evaluate Search Quality

How we evaluateAteve Search Quality

First, Define "the Best"

When building a search engine, the question you get asked most is: "Are you actually any good? And how do you prove it?"

It is also one of the hardest questions to answer. That is because search quality can't be captured by a single figure: it may refer to "whether the returned documents are relevant", "whether those results enable users to answer their questions correctly", or "whether the system can achieve this quickly and cost-effectively".

At Ateve, we hold to one principle when evaluating search quality:

An eval must answer the questions downstream users actually care about — not just the questions we want to win.

This post will make two things clear: how we test (the methodology), and why we test this way (the trade-offs behind it). All figures come from eval reports we have already published and made reproducible, and we do not shy away from our shortcomings.

Evaluation Results at a Glance

On a public dataset, we benchmarked Ateve head-to-head against three leading search APIs

Metrics	Ateve	Exa	Brave	Tavily
Accuracy	95.19%	94.38%	87.31%	84.40%
Correct answers	4,118	4,083	3,777	3,651
Wrong answers	208	243	549	675

Dataset: SimpleQA (OpenAI), 4,326 fact-based short-answer questions
Conclusion: Ateve ranks #1, leading Exa by 0.81 percentage points (35 questions), with a wider lead over Brave and Tavily

Ateve achieved the highest claim-accuracy score, leading Exa by 0.81 percentage points (35 questions), with a wider lead over Brave and Tavily. But a single accuracy number is not equal to an eval methodology. In the sections below, we'll break down where that number came from, what makes it credible, and where its limits are.

Why SimpleQA — and Its Limitations

In choosing a benchmark for comparing providers, we gave careful consideration to the options. We selected SimpleQA for clear reasons—and we are equally clear about its limitations.

Why we chose it?

Recognized and reproducible: published by OpenAI, all 4,326 questions come with gold answers
Official grading template: we use the GRADER_TEMPLATE from OpenAI's simple-evals repository. The evaluation criteria are publicly available and do not rely on our own self-defined, “lenient” standards.
Designed to elicit hallucinations: The questions are deliberately crafted as factual queries that LLMs are likely to hallucinate on without genuine retrieval support, effectively distinguishing genuine search capability from “background knowledge.”

Its limitations (which we openly acknowledge):

SimpleQA only tests fact-based short-answer questions. It doesn't cover long-form research, multi-turn search, complex reasoning, multimodality, or temporal freshness.
Leading on a dataset is not equivalent to leading across the full real-world search experience.

This is also why we include FreshQA (temporal freshness) in our evals — a single benchmark answers only one question, never all of them.

Eval Design: Making the Comparison Truly Fair

The biggest risk in head-to-head comparisons is not inaccurate measurement, but unfair measurement. If Provider A is paired with a stronger generation model while Provider B is graded more leniently, the numbers become distorted.

Drawing on industry best practices, we established three non-negotiable principles.

One Unified Evaluation Pipeline

All four providers go through the exact same pipeline — no special treatment for anyone:

Evaluation Flow

1SimpleQA Query
2
3Four providers each return their Top-10 search results
4
5Results are normalized, merged, and filtered
6
7A standardized answer-generation pipeline produces a short answer
8
9An LLM judge compares the predicted answer against the gold answer
10
11Output CORRECT / INCORRECT / NOT_ATTEMPTED
12
13Accuracy per provider

The "Single Variable" Principle

This is the single most important statement in the whole eval:

The answer-generation flow, judge model, grading prompt, and retrieval budget are identical across all four providers. The only variable is the search results each provider returns.

This means: if there's any difference in final accuracy, that difference can only come from retrieval quality itself — not from "who used a better generation model" or "who got graded more leniently."

(We apply the same fairness standard here: if a competitor has publicly self-reported scores, we compare against the higher of "our own measurement" and "their self-report.")

Grading Standard: The Official Three-Way Classification

We use the official OpenAI SimpleQA judge prompt directly to classify each predicted answer into one of three categories:

Verdict	Rule
CORRECT	The answer fully contains the key information of the gold answer, with no contradictory facts
INCORRECT	The answer contains facts that contradict the gold answer, or gives wrong/incomplete/extra candidates
NOT_ATTEMPTED	No valid answer is given (refusal, “I don’t know”, incomplete information)

A Strict Standard: An Error Is An Error

We use a strict binary standard — anything other than CORRECT, including INCORRECT, NOT_ATTEMPTED, and missing judge verdicts, all counts as wrong.

This standard leaves no backdoor for ourselves: a refusal isn't "not tested," it's "answered wrong." It makes the accuracy number stand up to scrutiny, and it removes any way for us to inflate our score with "selective answering."

Results and Interpretation

Overall Accuracy

Ateve ranks #1 with 95.19%, answering 35 more questions correctly than Exa, 341 more than Brave, and 467 more than Tavily.

The Composition of Errors Is More Informative Than the Total

Looking only at the total hides an important fact:

Provider	Total errors	Incorrect	Not_attempted
Ateve	208	94	114
Exa	243	196	47
Brave	549	270	220
Tavily	675	288	378

How to read this: Ateve has 67 more NOT_ATTEMPTED (114) than Exa (47), and 102 fewer INCORRECT (94 vs. Exa's 196). Under this report's strict standard, both count as errors, so Ateve ends up with a net lead of 35 questions over Exa.

Question-Level Differences

Comparison	Only Ateve correct	Only rival correct	Net wins
Ateve vs Exa	195	160	+35
Ateve vs Tavily	590	123	+467
Ateve vs Brave	470	129	+341

Of the 82 questions answered correctly by only one provider, Ateve led the field with 46—showing that its advantage comes not only from consistency, but also from differentiated retrieval and answer-generation capabilities.

Limitations and How We Address Them

A credible evaluation has to state honestly what it cannot measure. We openly acknowledge the following limitations:

Dataset limitations: SimpleQA consists of static factual questions and does not reflect real-world scenarios involving long-form content, multi-turn interactions, time-sensitive information, or multimodal inputs.
LLM judge limitations: Judgments rely on an LLM and have not undergone exhaustive human review. Final accuracy is influenced by retrieval, answer generation, and the judge itself; it should not be equated with the document-retrieval quality of the search engine alone.
Scope of this article: this article focuses on the accuracy dimension; latency, throughput, and token efficiency are covered elsewhere.
Temporal limitations: the providers weren't all run on the exact same date, and results only represent the current data and configuration; The results may change after index updates.

Our stance: these limitations don't make the numbers unreliable — but they do make them incomplete. We state them plainly, so readers understand the boundaries of every number.

Continuous Evaluation

This document presents only a portion of our evaluation work. Search quality is not limited to accuracy alone—latency, cost, timeliness, and complex reasoning are all areas that we continuously measure and disclose.

We believe that consistently publishing our evaluation methods and results — and enabling third parties to reproduce them—is more credible than any single number. This is also what we will continue to do moving forward.

Appendix: Core Standards at a Glance

Dataset: SimpleQA, 4,326 questions
Products compared: Ateve / Exa / Brave / Tavily
Denominator: all 4,326 questions
Error standard: anything that isn't CORRECT counts as wrong (including NOT_ATTEMPTED and missing judge)
Fairness: the only variable is the search results each provider returns; For competitor self-reported scores, the higher value is used.
Grading: OpenAI's official SimpleQA GRADER_TEMPLATE