The Ateve TeamJun 4, 2026

Ateve Latency Eval Methodology: How We Measure Search Speed

How we evaluateSearch Speed

Why Latency Matters for AI Search

Humans barely notice search latency under 500ms — but AI agents are different.

A deep-research agent can make dozens of search calls in a single task. An extra 200 ms on each call can add up to several or even more than ten seconds of waiting. In voice and real-time chat scenarios, a pause of more than one second can disrupt the conversational experience.

At Ateve, we treat latency as a metric that must be measured with a unified, reproducible method — not a slogan that “we're fast”. This article explains how we measure latency, why we measure it this way, and where the boundaries of every figure are.

Four Design Principles for Latency Evals

The biggest risk in comparing latency across providers is not measuring it inaccurately, but measuring it unfairly or in a way that cannot be reproduced. We follow four principles:

Prevent Caching Effects: Without cache prevention, the results measure “cache latency” rather than real retrieval latency, making the numbers artificially low.
Ensure Reproducibility: The dataset, load parameters, environment, and SDK versions are all fixed and documented, so third parties can rerun the tests under the same conditions.
Use a consistent standard: All four providers are tested in the same environment, during the same time window, and under the same load, using identical metric definitions.
State the boundaries honestly: State clearly what environment each measurement represents (local public internet / datacenter), and never pass off one environment's numbers as another's.

Methodology in Detail

Dataset and Cache Prevention

Dataset: 4,326 SimpleQA questions, from which we sample N unique queries per the agreed protocol (600 for this run).
Cache Prevention: We randomize numbers and years in queries, so that search-engine cache hits don't skew the latency.
This is the most critical step in latency evaluation. Exa uses the same approach— appending random words or randomizing numbers to avoid caching—and we follow that practice.

Connection Warm-up

Before the real load test, we warm up the connections with a small number of queries (connection pool, DNS, TLS handshake):

This run: 20 queries at 5 QPS, not counted in the official results.
Purpose: To exclude cold-start costs from establishing the initial connection, so the reported results reflect “steady-state latency” rather than “first-request overhead”.

Load Generation (Fixed Rate)

Mode: Open-loop fixed-rate scheduling — using a monotonic clock, one request is created at each fixed interval (e.g. every 100ms), for a fixed duration (e.g. 60 seconds).
Key point: This is not the closed-loop, saturation-style “send one, wait, send the next.” Requests are fired strictly on a schedule, so the measured latency reflects “real throughput capacity at the target QPS.”
Pinned parameters: Top-10 results, 0 retries, identical across all four providers.

Metric Definitions (the Core of Latency Evals)

We break “latency” into several separately attributable metrics, rather than reporting one vague “response time”:

Metrics	Definition	Note
E2E latency	Client-side end-to-end: from request initiation to fully parsed response	The latency users actually feel
searchTime	The search time returned by the server	The search engine's own compute time
Non-searchTime	E2E − searchTime	Path costs: network, DNS, TLS, gateway, SDK parsing, etc.
queue delay	Scheduled send time vs. actual send time	Detects backlog in local scheduling
P50 / P90 / P95 / P99	Latency percentiles	The official statistical convention — avoids the mean hiding the tail
Success rate / 429 / timeout	Request outcomes	Judges whether load exceeds capacity
Top-10 shortfall	Fraction of responses not returning a full Top-10	A latency-comparability check (unequal workloads would contaminate the latency comparison)

Why split E2E and searchTime? Because “client-side end-to-end” includes the local network path, while “server-side searchTime” is closer to the search engine's own speed. Reporting both separately is the only way to be honest (E2E is what users actually experience) without unfairly blaming the engine (searchTime is the part the product actually controls).

Result Aggregation and Evidence Layering

Aggregation: Summarized by P50/P90/P95/P99 + actual QPS + success rate + shortfall.
Evidence layering: Each layer of evidence supports only its own claim: code (a pinned commit), load (the target QPS was actually reached), API behavior (success rate), workload (Top-10 completeness), environment (local network or AWS), and quality (whether answer generation is included). No layer is allowed to stand in for another.

Results

Metrics	P50	P90	P95
Ateve	102.0 ms	128 ms	155.2 ms
Exa	528.7 ms	678.6 ms	709.7 ms
Tavily	1,745.31 ms	3,251.32 ms	3,869.52 ms
Brave	1,323.17 ms	2,255.18 ms	2,661.27 ms

Environment and Boundaries

A credible latency report has to say clearly where this measurement is taken:

Local public-internet client: Reflects what real users experience over the public internet, but includes uncontrollable path costs like local ISP, DNS, TLS, and gateway.
Datacenter, same environment: A fairer comparison closer to “the search engine's own speed,” requiring all four providers to be rerun in the same region (e.g. AWS us-west-1) at the same time window.

We state this explicitly: local results are not equivalent to data-center results. They cannot be converted from one to the other through simple subtraction. Official comparison results are based on reruns in the same environment, during the same time window, and under the same load for all four providers.

Continuous Evaluation

This article presents the methodological framework for our latency evaluation. Search speed changes continuously with index updates, network paths, and load levels, so we will continue measuring and publishing results across different environments and concurrency levels.

We believe that publishing how a metric was measured alongside the number itself is more trustworthy than reporting a P50 figure in isolation.

Appendix: Core Standards at a Glance

Dataset: SimpleQA, 600 sampled queries (extensible)
Cache Prevention: randomized numbers/years
Load: open-loop fixed rate, 10 QPS, 60 seconds, 0 retry
Warm-up: 20 queries / 5 QPS, not counted in results
Metrics: E2E, searchTime, non-searchTime, queue delay, P50/P90/P95/P99, success rate, Top-10 shortfall
Environment: local public internet (reference version) → four-way same-environment rerun (accurate version)

Why Latency Matters for AI Search

Humans barely notice search latency under 500ms — but AI agents are different.

Four Design Principles for Latency Evals

The biggest risk in comparing latency across providers is not measuring it inaccurately, but measuring it unfairly or in a way that cannot be reproduced. We follow four principles:

Prevent Caching Effects: Without cache prevention, the results measure “cache latency” rather than real retrieval latency, making the numbers artificially low.
Ensure Reproducibility: The dataset, load parameters, environment, and SDK versions are all fixed and documented, so third parties can rerun the tests under the same conditions.
Use a consistent standard: All four providers are tested in the same environment, during the same time window, and under the same load, using identical metric definitions.
State the boundaries honestly: State clearly what environment each measurement represents (local public internet / datacenter), and never pass off one environment's numbers as another's.

Methodology in Detail

Dataset and Cache Prevention

Dataset: 4,326 SimpleQA questions, from which we sample N unique queries per the agreed protocol (600 for this run).
Cache Prevention: We randomize numbers and years in queries, so that search-engine cache hits don't skew the latency.
This is the most critical step in latency evaluation. Exa uses the same approach— appending random words or randomizing numbers to avoid caching—and we follow that practice.

Connection Warm-up

Before the real load test, we warm up the connections with a small number of queries (connection pool, DNS, TLS handshake):

This run: 20 queries at 5 QPS, not counted in the official results.
Purpose: To exclude cold-start costs from establishing the initial connection, so the reported results reflect “steady-state latency” rather than “first-request overhead”.

Load Generation (Fixed Rate)

Mode: Open-loop fixed-rate scheduling — using a monotonic clock, one request is created at each fixed interval (e.g. every 100ms), for a fixed duration (e.g. 60 seconds).
Key point: This is not the closed-loop, saturation-style “send one, wait, send the next.” Requests are fired strictly on a schedule, so the measured latency reflects “real throughput capacity at the target QPS.”
Pinned parameters: Top-10 results, 0 retries, identical across all four providers.

Metric Definitions (the Core of Latency Evals)

We break “latency” into several separately attributable metrics, rather than reporting one vague “response time”:

Metrics	Definition	Note
E2E latency	Client-side end-to-end: from request initiation to fully parsed response	The latency users actually feel
searchTime	The search time returned by the server	The search engine's own compute time
Non-searchTime	E2E − searchTime	Path costs: network, DNS, TLS, gateway, SDK parsing, etc.
queue delay	Scheduled send time vs. actual send time	Detects backlog in local scheduling
P50 / P90 / P95 / P99	Latency percentiles	The official statistical convention — avoids the mean hiding the tail
Success rate / 429 / timeout	Request outcomes	Judges whether load exceeds capacity
Top-10 shortfall	Fraction of responses not returning a full Top-10	A latency-comparability check (unequal workloads would contaminate the latency comparison)

Result Aggregation and Evidence Layering

Aggregation: Summarized by P50/P90/P95/P99 + actual QPS + success rate + shortfall.
Evidence layering: Each layer of evidence supports only its own claim: code (a pinned commit), load (the target QPS was actually reached), API behavior (success rate), workload (Top-10 completeness), environment (local network or AWS), and quality (whether answer generation is included). No layer is allowed to stand in for another.

Metrics

P50

P90

P95

Ateve

102.0 ms

128 ms

155.2 ms

Exa

528.7 ms

678.6 ms

709.7 ms

Tavily

1,745.31 ms

3,251.32 ms

3,869.52 ms

Brave

1,323.17 ms

2,255.18 ms

2,661.27 ms

Environment and Boundaries

A credible latency report has to say clearly where this measurement is taken:

Local public-internet client: Reflects what real users experience over the public internet, but includes uncontrollable path costs like local ISP, DNS, TLS, and gateway.
Datacenter, same environment: A fairer comparison closer to “the search engine's own speed,” requiring all four providers to be rerun in the same region (e.g. AWS us-west-1) at the same time window.

Continuous Evaluation

We believe that publishing how a metric was measured alongside the number itself is more trustworthy than reporting a P50 figure in isolation.

Appendix: Core Standards at a Glance

Dataset: SimpleQA, 600 sampled queries (extensible)

Cache Prevention: randomized numbers/years

Load: open-loop fixed rate, 10 QPS, 60 seconds, 0 retry

Warm-up: 20 queries / 5 QPS, not counted in results

Metrics: E2E, searchTime, non-searchTime, queue delay, P50/P90/P95/P99, success rate, Top-10 shortfall

Environment: local public internet (reference version) → four-way same-environment rerun (accurate version)