LLM Scraped AI Answers vs API Results: Why They Don't Match

Two AI visibility tools can query the same brand with the same prompts and still hand you completely different reports because they collect data in fundamentally different ways, and they disagree far more than most teams realize.

Ask an AI visibility vendor how they get their data, and the answer usually falls into one of two camps: they call the model's official API, or they scrape the actual consumer product, the ChatGPT window, the Perplexity app, Google's AI Overview, the way a real user would see it. It sounds like a technical footnote. It isn't. 

Multiple independent studies now show these two methods disagree on which brands appear, how many, in what order, and which sources get cited, often to a degree that makes the two datasets barely comparable at all.

If your team is tracking AI search visibility, or evaluating a vendor that does, this distinction determines whether the dashboard in front of you reflects what your buyers actually see, or an approximation that quietly diverges from it.

Two Different Ways to Collect the Same "Answer"

Both methods query something real. The disagreement isn't about accuracy in a vacuum; it's about which one actually represents what a buyer researching your brand is looking at.

What a 13,779-Answer Study Found

A study running 1,000 prompts across five AI products, ChatGPT, Perplexity, Gemini, AI Overviews, and AI Mode, through both collection methods, collecting 13,779 total answers, found the two methods disagreeing on nearly every measurable dimension.

15.5–23.8% brand-list overlap between API and scraped answers, raw.

21.3–31.6% overlap after canonicalizing brand names.

1.85x more brands named via ChatGPT's API vs. its consumer interface.

2.05x longer answers from ChatGPT's API vs. the product users see.

4.8–19.7% page-level citation overlap on ChatGPT and Perplexity.

Brand Overlap is The Headline Number

Even after cleaning up brand-name variants (treating "SurferSEO," "Surfer SEO," and "Surferseo" as one entity), the overlap between which brands the API names and which the consumer product names topped out around 32%, and ran as low as 21% depending on the product. Gemini showed the highest agreement, and even there, only about three brands in ten repeated across both methods.

APIs Consistently Name More Brands, And Write Longer Answers

Across every product tested, the API version listed more brands than the version real users saw; ChatGPT's API named nearly twice as many. The same pattern held for answer length: ChatGPT's API responses ran roughly twice as long as what the consumer interface actually showed, meaning the shorter, real answer had already dropped content the longer API version kept, content that may include your brand's mention.

Citation Counts and Which Pages Get Cited Diverge Sharply

The pattern for how many sources get cited wasn't consistent in one direction: ChatGPT's consumer interface cited roughly four times more sources than its API (12.1 vs. 3.1), while Perplexity's API cited nearly twice as many as its app. But regardless of direction, the specific pages cited barely matched: page-level overlap ran as low as 4.8% on ChatGPT and topped out at 19.7% on Perplexity. For any strategy built around identifying which pages an AI model cites and pursuing outreach or optimization around them, that low an overlap means the API could be pointing a team at pages its actual buyers never see.

Even Whether a Web Search Happens Differs

Scraped consumer interfaces triggered a live web search on nearly every prompt, 88% of the time on ChatGPT, 100% on the other four products. API calls searched less consistently, from 83% on ChatGPT up to 100% on Perplexity. That's a difference in the underlying retrieval behavior itself, not just in how the result gets presented.

This Isn't One Vendor's Isolated Finding

Independent research points in the same direction. A separate study running 900 unbranded questions through ChatGPT, Gemini, and Perplexity via both channels found brand-set overlap between UI and API answers at just 6–12%, with the first-mentioned brand, the one effectively winning the recommendation matching in only 9–16% of answer pairs. A neutral methodology review of the AI-visibility-tooling category independently concludes that API-based collection often queries a different model than consumers actually use, without the web search, personalization, or interface-specific behavior real users get, and that any credible vendor should disclose exactly which method it uses, per channel, in writing.

A single sentence in a system prompt can turn an LLM's response by 180 degrees, and consumer products run extensive, undisclosed system instructions that API calls simply don't include. Low alignment between the two channels isn't a measurement bug; it's the expected result of comparing two different systems that happen to share an underlying model.

The Case for API-based Tracking, Fairly Stated

Scraping isn't free of trade-offs, and the API side of this debate has a real argument, not just a cost advantage. API responses are fully reproducible: the same prompt, the same parameters, produce a result any team can independently verify, without the variability of personalization and browsing history introduced into a scraped answer. For teams that specifically want a neutral, context-free baseline, what a model says with no user history attached, that reproducibility is a genuine strength, not just a shortcut.

There's also a real critique of how some scraping-based tools operate: testing through an incognito browser session doesn't necessarily replicate the real user experience either, since some products behave differently in incognito mode (for instance, running with web search disabled) than they do for a logged-in user with full product functionality, a gap that undermines the "true to reality" claim scraping vendors often make just as much as API gaps undermine API-based ones.

The honest conclusion is that neither method is a perfect mirror of "what buyers see" in every case; a scraped session without the right login state or region settings has its own blind spots, and an API call has different ones. What both sides of this debate agree on is the underlying point: the method matters enormously, and a vendor that can't explain exactly which one it uses, on which channel, is not giving you a number you can trust either way.

What This Means for How You Measure AI Visibility

• Treat "share of voice" or "citation" numbers as method-dependent, not absolute; the same brand can look dominant in one dataset and nearly invisible in another, depending purely on collection method.
• If a page-level outreach or optimization strategy depends on knowing exactly which pages get cited, API-sourced citation lists carry a real risk of pointing at pages your actual audience never sees.
• A single sample per prompt isn't enough either way. Since LLM answers are non-deterministic, the same prompt asked twice through the identical method won't always return the same result, so some disagreement is inherent to the systems, not just the method.
• Sampling frequency and prompt coverage matter as much as collection method; a tool running one pass a week across a handful of prompts is a poll, not a measurement, regardless of whether it scrapes or calls an API.

Questions Worth Asking Any AI Visibility Vendor

• Which collection method do you use, per channel: scraping, API, or a mix?
• If API-based, which specific model version, and how does it compare to what the consumer product actually runs?
• How many samples do you take per prompt, and how do you handle the non-determinism of repeated runs?
• Do your reported metrics come with denominators, number of runs, prompts, and the time window, or just a bare score?
• If historical numbers change, is that change disclosed and annotated, or silently rewritten?

A vendor that answers these plainly and specifically is giving you a measurement. A vendor that can't, or won't, is giving you a demo.

Frequently Asked Questions

Is Scraping Always More Accurate than Using an API?

Not automatically. Scraping captures more of what a real, logged-in user actually sees, including personalization, safety filtering, and the product's own answer-shortening, which API calls generally strip out. But a poorly configured scraper (wrong region, no login state, incognito mode with search disabled) can miss real-user behavior just as much as an API call does. The method matters less than whether it's disclosed and configured to match actual user conditions.

Why do APIs Name More Brands Than the Consumer Product Does?

Consumer products apply answer-shortening, safety filtering, and interface-specific logic on top of the underlying model before displaying a response to a user, trimming content, including brand mentions, that a raw API call doesn't filter out. The longer, more brand-dense API response isn't necessarily "more complete" from a user's perspective; it's showing content the real product decided not to surface.

Can I Compare AI Visibility Numbers Between Two Different Tracking Tools?

Only if both disclose the same collection method, model version, and sampling frequency. Given overlap between API and scraped results can run as low as 6–24% depending on the platform, two tools using different methods can report contradictory visibility scores for the exact same brand and prompt set without either being technically wrong, they're measuring different things.

Does Google AI Overviews Have an API to Test Against?

Not a dedicated one. Because Google documents AI Overviews as running on Gemini 3 and doesn't sell that exact model via API, API-based measurement of AI Overviews typically borrows the closest available method (often the AI Mode API) as an approximation, which adds an extra layer of imprecision beyond the API-vs-UI gap seen in products with a true matching API.

How Many Prompts do I Need to Track to get a Reliable Picture?

Coverage of topics and intents matters more than raw prompt count; a well-chosen set of around 20 intentional prompts spanning your key topics and both short comparison-style and longer exploratory questions can outperform a much larger, less deliberate list. What matters most is consistent, repeated sampling over time, not a single large one-off run.

Related Searches:

• Content Brief Template: How to Plan Better Content
• Search Everywhere Optimization: The Complete Guide
• AI Search KPIs: How to Measure Impact When There's No Click