AI Search Tracking Is Getting Noisy: What AEO Teams Do

16 October 2026 · The Cresia team

Can you trust your AI search and rank tracking data right now? Not from a single tool. Search Engine Roundtable reports that Google seems to be feeding third-party tracking tools fake data and paywalls, which means any number scraped from results is an estimate you should check against data you own.

Key takeaways

Treat any scraped rank or AI-visibility number as an estimate until it agrees with data you own.
Search Engine Roundtable reports that Google seems to be feeding tracking tools fake data and paywalls, and has launched a new model in AI Mode, so old baselines are weaker than they look.
AI engines build answers from many sub-queries, so one keyword and one position is the wrong unit. Track topics and fixed prompt sets.
Move your primary evidence to first-party sources: server logs, analytics referrals, Search Console and the questions prospects ask on sales calls.
Keep your prompt panel fixed, run it on a schedule, and read trends over weeks rather than single-day swings.
Don't rebuild strategy around one vendor's dashboard, or around click data Google has said it won't show.

What Search Engine Roundtable reported about tracking tools

In its 16 September video recap, Search Engine Roundtable said Google seems to be poisoning third-party tracking tools with fake data and paywalls, and that this is wreaking havoc on keyword tracking. The same recap says Google launched Gemini 4 Argon and that it is running in AI Mode. A separate Search Engine Roundtable post covers Google updating how links appear in AI Mode and AI Overviews again, and says Google won't show click data for them.

Two cautions belong here. First, the word is 'seems'. Google hasn't published a statement saying it is deliberately serving junk to scrapers, and you shouldn't repeat it as fact in a stakeholder deck. Second, you don't need the cause to be settled to act. Whether the noise is deliberate, a side effect of new defences, or a mix of both, the practical outcome is the same: the numbers your tracker shows may not match what real users see.

If your team is mid-quarter with a keyword report on screen, the question to ask is simple. Which of these figures would I still believe if the vendor disappeared tomorrow?

Why scraped visibility data is a weak base for AEO and GEO

Rank trackers were always an approximation. They query a search engine from a fixed location, with a clean profile, at a chosen time, and record what comes back. That was fine when results were fairly stable and the same for most people. It works much worse for AI answers.

An AI Mode or AI Overview answer is generated. It varies by phrasing, by the user's recent context, by location and by which model version is serving the request. Add a model launch like the one Search Engine Roundtable describes and the same prompt can produce a different set of cited pages next week even if nothing on your site changed. A tracker that records one run of one prompt is sampling a moving target with a single dart.

Now add deliberate or accidental data distortion on top. If a scraper is being served degraded pages, CAPTCHAs or placeholder content, its parser may still extract something and report it with confidence. Dashboards rarely say 'this reading looks wrong'. They draw a line. The line goes down, someone panics, and a content team spends a month fixing a problem that exists only in the tool.

Here's a worked example. A B2B software company sees its tracked share of AI citations for a core topic fall by a third in one week. Its organic sessions, branded search and demo requests are flat. Two explanations compete: the tool broke, or the company lost citations that don't drive visits. Without a second source, nobody can tell. With one, the answer takes ten minutes.

How AI answer engines choose and cite sources

The selection process matters more than any single tracker reading, because it tells you what to build. The pattern, in broad terms, is retrieval followed by synthesis. The system takes the user's question, breaks it into several narrower searches, pulls candidate pages for each, and then writes an answer that leans on a handful of them.

That first step has a name: query fan-out. The Ahrefs Blog published a study of how ChatGPT does it, drawing on 15 million queries. Read the study directly for its findings and numbers. The idea to take from it is structural. One prompt becomes many searches, so a page earns a citation by answering one of the sub-questions well, not by ranking for the headline phrase.

Three consequences follow.

Pages need to answer a complete sub-question. A section that states a definition, a condition and a caveat in a few clean sentences is easy to lift. A section that circles the point isn't.
Entity clarity beats keyword repetition. Engines have to decide what your page is about and who it's from. Plain naming of the product, the category, the author and the date helps. Vague marketing phrasing doesn't.
Being in the retrieval pool comes first. If a crawler can't fetch your page, or the content only appears after heavy client-side rendering, it can't be cited, no matter how well written it is.

Notice what isn't on that list: a particular position for a particular keyword. Position was the metric the old trackers were built to report, and it's the one that translates worst into answer engines.

The link changes covered by Search Engine Roundtable add another wrinkle. If Google keeps adjusting how sources are shown and declines to report clicks from them, you'll have visibility without a click count from Google itself. Plan on inferring impact from your own analytics rather than waiting for a native report.

What to change in your AEO and GEO workflow this quarter

None of this calls for a new strategy. It calls for moving the weight of your evidence from rented numbers to owned ones, and for making content easier to retrieve and quote. In order of effort:

Freeze a prompt panel. Write 40 to 80 real questions your buyers ask, grouped by topic and funnel stage. Don't change the wording between runs. A stable panel makes week-over-week change meaningful even when absolute values are noisy.
Run it by hand, too. Have a person check a small slice of the panel each week in a clean browser. Ten prompts, a screenshot, a note on which sources were cited. It's slow, and it's the cheapest way to catch a tool that has gone wrong.
Check your logs for AI crawler and referral traffic. Server logs show which bots fetch which pages. Analytics shows visits that arrive from AI products. Neither depends on a scraper.
Rewrite your top 20 pages for quotability. Lead each section with the direct answer, keep one idea per section, name things explicitly, and put dates on anything that changes.
Audit rendering and access. Confirm key content is in the delivered HTML, that robots rules reflect a deliberate choice, and that canonical tags point where you expect.
Assign one owner for the number. If five people each pull AI visibility from a different tool, nobody owns the story. Pick the panel, the method and the person.

Teams that run paid and organic together have a further advantage: paid search and AI-surface testing can cross-check each other. The MediaPilot page describes how Cresia approaches media and AI-search visibility, if you want to see how that work sits alongside the rest of a media plan.

How to measure AI search visibility when tools disagree

The aim is triangulation. No single input is trusted. A conclusion needs support from at least two sources of different kinds, and at least one of them has to be first-party.

Signal	Where it comes from	Trust level	Suggested owner	Cadence
Citation presence on a fixed prompt panel	Third-party tracker plus manual spot checks	Medium; verify against manual runs	SEO lead	Weekly
AI crawler fetches by page	Server or CDN logs	High for access, says nothing about citation	Web or platform engineer	Weekly
Referral visits from AI products	Web analytics	High for what it captures; misses unlabelled visits	Analytics lead	Weekly
Impressions and clicks on matching queries	Search Console	High for classic results, partial for AI surfaces	SEO lead	Weekly
Branded search volume and direct traffic	Analytics and Search Console	Medium; slow, indirect	Growth lead	Monthly
Prospect-reported discovery	Form fields, sales call notes	High; small samples	Sales ops	Monthly

Two rules make this table work. First, when the tracker moves and nothing else does, treat the tracker as the suspect. Second, when two first-party signals move together, believe them even if the tracker disagrees.

For referral traffic to be readable, your tagging has to be consistent. If AI-originated visits land in 'direct' or in an unlabelled bucket, your best owned signal is blurry. A written tracking specification that names every source and medium value, plus a way to check that tags stay intact after releases, is boring work that pays off here. The OmniSpec page covers how Cresia frames analytics governance.

Add a lightweight 'how did you hear about us' field with an AI-assistant option to your demo form. It's self-reported and imperfect, and it catches the visits that analytics can't attribute.

What not to do while the data is unreliable

Bad data tempts people into large reactions. A few to avoid:

Don't rewrite your content plan from one week of tracker movement. If a drop doesn't show up in logs, analytics or Search Console, hold your position and re-run the panel.
Don't switch vendors in a hurry and compare the new numbers to the old ones. Different tools sample differently. A change of vendor creates a break in the series, not an improvement.
Don't chase the new model. Search Engine Roundtable reports Gemini 4 Argon in AI Mode. Model behaviour will keep changing. Content that answers questions clearly and accurately survives model changes; content tuned to one model's quirks doesn't.
Don't scrape Google harder to get around it. Escalating against a system that appears to be defending itself wastes engineering time, risks your own IP reputation, and leaves you with data of unknown quality.
Don't promise clients or executives a precise AI share-of-voice figure. Give a range and a trend, and say how it was measured.
Don't invent a click number for AI surfaces. Google has said it won't show click data there, so any figure you produce is a model, not a measurement. Label it as such.

What's safe to ignore? Daily fluctuations, any single-prompt screenshot circulating on social media, and claims that one trick reliably earns citations. What's worth your time is slower: clear pages, clean tagging, open crawl access and a panel you run the same way every time.

What this means for planning and budget

If you present AI search results to leadership, this episode is a good moment to reset expectations. Measurement for answer engines is going to stay fuzzy for a while. Vendors are racing to keep up with products that change monthly, and Google has little reason to make external measurement easy.

The sensible stance is to invest in the things that pay off whatever the tracker says. Pages that answer real questions well also help classic search, and they help the sales team. Clean analytics helps every channel. A prompt panel costs a few hours to build. None of that depends on Google's cooperation.

For wider team context, the growth and analytics pages on this site (growth teams and analytics teams) set out how Cresia thinks about those roles. If you'd rather talk through a measurement plan for your own site, the team can be reached through a demo request.

Frequently asked questions

Is Google really feeding fake data to rank trackers?

Search Engine Roundtable reports that Google seems to be poisoning third-party tracking tools with fake data and paywalls. 'Seems' is the operative word, because Google hasn't confirmed intent. Whatever the cause, the sensible response is to verify tracker output against your own logs and analytics.

Should I stop using my AI visibility tool?

No. Keep it, but demote it from source of truth to one input. Use it for direction and for spotting which topics to check by hand, and make sure at least one first-party signal agrees before you act on a change.

How big should a prompt panel be?

Big enough to cover your main topics and funnel stages without becoming a chore, which for most sites means a few dozen to under a hundred questions. Consistency matters more than size. Keep the wording fixed and run it on the same schedule.

Can I measure clicks from AI Mode and AI Overviews?

Not directly from Google, according to Search Engine Roundtable's report that Google won't show click data for the updated links. You can read referral traffic in your analytics where it's labelled, watch Search Console for related queries, and ask prospects how they found you. Treat any single click figure as an estimate.

Sources

https://www.seroundtable.com/video-09-16-2026-42313.html
https://www.seroundtable.com/google-updates-links-in-ai-mode-and-ai-overviews-again-42308.html
https://ahrefs.com/blog/query-fan-out-chatgpt/