Measuring a Brand Inside an Answer: The Hard Problems Nobody Solved Yet

When a buyer asks ChatGPT "what's the best CRM for a 12-person remote team under 100 euros a month," the answer that comes back names three or four products, cites a handful of pages, and moves on. The buyer never sees a ranked list of ten blue links. They see a paragraph. Maybe they ask one follow-up. Then they decide.

That shift broke most of the tooling that brands used to understand their own visibility. Rank tracking measured position on a results page. There is no results page now. There is a generated sentence, different every time, assembled from sources the engine chose and phrased in language the engine wrote. Knowing whether your brand showed up in that sentence, and whether it showed up correctly, turns out to be a genuinely unsolved measurement problem.

This is the problem we work on at CiteSurge. Below are the parts of it that are still open. Not "open" in the marketing sense of a roadmap. Open in the sense that the research literature does not yet have a settled answer, and the teams shipping products in this space are making engineering bets, not following a playbook.

Why isn't matching the brand name enough?

The first instinct everyone has is wrong. You search the AI's answer for the brand name, and if the name is there, the brand was mentioned. Done.

It falls apart immediately. "Apex" is a CRM, a legal-software company, a Salesforce programming language, a gym chain, and a video game. When an answer says "Apex handles that well," which Apex did the engine mean? A naive string match counts every one as a hit for whichever Apex you happen to be tracking. Now your visibility dashboard is reporting numbers that are partly fiction, and the customer paying for that dashboard has no way to know.

This is entity resolution, and it is one of the older hard problems in information retrieval. It did not get easier when the text being searched started coming from a language model instead of a web page. It got harder, because the model paraphrases. It will refer to a company by a product name, a parent company name, a former name, or a description that never includes the name at all. "The Portuguese fintech that does instant SEPA transfers" is a mention. No string match finds it. A CRM described as "designed for teams that prioritize European compliance" may be mentioned without naming the product at all, and a naive system would score that as an omission when the brand was actually discussed in precise terms.

The research here moved in 2024 and 2025, with span-extraction models like GLiNER showing that you can pull named entities out of arbitrary text without training a model per entity type. That helps. It does not close the problem. Knowing that "Apex" is an entity in the sentence is a different task from knowing which real-world Apex it points to, and that second task, entity linking, is where the accuracy still drops.

The state of the published research is that GLiNER and similar token-classification models can identify candidate entity spans with reasonable recall. But linking those spans to real-world entities, deciding whether an "Apex" mention refers to your Apex, a different Apex, or no real entity, still relies on hand-curated knowledge graphs, fuzzy string matching, or learned disambiguation models. None of those scale automatically to a new entity without either building a training corpus or hand-maintaining a knowledge base. So the research literature solves "find mentions" but not "correctly identify which entity was mentioned," and that gap is where the work is.

Can a citation be valid and still be wrong?

Suppose the answer cites a page, and the page is real, and the page is on your domain. Surely that counts as a citation for your brand.

Not necessarily. The engine might cite your pricing page while making a claim about your security certifications, a claim that page does not support. The citation is structurally valid and substantively wrong. If you count it as a clean win, you are measuring the presence of a link, not the truth of what the link was used to say.

This distinction has a name in the research literature now: answer correctness and citation faithfulness are separate properties. Recent work on retrieval-augmented systems makes the point directly, that a system can produce a correct answer with unfaithful attributions, and the two failures need to be measured independently. Most visibility tools do not measure the second one at all. Checking it means extracting the specific claim the answer is making, finding the span of the cited source that should support it, and running an entailment check: does this source actually support this claim, contradict it, or say nothing about it?

That is a natural-language-inference problem stacked on top of a retrieval problem stacked on the entity problem. Each layer has its own error rate. They compound.

The size of that gap is now measured. A December 2024 study of retrieval-augmented attributions, Correctness is not Faithfulness in RAG Attributions, found up to 57% of citations were unfaithful even when the answer they supported was correct.

Here is what makes this a research gap rather than a solved problem. The published research on faithfulness evaluation has focused on RAG systems and retrieval-augmented models, where you control the documents in the corpus. But answer engines like ChatGPT do not retrieve a single passage and cite it. They synthesize an answer from potentially dozens of implicit sources, then retroactively cite which of those they could verify. The citation may be a post-hoc rationalization of something the model generated from pattern-matching rather than from the cited page. Detecting that requires not just checking whether a page supports a claim, but determining whether the claim was actually sourced from that page or whether the page was selected after the claim was generated. That distinction is not settled in the literature, and it is where the measurement gap lives.

There is already a baseline for how often this fails in practice. A 2023 audit of generative search engines, Evaluating Verifiability in Generative Search Engines, found only 74.5% of citations fully supported the sentence they were attached to.

Why does precision matter more than recall here?

There is a tradeoff in every classification system between catching everything (recall) and being right about what you catch (precision). Most systems can be tuned toward one or the other.

In brand monitoring the tradeoff is not symmetric, and that asymmetry shapes everything. A missed mention is a gap in a report. A false mention is a lie in a report. If we tell a customer their brand appeared in an answer where it did not, and they make a decision on that, we have done something worse than fail to inform them. We have misinformed them with confidence. One false positive on a dashboard costs more trust than several misses, so the entire pipeline has to be built precision-first, with recall as the thing you carefully claw back without contaminating the precision you started with.

That constraint has a real architectural consequence. The part of the system that generates candidate mentions wants to be greedy, casting a wide net. The part that decides what counts has to be strict. Those two impulses fight, and keeping them in separate stages, so the greedy half never leaks into the strict half, is harder than it sounds when both halves are partly powered by language models that like to be agreeable.

The cost asymmetry cuts deeper than it first appears. A false positive does not just mislead a customer. It also corrupts the confidence calibration of the entire system. If a monitoring dashboard reports visibility numbers that are inflated by false positives, a customer acts on those numbers and finds they do not match reality. The customer loses confidence not just in that one report, but in the system as a whole. Meanwhile, a false negative is invisible. A missed mention never breaks confidence because the customer never learns about the mention that should have been there. That means the measured precision floor has to be very high before a system is trustworthy at scale. Missing 10% of mentions while achieving 98% precision is worse than catching everything and being 90% precise, because the customer will find the errors and discount everything else.

So the real constraint is not just precision-first design. It is building a system where the precision budget is the binding constraint, and where every stage of the pipeline is designed to expand recall only as far as precision allows. That usually means an early broad candidate generation stage, then a series of increasingly strict filters, with LLM judgement reserved for the genuinely ambiguous cases where a person would also have to make a judgment call.

Why do you need a human-labeled answer key?

Here is the part that surprises people who assume this is all just "ask a better model."

To know whether your classifier is any good, you need an answer key: a labeled set of real answer-engine outputs where a human has decided, case by case, this is a true mention, this is a homonym, this citation supports its claim, this one does not, this source is a competitor, this one is just a directory. Without that answer key you are tuning a system against your own assumptions and calling the result accuracy.

Building that answer key is the actual bottleneck. Not the model. The labeled data. And labeling it well requires deciding the rules first: does a marketplace listing count as the client's own page or the platform's? Is a parent-company mention a mention of the product? When two human reviewers disagree on the hard cases, which of them is right, and what written guideline makes that repeatable? These are not questions a model answers. They are judgment calls that a person has to make and write down, so that the labeling stays consistent across thousands of cases and the eval set means something.

The synthetic-data research from 2025 is honest about this. Language models can help generate and pre-label data, and that speeds things up. But on the lexical diversity of what they produce, the studies keep finding the same thing: model-generated examples are less varied than what real humans write, so synthetic labels supplement human judgment rather than replace it. The frontier work is in how much you can safely automate before the answer key stops reflecting reality.

What the research does not address is the labor cost of building a defensible answer key at scale. A production brand-visibility system might need to evaluate hundreds of brands across thousands of prompts, generating millions of answer-engine responses. A labeled sample large enough to support statistical confidence intervals for all the decision thresholds and entity types requires thousands of hand-labeled cases. That is not a one-time cost. Answer engines change their output format, citation practices, and answer strategies regularly. An answer key built in March can be partially obsolete by June because the thing it measures shifted. So building an answer key is not just a data problem; it is a continuous data-operations problem. The research literature has not settled how to maintain an answer key under continuous change while keeping calibration costs manageable.

Why are the prompts themselves a research problem?

Step back from measuring answers and consider what you even ask the engines. To monitor a brand's visibility you need a set of prompts, the questions you fire at ChatGPT and Perplexity and Gemini week after week. Get the prompts wrong and every downstream number is measuring the wrong thing.

The lazy version writes a dozen "best X" questions from a brand brief and calls it monitoring. It misses how buyers actually talk. Real research queries carry constraints: a budget, a region, a team size, a compliance requirement, an existing tool the new one has to integrate with. "Best analytics platform that works with HubSpot and Snowflake for a Series A team in the EU" is a real question. "Best analytics platform" is not a question anyone asks an AI when they are close to buying.

Generating prompts that are realistic, diverse, grounded in the brand's actual capabilities, and balanced across branded and unbranded and comparison forms, without the whole set collapsing into near-duplicates, is its own synthetic-generation problem. The 2026 guidance points toward multi-stage pipelines: mine real queries from search logs and forums, expand them with personas, then run the candidates past critique and deduplication before any of them reach a dashboard. Each of those stages is a place where quality is won or lost, and there is no settled recipe for how to weight them.

The published research on query generation focuses on generating queries for evaluation and retrieval benchmarks. But those queries are typically generated to be diverse and varied within a narrow domain. Monitoring prompts face a different constraint: they must be diverse and realistic without drifting away from the brand's actual positioning and market. A generated prompt that is perfectly realistic but about a use case the brand does not serve wastes monitoring resources on invisible visibility gaps. A prompt too narrowly tailored to brand strengths misses where competitors are winning. Balancing those two pulls, and doing it at the scale needed to monitor competitive set changes and seasonal shifts in buyer behavior, is a place where the published research offers guidance but no solution.

Why do answer engines disagree on the same question?

Here is a wrinkle the published research mostly ignores. ChatGPT's answer to a question is not the same as Perplexity's answer to the same question, and neither matches what Google AI Overviews returns. The brands mentioned differ. The citations differ. The tone differs. Sometimes the correct answer differs.

From a measurement perspective, this creates a visibility multiplier problem. Your brand might appear in ChatGPT's answer to "best CRMs for small teams" but not in Perplexity's answer to the same question, even when both engines have access to the same pages. That is not because one engine is right and one is wrong. It is because they have different retrieval strategies, different synthesis approaches, and different answer-formatting conventions. ChatGPT tends toward more expansive answers with multiple options. Perplexity often leads with a single strong recommendation. Google's AI Overviews format fits different word counts than a chat response.

The visibility measurement problem, then, is not "is my brand visible in AI answers?" but the harder set of questions: "which answer engines matter for my buyers, what does visibility look like on each one, which engines are my competitors winning on, and how does share of voice compare across them?" That requires not just a single monitoring framework but a multi-engine monitoring framework that accounts for engine-specific behaviors.

The published research on LLM behavior and synthesis does not address this. Studies typically evaluate a single model or a small set. The comparative behavior of production answer engines at scale (how their mention frequencies, citation patterns, and recommendation rankings differ for the same queries) is not well-studied. We know from anecdotal evidence and user reports that they differ significantly. But quantifying those differences, building engine-specific evaluation criteria, and adjusting for engine drift independently while monitoring brand visibility across all three, is a place where there is no published playbook.

This also creates a secondary problem: choosing which engines to monitor. For a B2B SaaS company, the relevant answer engine might be ChatGPT (high adoption, broad audience) or Perplexity (high research intent), or Google AI Overviews if the brand is discovery-sensitive. For a consumer product, TikTok search and Reddit might matter more than formal AI platforms. A small news startup might care most about Claude's answer-engine adoption and whether Claude cites them. The monitoring stack needs to be market-and-audience-specific, which means the prompt set, the evaluation rules, and the success metrics all have to be customized per engine.

Why don't solved sub-problems stay solved?

Why this is interesting and not just hard is that the problems do not stay solved.

It would be easy to read all of the above as a list of reasons the work is annoying. We read it as the reason the work is worth doing.

Every one of these problems sits at a point where the published research is recent, incomplete, and moving. Entity linking on paraphrased generative text. Citation faithfulness as a metric distinct from correctness. Precision-first classification where the cost function is asymmetric and customer-facing. Answer-key construction where automated labeling helps but cannot be trusted alone. Prompt synthesis that survives contact with how people actually ask. Cross-engine behavioral differences that require multi-framework evaluation. None of these has a vendor you can buy the solution from, because the solution does not exist yet in a box.

The answer engines themselves do not hold still either. The way ChatGPT formats a citation, the way Google's AI Overviews decide when to trigger, the way Perplexity weights its sources, all of it changes without notice. A measurement system that was accurate in March can drift by June because the thing it measures moved underneath it. So even a solved sub-problem does not stay solved. It needs monitoring for drift, which is another open problem wearing a different hat.

This is where the temporal and maintenance cost of a visibility system becomes the binding constraint. Every component has a half-life. Prompts become stale as buyer behavior evolves. Answer-key labels become obsolete as answer engines change their output format. Entity records need updating as companies merge, rename, or shift product focus. A classifier trained in Q1 may have degraded precision by Q3 because the signal it learned from drifted. So the work is not building a system once. It is building a system and committing to continuous monitoring and retraining. That is a much larger bet than the initial engineering effort.

This is the work

It is closer to applied research than to building a dashboard, and the brands and investors paying attention to this space are starting to understand that the moat in answer-engine measurement is not the chart at the end. It is the accuracy of every judgment behind the chart, and the accuracy is where the unsolved problems live.

We are building a system because the problems matter. They matter to brands that need to understand where they show up and whether they show up correctly in the AI systems buyers use. They matter to investors evaluating whether a company has real competitive advantage or just SEO. They matter to the integrity of AI answer engines themselves. If a system is citing pages incorrectly or misrepresenting entities consistently, that erodes user trust. A measurement system that catches that is infrastructure for the answer-engine ecosystem.

The frontier work is not the infrastructure that executes the measurements once. It is the judgment-engineering system that ensures those measurements are stable, defensible, and granular enough to drive real business and product decisions. That is research work. We will keep writing up what we learn as we push on each of these.

The honest state of the field, in mid-2026, is that the questions are sharper than the answers. That is exactly why it is worth our time.

CiteSurge measures brand visibility across AI answer engines using precision-first classification, multi-engine evaluation, and continuous answer-key maintenance. It is built by BrainJuice Labs. If the problems above are the kind of thing you think about, the engineering is where the open questions are.