How to optimize content for Claude search

Of the pages Claude cites, 89.1% also appear in Brave's top 20 results for the same queries. That's the first thing we found when we analyzed the content Claude cites.

But it wasn't the most surprising. By running some simple experiments, we figured out that once Claude fetches a page, only a small slice of it ever reaches the generative LLM. We reverse engineered this process, and learned which embedding models and algorithms Claude likely uses — key information for brands looking to optimize their content for Claude.

In this piece, we'll cover:

• Why Claude's search runs on Brave, not another search engine, and how to make sure Brave indexes your pages

• What Anthropic's own research on contextual retrieval tells us about how Claude processes search results

• Why only about an eighth of your page reaches Claude, and how we tested which part gets through

• Four independent signals that Claude uses BM25, a decades-old ranking formula, to select content when responding to prompts

• Which public embedding models best match Claude's choices, and why a model trained on Claude's citations beats them all

• How to write and structure a page so that the right passage gets picked and cited

Disclaimer 

This article is based on external testing and empirical observation, not internal access to Anthropic's codebase. While we referenced Anthropic's public documentation and research papers, our findings are derived from measuring Claude's real-world behavior. Because we cannot inspect the internal pipeline directly, we evaluated different retrieval methods to see which ones best replicate Claude's actual choices. None of these conclusions have been officially confirmed by Anthropic.

Furthermore, Anthropic can change any part of this system at any time, and it may already work differently across Claude models. And retrieval is only one step. Even if a passage from a website scores highly and reaches the model, the model that writes the answer can still decide not to use or cite it. Therefore, treat these findings as strong evidence rather than a guarantee about what tends to work.

For the experiments, we used the Claude API, as it shows much higher interpretability than the Claude interface (directly exposing query fanouts and useful information about chunks of content that were selected during the retrieval process). 

Does Claude's web search use Brave?

Yes, it does.

Anthropic's trust center even confirms it, naming Brave as the company’s official search provider.

We ran a quick test to confirm it. The methodology was simple. We tried to determine what percent of URLs that Claude retrieves can also be found in Brave’s search results when running the same query fanouts. This is called coverage. Perfect coverage would be a 100% match between the URLs found for query fanouts Claude runs and those same queries entered into Brave. When testing 1,000 prompts, 8,500 URLs, and 1,000 queries from Claude, we see coverage of Brave results that’s closer to 90% (89.1% across the top 20 results, to be exact). We consider it strong evidence that Claude is in fact using Brave, not another search engine.

Get your key pages into Brave’s index 

The fact that Claude uses the Brave search engine implies some straightforward optimization steps brands should take. 

One of the easiest and quickest actions to take is to submit URLs to Brave’s index. It’s significantly smaller than Google’s index or Bing’s, so it makes sense to ensure some of the most important URLs from your website are indeed there. You can use this direct URL submission site from Brave. Find the pages other AI platforms cite most, combine that with data from Google Analytics to find the most popular URLs on your website, or find URLs that bring the most traffic from Google in Google Search Console. Then submit those URLs to Brave one by one. 

How does Claude choose what text to pull out of its search results?

This section is a short introduction to how AI systems retrieve information. If you already know RAG and BM25, skip ahead to our test.

What RAG is, and why AI systems need it

A language model only knows what it learned during training. To answer questions about anything newer or more specific, like today's search results, a company's help center content, or a set of legal cases, the model must be given that context in the prompt.

The problem is scale. AI systems like Claude can’t paste thousands of pages into their context. So AI systems use a method called retrieval-augmented generation (RAG). Before the model responds to a question, a separate retrieval step finds the few passages most likely to contain the answer and passes only those to the model for use in generative output.

Traditionally, RAG works in three steps:

1. Chunking. The documents are split into small pieces of manageable token amounts, called chunks.

2. Embedding. Each chunk is converted into an embedding, a list of numbers that represents its meaning. Chunks with similar meanings get similar numbers.

3. Retrieval. When a question comes in, it's also converted into an embedding, and the system picks the chunks that are relevant according to internal scoring formulas. Those chosen chunks go into the prompt and are passed to the generative model, which then produces its answer from them.

This is why retrieval matters so much for visibility. The model can only cite what the retrieval step hands it. For brands, it means that if a passage isn't picked, it won’t appear in the answer to a user’s prompt.

Where traditional RAG falls short

Traditional RAG has two weaknesses, and Anthropic's work (called contextual retrieval) addresses both.

1. Embeddings miss exact matches. Embeddings understand meaning, not exact terms. Anthropic uses the example of someone searching a support database for one specific error code. An embedding model might find pages about error codes in general, but still miss the page that names that exact code being searched.

2. Chunks lose their context. Splitting a document into chunks cuts each piece off from the rest. In Anthropic's example, a financial filing gets chunked, and one piece just says “revenue grew 3% over the previous quarter.” Alone, that sentence doesn't say which company’s revenue grew or in which quarter it happened, so the sentence can’t answer a question about that specific company's results.

BM25: Fixing poorly matched embeddings

To catch exact matches, retrieval systems often add BM25, a ranking formula from the 1990s that scores passages by the words they share with the query. There's no AI in it, just arithmetic over word counts, and it's been the default in search engines like Elasticsearch for years.

BM25 follows three simple rules. 

1. Rare words count more than common ones, because a word that appears everywhere can't distinguish passages from one another. 

2. Repeating a word helps only a little after the first few mentions.

3. Longer passages are discounted, because they're more likely to match words by accident. 

The rest of this article explores whether Claude uses BM25, and if so, how. We'll look at exactly how BM25 scores a page later.

In a hybrid system, BM25 and embeddings each rank the chunks independently, and often the two rankings are merged using formulas like reciprocal rank fusion (RRF). Embeddings find passages that mean the right thing while BM25 finds passages that say the right thing. Anthropic's testing found that this combination beats embeddings alone.

Contextual retrieval: Keeping chunks in context

In September 2024, Anthropic published a blog post about a technique called contextual retrieval that addresses the problem of retrieval with missing context. In contextual retrieval, once the chunking process is done, each chunk is sent to Claude together with the full document, and Claude writes a short explanation of where the chunk fits, usually at a size of about 50 to 100 tokens. That explanation is attached to the chunk, and the combined text is used for both the embeddings and the BM25 index.

Anthropic provided some benchmarks for this technique, finding that: 

• Adding context to the embeddings alone cut failed retrievals by 35%. 

• Adding contextual BM25 on top raised the reduction to 49%.

• Adding a reranker brought it to 67%. 

What this means for Claude's web search

Anthropic’s blog post describes processes for retrieving knowledge. The company hasn't said which of these techniques, if any, Claude uses when it reads search results. But the post does tell us what Anthropic's own researchers found works best: BM25 and embeddings used together, with a reranker on top. It would be surprising if the company's own search product ignored its own findings.

That gives us three concrete questions to test against Claude's real behavior:

1. Does BM25 help predict which passages Claude cites?

2. Which embedding model best matches Claude's choices?

3. How much of a page actually reaches the model?

The rest of this piece answers each question.

Does Claude use BM25?

Probably. Nobody at Anthropic has confirmed which algorithm picks the text Claude reads, for the same reason that nobody confirms much about ranking: saying it out loud invites people to game Claude’s results. But four separate signals point to BM25:

1. Anthropic tested it themselves. Their own contextual retrieval write-up reports that BM25 combined with contextual embeddings beat embeddings alone. They believed in the results enough to publish them on their own blog.

2. BM25 is nearly free. BM25 is a keyword-counting formula from the 1990s. Running it costs very little compared to generating embeddings for every chunk of every page. If a cheap method works well, there's little reason to pursue only marginal performance improvements at higher cost.

3. Their main competitor uses it. In ChatGPT's own debug output, which we captured at Peec AI, every search result carries a scoring breakdown, and one of the named components is labeled “bm25.”

4. Anthropic appears on Turbopuffer's customer list. Turbopuffer runs both vector search (embeddings) and full-text search (BM25), and its own documentation recommends combining the two for better quality.

These are clues, not proof, so we ran our own test.

Getting to the bottom of how Claude retrieves search results

To test whether Claude still uses BM25, and which embedding model it uses, we needed to check whether running these algorithms ourselves could reproduce Claude's actual choices.

That means looking at a page the way Claude does, and Claude sees far less than a person would. A typical page in our data runs to about 14,600 characters. A human would skim and scroll, but Claude forwards only approximately 1,900 characters to the model. Everything else is discarded before the model ever sees it, which means about an eighth of your page is all that reaches the part of Claude that writes an answer to a user’s prompt.

That eighth of a page decides what Claude receives and what can therefore be quoted, however. So any test of how Claude searches has to respect roughly the same limits.

How our test works

Claude's API response shows two things: the exact page it fetched, and the exact sentence it quoted from that page.

That's enough to reverse engineer the selection, making it possible to go back through Claude's decisions.

First, we cut each page into pieces (about 14 for a typical page). This is our own reconstruction. Claude's real chunk boundaries aren't visible from outside. The character budget is, though (roughly 1,934 characters), and we could match it.

Then we scored every piece against the query Claude actually ran. We did this with each method in turn: BM25, an embedding model, and a reranker. We kept only the highest-scoring pieces that fit the budget — about two pieces per page.

One question remained: Was the piece with Claude's real quote among the two we kept when using our own methods?

We repeated this across 734 pages and 1,568 quotes. No website appears in both the BM25 test and the embedding test, so nothing could be memorized by the model.

BM25 finds Claude's quote 4 times out of 10

When run this way, the test showed that BM25 picks the right piece 41.6% of the time. That means the 2 chunks out of 14 we selected with BM25 matched Claude’s quoted chunks in 4 out of 10 cases.

Is that good? Comparing the results to both random chunk selection and a selection methodology where the top chunks of the page are always chosen gives an answer:

BM25 nearly doubles the success rate of blind guessing. And it beats “just take the intro” by 7 percentage points.

If you put stronger constraints on the BM25 algorithm, the results look better still. When we gave it only one chunk (about a tenth of the page), the algorithm still found Claude’s chosen quote 23.7% of the time. Random picking at that size had a 9.8% match rate. That's 2.5x better than luck.

BM25 matches a reranker and beats embeddings

The BM25 results are impressive for an algorithm that comes from the 1990s. Retrieval has of course advanced since then, so you'd expect something modern to outperform the old methods.

So we tested that, too. We took two chunks from the same page: one Claude quoted, one it ignored. Then we checked how often BM25 and newer methods such as embeddings and a neural reranker rated the quoted piece higher than the ignored one.

BM25 beats embeddings, and matches a neural reranker, a model built specifically for this exact judgment, at a fraction of the cost.

These results on their own aren’t proof of Anthropic’s choice of BM25. But together with the four signals mentioned earlier, they’re a strong indication that Anthropic uses BM25 to pick what Claude reads.

Which embedding model does Claude use?

BM25 has one blind spot: it can only match words it can see.

If someone searches “how long does onboarding take,” and your page answers with “Most teams are live within a day,” you’ve provided an answer, but it shares no words with the query. To BM25 that sentence scores zero. A page saying “our onboarding process takes as long as it takes,” however, would score well, since it repeats the query's words even though it says nothing particularly meaningful.

Instead of counting words, an embedding model reads a passage and turns it into a list of numbers, usually a few hundred, that encode what it's about. Passages with similar meanings end up with similar numbers, whether or not they share vocabulary. “Most teams are live within a day” sits close to “how long does onboarding take,” and far from “quarterly revenue guidance.”

Embeddings come from dozens of models, and each one places passages slightly differently. The only way to answer the question of which model Claude uses is to test candidate models against Claude's real results.

Anthropic has pointed to its own candidates publicly. In its contextual retrieval research, Anthropic reported that Voyage and Gemini embeddings performed best in its testing, and its RAG sample code uses Voyage. That gave us two candidates to test.

Testing 12 embedding models against Claude's real citations

We used the same method as the BM25 test to test these embedding models. We took a page Claude fetched, the fanout queries Claude searched, and the chunk Claude cited. We scored every chunk on the page with 12 various embedding models, kept the top-ranked chunks that fit within a given share of the page, and checked whether Claude’s actual citation was among them.

Below is the score for Gemini Embedding 002: 

Of the public embedding models we tested (12, including Qwen, Gemini, Voyage, Jina, Harrier, and ModernBERT), Voyage and Gemini came closest to Claude's real choices, consistent with research done by Anthropic back in 2024 discussed in its contextual retrieval blog post. 

Off-the-shelf models don't explain everything

Gemini and Voyage are the best public match to Claude’s own choices. But when we trained our own model, it reproduced Claude's citations better than any public embedding model did.

That suggests Claude's retrieval isn't simply Voyage or Gemini, used as-is. More likely, Anthropic uses a custom embedding model, possibly shaped by the same approaches that make Voyage and Gemini strong. They could also use public embeddings combined with other ranking steps, such as a reranker or the way the lexical and semantic scores are merged. Either way, the takeaway remains unchanged: to perform well with the embedding model, write chunks that carry an unmistakable meaning.

Summary: What we found and what it means for you

1. Claude searches with Brave, not Google
Of the pages Claude cites, 89.1% appear in Brave's top 20 for the same queries. Anthropic's trust center names Brave as its search provider.
What to do: Ranking on Google doesn't guarantee Claude will find you. Make sure your most important pages are in Brave's index, and submit them if they aren't.

2. Claude searches with rewritten queries, not the user's question
Claude turns a question into several short fanout queries and adds words the user never typed, such as the year, the audience, and specifics like “16 weeks.”
What to do: Optimize for the fanout queries, not the original prompt. Look at the real ones (in Peec AI: Results → Fanouts) and make sure your page uses the words they contain.

3. Only about an eighth of your page reaches the model
A typical page in our data has about 14,600 characters, but Claude forwards only around 1,900 of them. Being fetched doesn’t guarantee the right information gets sent to the LLM for response. The passage that answers the question has to be selected.
What to do: Identify the one passage on each page that has to be picked, keep it tight, and move it up the page if it's buried near the end.

4. BM25, a keyword formula from the 1990s, predicts what Claude picks
The BM25 formula found the passage Claude quoted from a web page 41.6% of the time, against 21.9% of the time for randomly picked passages and 34.1% of the time for passages always taken from the top of the page. Its performance matched a neural reranker and beat embeddings. Anthropic's own research points to its use.
What to do: Write precisely. Rare, specific words like numbers, years, audiences, and exact names perform well in the BM25 algorithm. Generic words and repeating your main keyword barely help.

5. Meaning matters, too, and Claude's embeddings appear custom-tuned
Voyage and Gemini, the two embedding models Anthropic's research rated best, both predict Claude's choices far better than chance. But a model we fine-tuned on Claude's real citations beat both, which suggests Claude doesn't use an off-the-shelf solution.
What to do: Web pages are judged twice: on words (BM25) and on meaning (the embedding model). The passage that Claude selects uses the query's words and answers the question clearly.

The short version: You don't need to know Claude's exact stack. Each important page needs one passage that passes two tests. 

1. Does it contain the specific words a rewritten query will use? 

2. Does it still make sense once it's cut out of the page? 

Creating content that passes these two tests is simple good practice, and holds for Claude, for ChatGPT, and for whatever comes next.

Bonus for geeks: Contextual chunking

How Anthropic fixes the problem of keeping chunks with their context

As noted above, we ran these tests because Anthropic has developed a method called contextual retrieval to fix the issue of lack of context of chunks. It’s Anthropic’s way of using an LLM to add context to each of the chunks. The exact prompt looks like this: 

<document>

{{WHOLE_DOCUMENT}}

</document>

Here is the chunk we want to situate within the whole document

<chunk>

{{CHUNK_CONTENT}}

</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else

<document>

{{WHOLE_DOCUMENT}}

</document>

Here is the chunk we want to situate within the whole document

<chunk>

{{CHUNK_CONTENT}}

</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else

<document>

{{WHOLE_DOCUMENT}}

</document>

Here is the chunk we want to situate within the whole document

<chunk>

{{CHUNK_CONTENT}}

</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else

To keep costs low, Anthropic used Claude 3 Haiku with prompt caching.

Why it matters for AI search

Other AI platforms, such as ChatGPT, chunk web pages too. Whether they add context to chunks the way Anthropic described isn't public. Some might, others might score raw chunks.

That uncertainty means you can't count on the engine to repair your chunks. 

You can, however, write chunks that don't need repairing. A chunk that is vague, full of pronouns, or contradicts another part of the page is less likely to be retrieved, and a chunk that isn't retrieved can't be cited.

The geeky idea: Use Anthropic’s trick on your own pages

Turn Anthropic's retrieval-side method into a publishing-side audit.

1. Chunk your page using basic chunking techniques: split on H2 and H3 headings or use semantic chunkers that take sentence similarity into account, such as Chonkie. 

2. Generate context for each chunk with Anthropic's prompt above.

3. Score each chunk twice against real user queries, with any LLM as the judge, or combination of BM25, embeddings, and reranker. Score it once alone, then once with the generated context. For the queries, use the fanout queries AI engines actually run rather than invented ones. Tools like Peec AI show actual fanouts from AI platforms like Claude or ChatGPT. 

4. Look at the gap. If both scores are high, the chunk is self-contained. If the raw score is low but the contextual score is high, the chunk depends on the rest of the page and needs fixing. If both are low, the chunk simply doesn't answer the query, which is a content problem, not a chunking one.

5. Merge the context into the chunk. For flagged chunks, rewrite the text so the generated context is part of it. Name the product, replace pronouns, and add scope and dates, without pasting an intro sentence on top.

6. Test the rewrite without the context. Score the rewritten chunk on its own. If it now scores as high as the original did with the context, the fix worked. 

It’s also worth doing a final pass by having an LLM compare all chunks on the page against each other and flag contradictions such as different prices, numbers, or claims.

What you'll usually end up fixing

• Pronouns and vague references like “it,” “the tool,” and “this feature.” Replace them with the actual name.

• Generic headings like “Overview.” Use descriptive headings, which often mark chunk boundaries.

• Buried answers. Put the direct answer first in each section.

• Missing scope. State the year, the audience, and the platform in the text.

• Orphaned tables and lists. Give each one a caption or lead sentence naming its subject.

Finally, be aware that this technique isn’t limited to Claude. It should help you improve retrieval in multiple AI platforms.