Semantic Search vs. Keyword Search for Agent Knowledge Bases: Benchmark and Implementation

If you’ve built a RAG pipeline for a Claude agent and wondered why it keeps surfacing irrelevant chunks — or worse, missing obviously relevant ones — the retrieval layer is almost certainly the culprit. The debate around semantic search vs keyword search for agent knowledge bases isn’t academic. It directly determines how often your agent says something confidently wrong because it retrieved the wrong context.

I’ve run both approaches against the same knowledge bases across three production-ish setups: a customer support bot with a 4,000-document FAQ corpus, a technical documentation agent over API reference material, and a compliance Q&A system with dense regulatory text. Here’s what the numbers actually showed — and why the answer isn’t “just use semantic search.”

What Each Approach Actually Does

Before the benchmarks, let’s be precise about what we’re comparing — because “semantic search” gets used loosely enough to be meaningless.

Keyword Search (BM25)

BM25 is the algorithm behind Elasticsearch, OpenSearch, and most production search you’ve used. It scores documents based on term frequency (how often a query term appears in a doc) and inverse document frequency (how rare the term is across the corpus). It’s fast, deterministic, and completely interpretable — you can always trace why a result ranked where it did.

The main implementation for Python is rank_bm25. It runs entirely in memory with no GPU requirement:

from rank_bm25 import BM25Okapi
import re

def tokenize(text):
    # Simple whitespace + lowercase tokenization
    # For production, add stemming via nltk or spacy
    return re.sub(r'[^a-z0-9\s]', '', text.lower()).split()

corpus = [doc['content'] for doc in documents]
tokenized_corpus = [tokenize(doc) for doc in corpus]

bm25 = BM25Okapi(tokenized_corpus)

def keyword_search(query: str, top_k: int = 5) -> list[dict]:
    tokenized_query = tokenize(query)
    scores = bm25.get_scores(tokenized_query)
    top_indices = scores.argsort()[-top_k:][::-1]
    return [
        {"doc": documents[i], "score": float(scores[i])}
        for i in top_indices
        if scores[i] > 0  # Filter zero-score results
    ]

Semantic Search (Dense Vectors)

Semantic search embeds both documents and queries into a high-dimensional vector space, then retrieves by cosine similarity. The query “how do I cancel my subscription” will match a document about “terminating your account” even with zero term overlap.

from sentence_transformers import SentenceTransformer
import numpy as np

# text-embedding-3-small via OpenAI API is ~$0.00002 per 1K tokens
# For local, all-MiniLM-L6-v2 is fast and surprisingly good
model = SentenceTransformer('all-MiniLM-L6-v2')

# Index time — do this once and persist
doc_embeddings = model.encode(
    [doc['content'] for doc in documents],
    batch_size=32,
    show_progress_bar=True
)

def semantic_search(query: str, top_k: int = 5) -> list[dict]:
    query_embedding = model.encode([query])[0]
    
    # Cosine similarity via dot product on normalized vectors
    similarities = np.dot(doc_embeddings, query_embedding) / (
        np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(query_embedding)
    )
    
    top_indices = similarities.argsort()[-top_k:][::-1]
    return [
        {"doc": documents[i], "score": float(similarities[i])}
        for i in top_indices
    ]

For production at any volume, you’d swap the numpy brute-force for a vector database. Our Pinecone vs Qdrant vs Weaviate comparison covers the tradeoffs in detail — Qdrant is my current default for self-hosted deployments.

The Benchmark Setup and Results

I tested three query categories against each approach, using a 50-question evaluation set with manually graded relevance (1-5 scale). Retrieval quality was measured as NDCG@5 (normalized discounted cumulative gain at 5 results). Higher is better, max is 1.0.

The pattern is clear: BM25 dominates on exact terms, IDs, and jargon. Semantic search dominates on paraphrased, conceptual, and natural language queries. Hybrid wins or ties in nearly every category.

Latency numbers (p95, 10K docs, M1 Pro, local embeddings):

Keyword Search: When BM25 Is the Right Tool

BM25 is underrated in the LLM tooling world because everyone got excited about embeddings. But for certain workloads it’s genuinely superior — and the failure mode of semantic search on these cases is subtle enough to bite you in production.

Use keyword search when your queries contain:

• Product codes, invoice numbers, SKUs, ticket IDs — semantic search is actively bad here. An embedding model has no idea that “INV-2024-0891” is more relevant than “INV-2024-0892”.
• Specific version numbers or API names — “python 3.11 breaking changes” shouldn’t match “python 3.9 changes” just because they’re semantically similar.
• Legal or regulatory citations — “Section 12(b)(3)” needs exact term matching.
• Short corpora under 500 documents — the overhead of embedding indexing rarely pays off at this scale.

BM25 also has zero infrastructure requirements. No vector DB, no GPU, no embedding API costs. For an internal tool hitting a 200-doc knowledge base, it’s the pragmatic choice.

Semantic Search: When Embeddings Win

Semantic search earns its complexity on natural language queries from non-technical users. If your agent is customer-facing — a support bot, an onboarding assistant, an HR tool — users won’t know the exact terminology in your knowledge base. They’ll ask “how do I change my email” when your document says “update account credentials.” BM25 scores that query at zero. Semantic search nails it.

The embedding model choice matters more than most tutorials acknowledge. text-embedding-3-small from OpenAI costs roughly $0.00002 per 1K tokens and outperforms older ada-002 on most benchmarks while being cheaper. For local/self-hosted deployments, BAAI/bge-small-en-v1.5 punches well above its 130MB weight.

Where semantic search quietly fails: when your corpus uses very domain-specific terminology that wasn’t well-represented in the embedding model’s training data. Medical device documentation, niche financial instruments, proprietary internal jargon — embedding models will hallucinate similarity between unrelated concepts because they’ve never seen the terms in context. This is one of the root causes of the hallucination patterns described in our guide on reducing LLM hallucinations in production.

Hybrid Retrieval: The Production Default

The benchmark data above makes the case for hybrid retrieval. It’s not a magic bullet, but it removes the worst failure modes of each approach. The standard implementation uses Reciprocal Rank Fusion (RRF) to merge results from both retrievers without needing to normalize scores:

from collections import defaultdict

def reciprocal_rank_fusion(
    bm25_results: list[dict],
    semantic_results: list[dict],
    k: int = 60  # RRF constant — 60 is the standard default
) -> list[dict]:
    """
    Merge two ranked lists using Reciprocal Rank Fusion.
    k=60 is from the original Cormack et al. paper and works well empirically.
    """
    scores = defaultdict(float)
    doc_map = {}
    
    for rank, result in enumerate(bm25_results):
        doc_id = result['doc']['id']
        scores[doc_id] += 1 / (k + rank + 1)
        doc_map[doc_id] = result['doc']
    
    for rank, result in enumerate(semantic_results):
        doc_id = result['doc']['id']
        scores[doc_id] += 1 / (k + rank + 1)
        doc_map[doc_id] = result['doc']
    
    sorted_ids = sorted(scores.keys(), key=lambda x: scores[x], reverse=True)
    
    return [
        {"doc": doc_map[doc_id], "rrf_score": scores[doc_id]}
        for doc_id in sorted_ids
    ]

def hybrid_search(query: str, top_k: int = 5) -> list[dict]:
    bm25_results = keyword_search(query, top_k=20)   # Cast wider net
    semantic_results = semantic_search(query, top_k=20)
    fused = reciprocal_rank_fusion(bm25_results, semantic_results)
    return fused[:top_k]

One thing the documentation consistently undersells: you need more candidates from each retriever before fusion than you actually want as output. Fetching top-20 from each and fusing to top-5 consistently outperforms fetching top-5 from each. The diversity in the wider net is the point.

If you’re building the full RAG pipeline around this retrieval layer, the RAG pipeline from scratch guide covers chunking strategy, metadata filtering, and context window packing — all of which interact with retrieval quality in non-obvious ways.

Cost Comparison for Agent Knowledge Bases

At 100,000 queries/month using OpenAI embeddings for semantic search, you’re looking at roughly $0.20/month in embedding costs — negligible. The real cost is the vector database. Managed Pinecone starts at $70/month for the serverless tier with any meaningful usage. Qdrant self-hosted is free but requires a container to manage. This is where your architecture choices matter: if you can run a local embedding model, the cost profile drops to near-zero even for semantic search.

Failure Modes You’ll Hit in Production

Semantic search failure: embedding drift. If your knowledge base uses terminology that shifts over time (product names change, processes get renamed), embeddings indexed 6 months ago may cluster differently than new queries. Re-indexing is the fix, but you need to monitor for it. Retrieval quality silently degrades without any error thrown.

BM25 failure: vocabulary mismatch. Users say “unsubscribe,” your docs say “cancel membership.” BM25 returns nothing useful. This is the failure mode that drives people toward semantic search without questioning whether hybrid would be sufficient.

Hybrid failure: RRF weighting assumptions. RRF assumes both retrievers are roughly equally reliable. If your BM25 index is on poorly tokenized text (lots of camelCase, snake_case, technical symbols), its rankings are garbage and will pollute the fusion. Pre-process your text for BM25 separately from your semantic chunks. They can and should be different representations of the same documents.

Building robust retrieval is also a reliability engineering problem. The patterns in our article on LLM fallback and retry logic for production apply here too — if your vector DB is down, falling back to BM25-only is better than returning an error to your agent.

Verdict: Choose Based on Your Query Distribution

Choose pure BM25 if: your corpus is under 500 documents, your queries are primarily exact-term or identifier-based, you need zero-infrastructure deployment, or you’re prototyping and want to iterate without managing a vector DB.

Choose semantic search if: your users are non-technical and use natural language, your knowledge base topics have significant synonym/paraphrase diversity, or you’re building on top of a vector DB you already have for other purposes.

Choose hybrid (RRF) if: you’re in production with real user queries, your corpus is over 1,000 documents, you have a mix of natural language and exact-term queries, or you can’t afford retrieval failures to reach the LLM (because bad retrieval is a leading cause of hallucinations in RAG systems).

My default recommendation for any serious agent knowledge base: hybrid from day one. The incremental complexity over pure semantic search is one function (~20 lines). The NDCG improvement on the query types that matter most — short, ambiguous, mixed — is consistent enough that I’ve stopped deploying pure semantic search for anything customer-facing. The semantic search vs keyword search debate resolves in production not as a binary choice, but as a weighting problem: most real query distributions need both.