Production Large Language Model Engineering: Part 3

Embeddings, semantic search, and why vector search fails quietly

Traditional database search leans on exact fields, keywords, structured filters, and full-text indexes. A telehealth platform might let you search clinics by name, location, service type, hours, or specialty — and that works fine for Clinic name: Kisumu Children's Clinic. It falls over the moment a user asks something like “where can I treat a child with fever near Kisumu?” — because nothing in that sentence is guaranteed to match the database’s actual service labels.

That’s the gap semantic search closes. Traditional search matches words and attributes. Semantic search tries to match meaning, regardless of whether the wording lines up.

Embeddings: coordinates on a meaning map

An embedding is a numerical vector representing the semantic characteristics of something — a sentence, a query, a clinic description, a document chunk, an image, a code fragment. It looks like a long list of floats: [0.12, -0.41, 0.83, 0.09, ...], often hundreds or thousands of dimensions deep. Individually, those numbers don’t mean anything you can point to. What matters is the geometry between complete vectors — the mental model worth keeping is an embedding as a coordinate on a multidimensional map of meaning.

Two descriptions that mean roughly the same thing end up near each other on that map: “Pediatric clinic offering malaria treatment” and “Children’s health center treating fever and tropical diseases” should land close together, while “Car insurance provider” should land far away from both. That closeness is exactly what makes retrieval by meaning possible instead of retrieval by keyword overlap.

How semantic search actually runs

Take that same query — “where can I treat a child with fever near Kisumu?” — through the pipeline:

User query → Embedding model → Query vector → Vector search
    → Similarity ranking → Relevant clinic IDs → Complete clinic records

Embed the query, compare it against stored embeddings, rank by similarity, apply whatever metadata filters apply (country, language, service type), and return the matching records. The vector search step finds conceptual neighbors, not exact string matches.

Cosine similarity is the usual metric for “how close.” It measures the angle between two vectors — close to 1 means strongly aligned, close to 0 means weakly related or unrelated, close to -1 means pointing in opposite directions. “Pediatric malaria treatment” against a clinic described as treating “children’s fever and tropical disease” should score high. But don’t copy a threshold number from a blog post (including this one) and assume it transfers — the exact score range depends on the embedding model, whether vectors are normalized, the vector database’s distance metric, and the domain. Validate any threshold against your own data.

Where embeddings live, and why they’re not the source of truth

Embeddings get stored in something purpose-built for vector search — pgvector on top of Postgres, Pinecone, Weaviate, Milvus, Qdrant, Elasticsearch, Redis, or a managed cloud option. Most production setups end up running two stores side by side rather than one.

The primary database holds the actual business record — clinic ID, name, services, location, hours, contact info. The vector store holds only what retrieval needs: the record ID, its embedding, the searchable text, and metadata. Vector search returns IDs; the application then goes and fetches the current, complete record from the primary database.

Splitting them this way isn’t extra complexity for its own sake. It keeps business data under one clear owner, gives you transactional consistency where it matters, lets each store scale on its own terms, and makes re-embedding, deleting, or updating a record far less painful than it would be if the vector store were also your system of record.

What you embed matters more than which model you pick

Embedding quality lives or dies on the input text, and it’s easy to under-feed the model without realizing it:

// Weak: the model gets almost nothing to work with
String content = clinic.getName(); // "Smile Bright Dental"

// Better: rich, structured context
String content = String.format(
    "Clinic: %s. Services: %s. Specialties: %s. " +
    "Location: %s. Patient groups: %s.",
    clinic.getName(), clinic.getServices(), clinic.getSpecialties(),
    clinic.getLocation(), clinic.getPatientGroups()
);

The indexed text should reflect the actual questions people are going to ask, not just the record’s label. Name, description, services, specialties, patient categories, location, relevant terminology, synonyms, and known restrictions all belong in there if they’re the kind of thing a real query might reference.

Top-K and thresholds are two different knobs

Top-K controls how many candidates come back — Top-K = 5 returns the five closest records, full stop, whether or not any of them are actually good matches. A similarity threshold is the separate control that decides whether a candidate is relevant enough to return at all:

SearchRequest.query(query)
    .withTopK(5)
    .withSimilarityThreshold(0.75);

Skip the threshold and you can end up returning the least-bad answer from a bad set, dressed up to look like a real result. Top-K answers “how many.” Threshold answers “are these actually good enough.” You need both.

Debugging a bad semantic search result

Say a search for “pediatric malaria treatment” comes back with dental clinics. Work through it in this order rather than guessing:

Check what was actually indexed. Was only the clinic name embedded, with services and specialties left out? Was the source data stale, empty, or truncated before it got embedded?

Verify the query and documents share an embedding space. Don’t embed your documents with one model and your queries with a different, unrelated one — unless those specific models were built to be compatible with each other.

Look at the raw similarity scores, not just the ranking. What’s the top score? How much separation is there between candidates? Are the relevant records scoring meaningfully higher, or is everything clustered in a weak, undifferentiated band?

Apply a minimum threshold if you haven’t — weak matches shouldn’t be treated as relevant just because they’re the closest thing available.

Check the filters. Metadata filtering on country, city, service type, language, or facility status can be wrong or silently missing.

Reconsider the embedding model. A general-purpose model may not represent specialized medical vocabulary well enough. Options include a stronger general model, a domain-specific one, richer indexed descriptions, synonym expansion, or hybrid search.

Build a retrieval evaluation set. Query, expected relevant records, expected irrelevant records, required filters — then measure recall@K, precision@K, MRR, NDCG, threshold precision/recall, and how often the system correctly returns nothing when nothing relevant exists.

Hybrid search: keyword and vector aren’t competitors

Keyword search wins on exact clinic names, codes, product numbers, drug names, regulation numbers, and acronyms — anything where the literal string matters. Vector search wins on paraphrases, conceptual similarity, and natural-language intent. They’re complementary, not competing approaches, and most production retrieval systems combine both:

Final score = weighted keyword score + weighted vector similarity

A reranker can then reorder the initial candidate set with a more precise (and usually more expensive) model, applied only to the shortlist rather than the whole corpus.

Quick reference

Embedding — a dense vector representing meaning. Similar meaning, nearby vectors. Not reversible into the original text.

Semantic search — retrieve by meaning instead of exact wording, using vector similarity as the ranking signal.

Interview-ready: what is an embedding? A dense numerical vector representing the semantic properties of content. Similar meanings produce nearby vectors, which is what makes semantic search and retrieval possible in the first place.

Interview-ready: why does vector search return irrelevant results? A vector database returns its nearest candidates even when none of them are actually good matches. The usual causes are thin indexed content, a missing similarity threshold, wrong or absent metadata filters, a query and document set embedded with incompatible models, or a general-purpose embedding model that doesn’t represent the domain vocabulary well.

Part 4 builds on this directly — retrieval isn’t useful on its own, it’s the input to Retrieval-Augmented Generation, which is where semantic search actually earns its keep in a production system.