What is a vector database?

What is a vector database?

A vector database stores high-dimensional numeric arrays and answers one question extremely well: given this vector, which stored vectors are closest to it? That is a different question from anything a relational database is built for. There is no equality to test and no range to scan - just a notion of distance in a space with hundreds or thousands of dimensions.

The reason anyone cares is that those coordinates can encode meaning. Run text through an embedding model and semantically similar text lands near each other in the space, whether or not it shares any words. Nearest-neighbour search over embeddings is therefore search by meaning, which is the retrieval half of every retrieval-augmented generation system, every semantic search box, and most recommendation engines built in the last few years.

Any database can store a list of floats. What makes a vector database one is that it indexes them for approximate nearest-neighbour search.- the actual dividing line

Hold on to that distinction, because it decides the build-versus-buy question in section 07. Storing vectors is trivial. Finding the nearest ones among fifty million of them, in ten milliseconds, is the entire product.

Embeddings, briefly

An embedding is a fixed-length array of floating-point numbers produced by a model from some input - a sentence, a paragraph, an image, a product description. Common text embeddings run from 384 to 3,072 dimensions. The model is trained so that inputs humans would call similar end up close together and unrelated inputs end up far apart.

You do not interpret the individual numbers. Dimension 412 does not mean "formality" or anything else nameable. The only thing the array is good for is comparison with other arrays from the same model.

That last clause is a hard constraint and the source of a genuinely nasty class of bug. Two embeddings from different models, or from different versions of the same model, occupy different spaces and their distances are meaningless. A corpus embedded with one model and queried with another does not error - it returns confident, plausible, wrong results. Store the model name and version alongside every vector you persist, and treat a model change as a full re-embedding migration.

For what the models themselves are doing, the LLM article in the AI path is the companion piece. For this article the model is a black box that turns content into coordinates.

Why exact search does not scale

The obvious implementation of nearest-neighbour search is to compare the query against everything. For each stored vector, compute the distance; keep the best k. This is exact k-NN, it is trivially correct, and it is a full scan.

The arithmetic gets uncomfortable quickly. One million vectors at 1,536 dimensions is about 1.5 billion multiply-add operations per query, over roughly 6 GB of raw float data. Modern CPUs are fast and this is embarrassingly parallel, so a well-optimised exact search over a million vectors is genuinely feasible - it is the ten-million-vector corpus at fifty queries per second where it collapses.

So production systems use approximate nearest neighbour search. ANN indexes give up the guarantee of finding the true top k in exchange for enormous speedups, and the quality of that trade is measured as recall: of the true top 10, how many did the index actually return? A well-tuned index runs at 95 to 99% recall while touching a fraction of a percent of the data.

Being comfortable with approximation is the mental shift. Nobody accepts a relational index that returns 97% of matching rows. Here it is not only acceptable but correct, because the ranking is already a heuristic about meaning - the difference between the 9th and 11th nearest chunk is almost never the difference between a good answer and a bad one.

HNSW, IVF, and the knobs

Two index families cover the great majority of deployments.

HNSW

Hierarchical Navigable Small World builds a layered proximity graph. Each vector is a node connected to some of its neighbours; upper layers are sparse and act as express lanes, lower layers are dense and local. A search enters at the top, greedily walks toward the query, drops a layer, and repeats - a few hundred distance computations instead of millions.

It is the default choice in most systems because its recall-versus-latency curve is excellent. The costs are real: the graph lives in memory and carries significant per-vector overhead beyond the vectors themselves, and building it is slow because every insert searches the existing graph to find its neighbours.

IVF

Inverted File indexes cluster the space first. A training pass computes centroids; each vector is assigned to its nearest one. At query time you compare against the centroids, pick the nearest few clusters, and search only inside those. Cheaper in memory than HNSW and much faster to build, at the price of a training step that needs data representative of what you will store, and a sharper recall cliff when a true neighbour sits just across a cluster boundary.

Both are frequently combined with quantization - compressing each vector into a much smaller approximate form so more of the index fits in memory, at some further cost to recall.

The knobs are always the same shape

Whatever the system, you get one parameter set at build time that trades index size and build time for quality, and one set at query time that trades latency for recall:

• HNSW build: m (connections per node) and ef_construction (how hard to search while inserting). Higher means a better graph, a bigger index, and a slower build.
• HNSW query: ef_search (how wide to explore). The single dial you turn when recall is too low, and the one that costs latency.
• IVF build: nlist (number of clusters). IVF query: nprobe (clusters to search).

Tune these against a labelled evaluation set, not by feel. Measure recall@k on queries you know the right answers to, then raise the query-time parameter until recall is where you need it and latency is still acceptable. Without that measurement you have no idea whether your retrieval is at 98% or 60%, and both feel about the same in a demo.

Distance metrics

"Closest" needs a definition, and there are three in common use:

• Cosine similarity measures the angle between two vectors and ignores their magnitude. This is the standard choice for text embeddings, because what you care about is direction in the space, not how long the arrow is.
• Dot product multiplies the components and sums. For vectors that have been normalised to unit length it ranks identically to cosine and is cheaper to compute, which is why many systems normalise on write and then use dot product.
• Euclidean distance (L2) is ordinary straight-line distance. Common for image and audio embeddings where magnitude carries information.

Use whatever the embedding model was trained and documented for. This is not a tuning parameter to experiment with - a mismatched metric does not fail loudly, it just degrades your ranking quietly and permanently. Check the model card, set the metric once, and make sure the index and the query agree on it.

Filters and hybrid search

Two things get skipped in most introductions and then dominate the actual work.

Metadata filtering

Real queries are almost never pure similarity. They are "nearest, but only in tenant 42, only documents this user may read, and only from the last year". How the system combines the filter with the ANN search matters enormously:

• Post-filtering retrieves the top k by distance and then discards what fails the filter. Simple, and it silently returns three results when you asked for ten - or zero, if the tenant's documents are not in the global top k at all.
• Pre-filtering restricts the candidate set first. Correct, but a naive implementation defeats the index, since ANN structures are built over the whole collection.
• Filtered search pushes the predicate into the graph or cluster traversal itself. This is what mature systems do, and it is one of the clearest differentiators between them.

If your workload is multi-tenant, ask this question of any candidate system before anything about raw QPS. Tenant isolation with correct result counts is a much harder requirement than speed.

Hybrid search

Pure vector search is genuinely bad at some things: exact identifiers, product codes, surnames, error numbers, and any rare token whose whole value is that it matches literally. Search for ORA-01555 and semantic similarity will happily return five documents about other Oracle errors.

The standard answer is hybrid search: run a keyword query (BM25) and a vector query, then merge the two ranked lists, usually with reciprocal rank fusion. Keyword search handles precision on literal tokens, vectors handle recall on paraphrase, and the combination beats either alone on almost every realistic corpus. If you are building retrieval for a product, plan for hybrid from the start rather than discovering it after the first support ticket about a part number.

Do you need a separate one?

Often not, and this is where a lot of unnecessary infrastructure gets adopted. The honest decision tree:

• You already run PostgreSQL, and you have under a few million vectors. Use pgvector. It supports HNSW and IVFFlat, it filters using the same WHERE clause as everything else, and your vectors stay transactionally consistent with the rows they describe. This covers a large majority of real applications.
• You already run SQL Server or Azure SQL. Recent versions have native vector storage and distance functions, which is enough for moderate corpora and keeps everything in one database with one backup and one security model.
• You need keyword and vector search together, with rich filtering. A search engine is the better fit - Azure AI Search, Elasticsearch, or OpenSearch all do hybrid retrieval as a first-class feature rather than something you assemble.
• You have tens of millions of vectors, high query rates, or hard latency targets. Now a dedicated store earns its place: Qdrant, Weaviate, Milvus, Pinecone, or Cosmos DB's vector support. What you are buying is quantization, sharding, replication, and tuning controls that a general-purpose database does not expose.

The strongest argument for starting in the database you already run is not performance, it is consistency. Two datastores means keeping them in sync: a document updated in Postgres and not re-embedded in the vector store leaves your retrieval quietly serving stale content. Inside one database that is a transaction. Across two it is a distributed systems problem you now own, complete with backfills and reconciliation jobs.

Start with what you have. Move to a dedicated store when you have measured a reason to.

Getting it right in production

• Chunking matters more than the index. How you split documents before embedding sets the ceiling on retrieval quality. Chunks that are too large dilute the embedding with unrelated content; too small and they lose the context that made them meaningful. Overlap between chunks, and keep enough metadata on each to reconstruct where it came from.
• Store the model and version with every vector. Non-negotiable. It is the only thing standing between you and silently mixing two incompatible spaces.
• Budget for re-embedding. Changing the model means recomputing every vector in the corpus. For a large collection that is real money and real time, and it needs a plan for serving traffic during the transition - usually a parallel index and an atomic switch.
• Evaluate with a labelled set. Fifty to a hundred real queries with known-good answers, scored as recall@k, run in CI. Without it, every retrieval change is a guess, and quality regressions are invisible until a user reports one.
• Consider dimensionality deliberately. Cost, memory, and latency all scale with it. Some modern models are trained so their embeddings can be truncated to a shorter prefix with modest quality loss, which is a cheap and effective lever.
• Plan filtering and hybrid search up front. Both are difficult to retrofit and both turn out to be requirements far more often than a first prototype suggests.

A vector database is a narrow, well-understood tool: it makes approximate nearest-neighbour search fast and operable. It does not understand your documents, it does not fix bad chunking, and it will not tell you when your recall has quietly fallen to 60%. Treat it the way you would treat any index - know what it is doing, measure whether it is working, and keep it honest about the data it describes.