← writing
7 December 2020

Searching by resemblance

In 2020, comparing documents by their vectors meant assembling the pipeline yourself. The state of the art at the time, and what vector databases have changed since.

Keyword search finds what you can name. Search by similarity finds what you can only recognise, and the operation is different. The word is replaced by a vector, and the question “which one contains this term?” by “which one is closest?”. Everything that came after, including what is now called a vector database, follows from that substitution.

01The state of the art in 2020

The pipeline had three parts, all of them necessary. First the vectors: you took a network trained for classification, removed its last layer and kept the rest, a few hundred numbers per document. Then an approximate index, such as Annoy, published by Spotify, or FAISS, published by Facebook, because comparing a million vectors one by one is too slow for an interactive query. Finally storage, the least developed part: usually a conventional database placed next to the index, which had to be kept consistent with it.

02The trade accepted

The word approximate names the central trade-off of these indexes. They accept sometimes missing the nearest neighbour in exchange for an answer a hundred times faster. You set the balance between exactness and speed, measure the loss on a sample whose answer is known, then decide. That trade-off has not changed, and it is the only passage of my 2020 notes I would rewrite unchanged.

03What vector databases changed

The mathematics has not changed: the same families of index, often with the same libraries. Vector databases changed how it is operated. Index and data sit in one system, a write is visible in both, you can filter on metadata while searching by distance, and backup covers everything. The progress is in engineering more than in science, and it fills the part that was missing in 2020.

04What improved most: the vectors

The subject became central because of the vectors more than the databases. The embeddings of 2020 were specialised: one model for images, one model per language for text, and matching two languages required a dedicated model. Since then a single space holds text in several languages and sometimes other media, with enough quality that similarity search is now used to feed a language model. The part that improved most is the one least discussed.

05Current products, and the threshold

I do not compare current products, which change too fast for a dated article to stay useful. Nor do I recommend a vector database below a few tens of thousands of vectors: an exhaustive in-memory comparison answers faster than a service to administer, and takes an afternoon to write. That threshold matters more than the choice of product, and it is the question asked least often.