Design the retrieval store for 8,000 chunks and 200 queries a day, and name the corpus size and query rate at which a dedicated index earns its place.
designmediumExercise · 3 min · 37 of 77
On this page
A proposal to deploy a managed vector database for internal documentation is on the table. The design question is what the retrieval store has to do, and the arithmetic says how little that is.
A 40-engineer company keeps about 3,000 pages of engineering documentation in a wiki, which the pipeline has split into roughly 8,000 chunks of 400 tokens, embedded at 1,024 dimensions with the same model used for queries. Engineers will ask about 200 questions a day. Around 50 pages change in a typical week. The existing stack is a Postgres database and a Python service with a few hundred MB of free memory. The proposal is a three-node managed vector database cluster, chosen "so retrieval scales". Nobody has written down what it needs to scale to.
Constants: a 1,024-dimension float32 vector is 4 KB; a round trip inside a datacenter is about 0.5 ms; a single core does on the order of a few billion multiply-adds a second and streams memory at about 10 GB/s.
Design the retrieval store — where the vectors and the keyword index live, how a query runs, how changes get in — and state the corpus size or query rate at which the design should change.
Count the work first. 8,000 × 1,024 ≈ 8M multiply-adds a query: about 3 ms on one core at a few billion a second. Memory: 8,000 × 4 KB ≈ 33 MB, which fits the spare RAM and streams in about the same 3 ms. So brute force in process, no index: one matrix loaded at start-up and normalised once; a query is a matrix-vector product and a top-100 heap. A managed cluster could save at most those 3 ms, invisible beside a model call of seconds, and adds a 0.5 ms hop and a service to run.
The store of record is the existing Postgres: a chunks table with text, heading path, document id, position, updated-at and the vector as a binary column, plus Postgres full-text search for the keyword side. A query runs full-text and the in-memory dot product in parallel, fuses the two top-100 lists by reciprocal rank, and returns 50 to the reranker. Changes: 50 pages a week ≈ 130 chunks ≈ 52k tokens, re-embedded nightly for the chunks whose hash changed; the service then reloads its matrix.
Put the retriever behind an interface, search(query, k), so an index can replace brute force without touching the prompt code. Acceptance test: recall at 5 on 100 labelled questions, re-run after every chunking or embedding-model change.
The design changes at about a million vectors (4 GB, roughly 0.4 s a query on one core) or at a few hundred queries a second, where 3 ms a query fills a core. At 8,000 chunks and 200 queries a day, the first is a hundred times away and the second a hundred thousand. Until then the cluster is a cost with no number behind it.
"Use the managed cluster; vector search is what it is for." No number for the corpus or the query rate. 8M multiply-adds a query is about 3 ms in process, and the cluster adds a network hop and an operating burden to save milliseconds nobody sees beside the model call. The reviewer wanted the arithmetic that was skipped.
"Store the vectors in Postgres with a vector extension and an approximate index." Closer, but an approximate index over 8,000 vectors trades recall for a speed-up nobody needs; the exact scan is already about 3 ms. Keep the table, skip the index, measure the recall the extension would have cost.
"Brute force in memory and we are done." Half the design: no keyword side, so an exact function name or error code in a question goes to a vector search that blurs it; no re-embed path, so the weekly 50 pages drift out of date; no interface, so the day the corpus does outgrow one machine the prompt code is rewritten too.
