How recommendation engines actually scale (lessons from Compass)
Two-stage retrieval, feature weighting, and the latency crossover at 10M items. The architecture we use for recommendation engines that scale beyond a demo.
by Ankor
Most recommendation engines we’ve audited fail the same way: the team picks a model before they pick a data shape, builds a single monolithic retrieval step, and wonders six weeks later why relevance is bad, latency is worse, and every tweak breaks something else. The system looks like a recommendation engine. It behaves like a fragile demo that nobody trusts with a metric.
We’ve shipped recsys on a couple of dozen products — career matching for Compass, next-best-offer for Ferry, content recommendations across a handful of media clients. The architecture we keep arriving at is the same. It’s also not the one most blog posts describe. This is what we build, why we build it that way, and where it breaks at scale.
The shape that keeps working
Two stages, not one. Always.
┌────────────────────────────────┐ ┌──────────────────┐
│ STAGE 1 — Candidate generation│ ─────────────► │ STAGE 2 — Ranking │
│ (recall-optimised, fast, dumb)│ shortlist │ (precision-tuned,│
│ feature score + vector recall │ (typically │ feature-weighted│
│ + tag / metadata filter │ 100–500) │ + vector rerank │
└────────────────────────────────┘ │ + LLM rationale │
└──────────────────┘Stage one is cheap and forgiving. Its job is to return a few hundred plausible candidates as fast as possible, with high recall — you’d rather over-include a few wrong items than miss the right one. Stage two is expensive and strict. Its job is to pick the right handful from the shortlist, with high precision, and explain why.
If you collapse the two stages into a single neural ranker, you get one of three failure modes: latency you can’t ship (because you’re running the heavy model on the full corpus), cold-start you can’t recover from (because the model needs history you don’t have), or an editorial loop you can’t intervene in (because the model’s reasoning lives in weights, not features). Same lesson we wrote about in harness engineering: the model is the easy part. The shape around it is the actual product.
Why feature weighting usually beats raw embedding similarity
The instinct in 2025–2026 is to embed everything and call it done. pgvector plus cosine similarity produces a recommendation. It will even be a plausible recommendation. It will also be the wrong shape for most product use cases, for three reasons.
First, embeddings compress the kind of an item but not its value to this user. “Article about Postgres indexing” and “article about MySQL indexing” have near-identical embeddings. They are not interchangeable — the user’s history decides which is the right read next, and that history lives in a structured feature vector, not a single query embedding.
Second, plain cosine similarity is non-editable. When the editorial team says “downweight paid content,” or “boost newer items,” or “respect inventory caps,” you can’t express those as a vector math operation. You can express them as 0.3 × recency + 0.6 × affinity + 0.1 × editorial_boost.
Third, the feature vector is your evaluation surface. The same logic you use to score a recommendation is the logic you use to debug why a user saw something they didn’t want. A dense embedding offers no such entry point. We covered this discipline more broadly for RAG systems — the same principle applies: retrieval quality lives in the feature layer, not the embedding table.
The pattern that holds up: compute a handful of features per item per request (recency, category affinity, prior engagement, editorial boost, price tier), pass them through a weighted scorer to produce a shortlist, then use the embedding as a rerank signal if you want one. The embedding becomes one of several signals, not the only one.
A concrete shape, from Compass
Compass is a career-counselling platform where students run structured assessments, try on professions experientially, and expect a personalised recommendation report at the end. The naive build would have been a single LLM prompt: “given this student’s answers, recommend careers.” That is a recommendation that sounds competent and reads as generic — and it gives the counsellor nothing useful to edit, which was the actual product problem.
Here’s the shape we shipped instead, on a stack we’d use again:
# Stage 1 — deterministic shortlist, pgvector + features
shortlist = (
db.query(CareerProfile)
.filter(features.applied == True) # structured filter first
.filter(career.aptitudes ∩ student.aptitudes) # feature overlap
.order_by(score_compatibility.desc())
.limit(50)
.to_list()
)
# Rerank with vector similarity over experiential notes
ranked = rerank(
shortlist,
query_embedding=embed(student.experiential_summary),
top_k=10,
)
# Stage 2 — LLM writes rationale only, never decides the ranking
rationale = llm.generate(
prompt=build_prompt(student=student, ranked=ranked),
tools=[], # no tool calls, no ranking influence
guardrails={
"must_cite_aptitude_match": True,
"may_not_invent_pathway": True,
"max_words": 180,
},
)Three things to notice. Stage one is fully deterministic and testable — you can write eval cases that say “given this student vector, this shortlist should appear.” Stage two is the only place an LLM is involved, and its output is text attached to a ranked list, not a ranking in itself. The counsellor is still the editor of record, and the rationale the LLM produces is grounded in the ranked candidates it was handed — there’s no path by which the model invents a career that didn’t make the shortlist. This is the same “LLM as explainer, not as judge” pattern we used in proctoring pipelines, and for the same reason: you want a human in the loop, and you want the loop to be audit-able.
The result, in production numbers from the Compass case study: counsellor turnaround dropped from a multi-day report-writing cycle to minutes of editing on a draft. Volume per counsellor multiplied. Editorial control stayed with the counsellor. That’s the whole win condition, and it’s the one a single-LLM build never reaches.
The latency crossover at ten million items
The most under-discussed number in recommendation architecture is the corpus size where your choice of database stops being a preference and starts being a constraint. Based on our experience across engagements:
| Corpus size | What holds up | What breaks |
|---|---|---|
| Under 100K | PostgreSQL with pgvector + features, no reranker | Almost nothing |
| 100K–1M | pgvector HNSW index + feature filter, single LLM rerank | Embedding-only retrieval, no features |
| 1M–10M | pgvector with sharding, or Qdrant/Weaviate, pre-filtered shortlist | A naive “embed everything, no filter” query |
| 10M+ | Dedicated vector store (Pinecone, Vespa, Milvus) + a separate feature store | Single-DB attempts at sub-300ms P95 |
| 100M+ | Two-tier index (summarised partitions → fine-grained), learned rerankers | Everything that isn’t a real recsys team |
The row that matters most is the second one. For the 100K–1M range — which is most early-stage products, including Compass on launch — you do not need a dedicated vector database, and adopting one is usually a step backwards. It splits the storage, adds an operational surface, and forces you to keep two sources of metadata in sync. PostgreSQL with pgvector and HNSW handles this range comfortably at sub-100ms P95 for shortlist generation, and you already know how to operate it. The instinct to reach for a managed vector store is, for most teams, premature.
The crossover at 10M is real. Once you cross it, you stop pretending one database does both jobs well — you split feature storage (your application database) from vector storage (a dedicated index), and you treat them as separate systems with their own backup and access-control stories.
Where LLM-augmented recommendation earns its place
LLMs in recsys earn their place in three specific spots, and lose money in the others.
Spot one: rationale generation. Show me a ranked list of ten careers; tell me why they fit the user. The LLM does text well, and this is a text job. Compass ships this. So do most of our media clients. Constrain the prompt to the ranked list, ground every claim in features from the ranker, and you get trustworthy explanations.
Spot two: cold-start from declarative signal. A new user with no click history, but a profile that says “I like back-end work, static-typed languages, slow-paced teams.” An LLM can turn that sentence into a feature vector you can score against — useful when you have less behavioural data than declarative data. We use this for first-session personalisation on Ferry, where a brand-new customer has rich preference text but no purchase history.
Spot three: editorial queries. “Find me products to feature this week that aren’t alcoholic drinks, suit a price band under $30, and have been in inventory more than 30 days.” A natural-language query plus an LLM that emits a structured filter is more useful to a merchandising team than a SQL prompt they have to write themselves. This is the same mechanism as the natural-language interface layer we ship for ops tooling — narrow, scoped, structured outputs.
The spots where LLMs don’t earn their place: as the final ranker in place of a weighted feature scorer (slower, less editable, less debuggable), and as the cold-start replacement when behavioural data exists (just use the features — the LLM is unneeded latency). A recsys that leans on an LLM for ranking rather than for explanation will be slower, more expensive, and worse than one that doesn’t.
The failure modes that will hit you anyway
Three failure modes show up across most recommendation builds we’ve audited, regardless of stack.
The popularity tax. A recommender trained on engagement data quickly learns that the most-engaged-with items are the most popular items, and recommends them more, which makes them more popular. Within weeks you’ve built a popularity amplifier that crowds out new, niche, or editorially-important content. The fix is to hold out a freshness slot in every recommendation — one item the ranker would otherwise exclude, rotated in by recency or editorial rule. The harness catches this only if your eval set explicitly contains long-tail items as positive examples.
The cold-start cliff. The first hundred users on a new feature see bad recommendations because the ranker has no feedback for them, then disengage, so the ranker never gets the feedback it needed. This is the same silent failure pattern we see in agents: the system looks fine on dashboards while the actual quality is collapsing because the operators have stopped believing it. The fix is a separate evaluation track for new users, with a deliberate baseline (“new users see the top-N most-popular items, period, until we have data”) instead of the broken model. Don’t grade the cold-start strategy on the same metric as the warm strategy.
The feedback loop going the wrong way. User clicks on recommendation → that item is shown more → more clicks → more show. With retraining on click data, you can write yourself into a corner where the ranker is permanently biased toward a historical artefact. We covered the discipline for catching this in harness engineering: the eval set is built from your failure cases, not generic benchmarks, and it runs on every change. Without that rig, you don’t notice the loop until the editorial team is in your inbox asking why their new product line never gets shown.
What to do next
If you’re building a recommendation engine, or rescuing one that stalls:
- Split candidate generation from ranking. Two stages. Stage one is fast and recall-optimised — features and vector recall together. Stage two is precise and reorders the shortlist, with the LLM only writing rationale. Anything else is harder to debug and slower to ship.
- Make the feature layer the source of truth. Whatever signals the ranker uses (recency, affinity, popularity, editorial boost, price tier), persist them as named features you can inspect, edit, and weight. Embeddings are a rerank signal, not the only signal.
- Pick the storage tier by corpus size. Under 1M items, PostgreSQL with pgvector and HNSW is correct and boring. Reserve the dedicated vector stores for when the data forces the decision, not when a vendor blog does.
- Constrain the LLM to text, never ranking. Generate rationale, fill editorial queries, parse declarative input into features. Do not let the LLM produce the final ranked list.
- Carry a freshness slot. Reserve at least one slot in every recommendation for an item the ranker would otherwise exclude. Without this, you are running a popularity amplifier.
- Eval the cold-start path separately. A/B-test new-user recommendations against a curated baseline until you have evidence the model is helping. Until then, ship the baseline and save the GPU time.
The through-line is the same one we keep coming back to. Recommendation engines that ship reliably are not the ones with the cleverest model. They’re the ones with two clear stages, a feature layer you can read, an LLM scoped to text, and an evaluation harness that catches regressions before the user does. That’s the architecture we keep arriving at — and the one that’s worth building first.
Ankor builds production-grade AI systems for products that can’t afford to demo well and fail in production — including the Compass career recommendations engine and the Ferry retention stack. If you’re evaluating a recommendation build and want a second pair of eyes on the architecture or the data shape, book a 30-minute call or write to us at team@ankor.us.
