How AI customer retention actually works

A loyalty platform ships with a recommender. The team bolts a churn model on top. Sales calls it “AI-powered retention.” Six months later, repeat-visit rate is flat, merchants are blasting every customer with the same weekly deal, and the data team is asking why nobody opens the emails.

We rebuilt Ferry’s engagement stack from scratch. Repeat visits tripled on matched merchant cohorts in a single quarter. The interesting part isn’t the headline number — it’s that the system underneath looks almost nothing like what most loyalty platforms call “AI retention.”

A retention engine is a lifecycle agent, not a recommender. Once you frame it that way, most of the design decisions fall out by themselves.

TL;DR — A recommender answers “what should we show this user?” A retention engine answers “should we act on this user at all, and how?” The second question is harder, more valuable, and almost never the question loyalty platforms are set up to answer. Building it right means three pieces: a feature pipeline that scores churn risk and affinity, a next-best-offer agent that picks from a structured catalogue (not a chatty LLM), and a merchant-side control surface with hard guardrails. Get the framing right, and the model becomes the smallest part of the system.

Why “retention AI” usually means a recommender in a retention costume

Walk into most loyalty platforms and you’ll find the same architecture. A user-event stream feeds into a candidate generator — usually collaborative filtering or a two-tower model — that produces a ranked list of offers. The list gets filtered by “active promotion,” sorted by predicted click-through, and shipped to the app.

That system answers one question well: given this user is going to see something, what’s the best thing to show?

Retention is a different question: given this user’s lifecycle state, should we even reach out, and if so, what’s the smallest intervention that moves the right metric? A recommender treats every active session as an opportunity. A retention engine treats most sessions as moments to stay quiet.

The difference shows up in the data. On Ferry, when we instrumented the legacy system, the top decile of users by predicted CTR was also the decile with the highest opt-out rate from push notifications. The model was optimising for the wrong axis: it was maximising the chance of a click on the next message, which is the exact opposite of maximising lifetime value. A user who clicks three offers a week and churns in two months is a worse outcome than a user who ignores three offers and comes back next month for one they actually wanted.

This is the failure mode most loyalty platforms are stuck in, and it’s invisible until you separate the layers. The recommender is doing its job. The framing is wrong — and the reason it stays wrong is that nobody measures the long-term effect of a high-CTR offer on a user who’s already at the edge of opt-out.

Framing the problem: a lifecycle agent, not a recommender

The first decision on Ferry was to throw out the question “what should we show?” and replace it with a different one: per customer per day, what is the right action — stay quiet, surface an offer, or escalate to the merchant?

That reframe changes what the system has to compute. Instead of a single ranked list, we need three things:

1. Churn risk and affinity, refreshed often enough to act on. Not a score that updates weekly. Hourly. Retention breaks on day-boundaries — the user who was active yesterday and is silent today is a different intervention than the user who’s been silent for three weeks.
2. A decision about whether to act at all. Most candidate generators produce a list. A retention engine produces a decision tree with at least one branch that says “no message today, because intervening would teach this user to ignore us.”
3. A fallback path that doesn’t depend on the model. When the model is uncertain, the offer is bad, or the merchant’s budget is exhausted, the system needs a rule-based default that’s still better than silence. This is the part most teams skip because it feels like admitting the model isn’t enough. It’s actually the part that makes the whole system shippable.

The lifecycle-agent framing also gives you a clean answer to the question every product lead asks: why is this better than the rule we already had? The honest answer is that it isn’t, for most users most of the time. It is materially better for the long tail — the 5–10% of users in lifecycle states where the right action depends on history the rules can’t see.

The feature pipeline: signals, not features

The first piece we shipped wasn’t a model. It was a feature pipeline that turned raw events into the two scores the agent actually needs to make decisions: churn risk and offer affinity.

“Churn risk” sounds like a single number. It isn’t. On Ferry it decomposed into three signals, each computed from raw events and refreshed on its own cadence:

# Pseudocode — the actual pipeline runs hourly in Airflow, not real-time
def churn_signals(user_id: str, now: datetime) -> dict:
    return {
        # Cadence: how often has this user visited over the last 8 weeks?
        # Exponentially weighted so recent silence counts more.
        "cadence_drop": cadence_zscore(user_id, window_days=56),
        # Engagement: are they opening offers but not redeeming?
        # High open + low redeem = "trained to ignore us" — separate risk.
        "engagement_decay": open_to_redeem_ratio(user_id, window_days=28),
        # Cohort drift: has this user's behaviour diverged from
        # the cohort they were clustered with at signup?
        "cohort_drift": embedding_distance(user_id, signup_cohort=user.signup_cluster),
    }

The interesting bit is the middle signal. A user who opens every offer and never redeems is more of a churn risk than a user who doesn’t open anything. They’ve been trained by the legacy system to expect offers they don’t want, and the next push is the one that gets them to disable notifications. A single-number churn score misses this. Three signals composed by the downstream agent don’t.

cadence_drop is the early-warning signal. engagement_decay is the late-stage damage signal. cohort_drift catches the case where a user has outgrown the merchant they’re matched with — they used to fit, now they don’t, and the right intervention is a category change, not an offer.

The pipeline is refreshing on an hourly cadence because that’s the smallest window where the lifecycle state can change enough to matter. Faster than hourly, and you’re paying for features nobody can act on. Slower than daily, and you’ve missed the moment where the user was still reachable.

This part of the system is 80% of the engineering work and the smallest part of any demo. That’s normal, and it’s the inverse of what most teams prioritise.

The next-best-offer agent: structured tool calls, not a chatty LLM

The next piece is the part that gets the most attention, because it’s the part that uses an LLM. It’s also the part where the most teams go wrong by using the LLM as a chat endpoint.

A “next-best-offer” agent that asks an LLM to write an offer is a liability. The LLM will confidently invent deals that don’t exist, misprice discounts, and copy tone from offers the merchant disabled last week. The right shape is the LLM as a writer on top of a selector that has no language model in it at all.

Concretely:

# The agent is a loop: pick → write → check → send (or skip)
def decide(user_id: str, hour: datetime) -> Action:
    state = lifecycle_state(user_id, hour)  # from the feature pipeline
    if not state.should_intervene(hour):
        return Action.skip(reason=state.skip_reason)

    # The selector is a deterministic ranker over the merchant's active catalogue.
    # No LLM here. Just rules + the affinity scores.
    candidates = selector.rank(
        merchant_id=state.primary_merchant,
        affinity=state.affinity_scores,
        exclude=state.recently_shown(user_id, days=14),
    )
    if not candidates:
        return Action.skip(reason="no_eligible_offer")

    offer = candidates[0]

    # The LLM only runs at this step, and only as a templated writer.
    # Inputs are constrained. Outputs are constrained. There is no chat.
    copy = llm_writer(
        template="merchant_offer_v3",
        merchant_voice=offer.merchant.tone_profile,
        user_history=state.history_summary,  # pre-summarised, not raw events
        offer=offer.canonical_fields,         # strict schema
    )

    # Guardrails run before any send.
    if not passes_guardrails(copy, offer, state):
        return Action.fallback_template(offer)  # static copy, no LLM

    return Action.send(offer=offer, copy=copy)

Three details that matter:

The selector is not an LLM. It is a deterministic rank over the merchant’s active catalogue, scored against the affinity vector. If the merchant has no eligible offer for this user, we skip — and “skip” is a valid, observable action the agent takes hundreds of times more often than “send.” That’s the point.

The writer is scoped to a template, with a strict schema. Merchant voice, user history summary, and the offer fields go in; a short message comes out. The template is versioned. If a merchant wants different copy, the change is a template edit, not a prompt edit. This is how you avoid the “we tweaked the prompt and now nothing sends” failure mode.

Guardrails are explicit, not emergent. Per-user frequency caps, merchant-budget caps, a low-confidence fallback to a static template, a kill switch the merchant can pull from their dashboard. None of these are “we’ll add them later” — they’re the first thing wired up, because the moment the system can send a message on its own, it needs a way to stop. The reason: every guardrail you add after the first incident is a guardrail that learned its lesson from a user it shouldn’t have.

What changed in production

The headline number was the repeat-visit rate — three times the baseline on matched merchant cohorts, measured 90 days post-rollout against the prior 90-day baseline on the same merchants. That’s the number on the case study.

The numbers underneath that number are more interesting:

Message volume dropped while redemptions rose. The legacy system was sending roughly the same offer to every active user every week. The retention engine sent fewer messages — most users got none — and the ones it did send had materially higher redemption rates. That’s the shape of retention working: smaller, better-targeted interventions that don’t train users to tune out.

Merchant-side intervention became a real workflow. The retention engine escalates to the merchant dashboard when a cohort is at risk in a way that needs a human decision — a category change, a campaign approval, a budget reallocation. Merchants started using the dashboard as a daily tool, not a monthly report. That was the operational shift that let the AI system compound: the merchants were now part of the loop.

The agent became the default onboarding path. New merchants landing on the platform get the retention engine on day one. They don’t have to “turn on AI.” That’s the right default — the rule-based system is still there as a fallback, but the lifecycle agent is what the platform ships as.

The guardrails that mattered (and the ones that didn’t)

Six months in, the guardrails we’d assumed would be the load-bearing ones were not. The ones that mattered:

Per-user frequency caps, with hard ceilings. Once a day. Not “soft caps” the model could override. Hard ceilings. The moment you let the model break its own frequency cap because “this offer is really good,” you’ve trained the user to opt out — and that’s why hard caps beat soft caps in every A/B test we’ve run.

Merchant budget caps, visible to the merchant in real time. Merchants care about cost per redeemed offer. A dashboard that shows running spend against cap, with a one-tap pause, prevents the only conversation that goes badly: “we spent our whole Q3 budget in two weeks.”

A low-confidence fallback to static copy. When the writer’s confidence is below a threshold — measured by a small classifier over the LLM output, not by the LLM itself — the system ships a static template instead. No empty sends. No model-coasting-on-vibes. The fallback is visible in the merchant dashboard as a percentage of all sends.

The guardrails we’d expected to matter and didn’t: content safety filters, PII redaction (the structured inputs make PII leaks hard by construction), and brand-voice enforcement (the template system handled it). They got built and stay in the harness, but they’ve never been the failing part.

When this approach doesn’t fit

The lifecycle-agent framing isn’t universal. It works when:

• The user has repeated interactions over a long window (loyalty, subscription, marketplace).
• The cost of a bad intervention is meaningful (opt-out, churn, support ticket).
• There’s enough signal in the event stream to score lifecycle state hourly.

It doesn’t work when:

• The relationship is transactional and one-shot (a single purchase, a single support ticket).
• The intervention is high-stakes and irreversible (a financial action, a hiring decision — those want human-in-the-loop checkpoints, not silent automation).
• The catalogue is too sparse for the selector to have a meaningful candidate pool. If the merchant has two offers, “next-best-offer” is a coin flip.

We tell clients this up front. The retention engine is not the right shape for every engagement, and pretending otherwise is how you end up with a system that’s expensive to run and doesn’t move the metric it was sold on.

Actionable next steps for a similar engagement

The current shape of a retention engagement at Ankor looks like this:

1. Audit the signal. What events are already being captured, what lifecycle states they support, where the gaps are. Two weeks, written report, no code yet.
2. Build the feature pipeline first. Churn-risk and affinity scores refreshed hourly, with the three-signal decomposition above as the default. This is where the engineering effort goes.
3. Ship the selector and the writer as separate services. The selector is a deterministic ranker with no LLM in it. The writer is a templated LLM call with strict schemas and a low-confidence fallback. Different reliability profiles, different monitoring, different SLOs.
4. Wire the guardrails before the first production send. Frequency caps, budget caps, fallback path, merchant kill switch, observability traces for every action — including “skip.”
5. Treat the merchant dashboard as a first-class product. The retention engine is only useful if the merchant can see what it’s doing, override it, and trust it. The dashboard is not an afterthought.
6. Run shadow mode for at least two weeks before any send goes live. The agent decides; the legacy system sends. Compare redemption, opt-out, and complaint rates side by side. Cut over only when the new system wins on the metrics the merchant cares about.

That sequence is what produced the Ferry case study. It’s also what we’d build on a blank slate for any client whose retention problem looks like that shape — which, in our experience, is most consumer-facing businesses with a repeat-visit model.

If you’re sitting on event data and your retention number isn’t moving, the issue is almost never the model. It’s the framing. Decide whether you’re building a recommender or a retention engine, build the feature pipeline first, and put the LLM where it earns its place — as a writer on a structured selector, not as a chat endpoint trying to be clever. The metric will follow.

Want to see this shape applied to your retention problem? Start with an AI readiness assessment — we’ll tell you in two weeks whether the data you have is enough to build the engine, or whether you need to fix the data first.