Recommendation system readiness: seven decisions before building a model

Prove that the data represents the decision

Recommendation data usually combines people or contexts, items and interactions. Availability is not the same as fitness. An interaction may show exposure rather than preference: a click can reflect prominent placement, and no click can mean the item was never seen. Historical data also reflects past catalogue, policy and interface choices.

Create a data map that identifies purpose, source, owner, legal basis, update frequency, retention, quality checks and known bias for every feature and label. Measure catalogue coverage and interaction sparsity. Determine how new users and new items receive a useful result before enough history exists. A cold-start policy is a product requirement, not an edge case for the modelling team.

Inspect the item side as carefully as user data. Missing categories, inconsistent metadata and delayed availability can make good items invisible. Agree on catalogue quality rules and a correction process with the teams that create or maintain the items; the model cannot reliably compensate for an unmanaged catalogue.

Minimise data rather than collecting every possible signal. Sensitive or proxy attributes can create risks even when they improve an offline metric. Review whether the same value can be achieved with less personal data, shorter retention or aggregation. Document what the model must not infer or optimise.

• Users or contexts, items and interactions have clear owners.
• Exposure bias, missing data and historical policy effects are understood.
• New-user and new-item behaviour is designed explicitly.
• Data use has a defined purpose, legal basis and retention period.
• Sensitive attributes and plausible proxies receive a risk review.

Define value and guardrails as separate measures

Optimising one engagement metric can create a system that is accurate but not useful. Define a primary product outcome, such as completed discovery or successful task progression, and guardrails for diversity, coverage, novelty, user control, complaints, latency and business constraints. State the time horizon: immediate clicks can conflict with long-term satisfaction.

Offline evaluation tests whether a model can rank held-out data under historical conditions. It is fast and repeatable but cannot prove causal value in the live product. Online evaluation tests behaviour in context but needs safe experiment design, sufficient traffic and pre-agreed stop conditions. Qualitative research reveals why people accept, ignore or distrust recommendations.

Use all three forms of evidence. Compare against simple baselines such as popularity, recency, editorial rules or a non-personalised ranking. A more complex model should earn its operating cost by producing a meaningful improvement without breaking guardrails.

Segment results before accepting an average. New and returning users, sparse and active categories, or different device contexts may experience the ranking differently. Choose segments because they relate to product risk, not because repeated slicing happens to produce a favourable number.

Design a recommendation service, not a notebook

A model only creates value when the surrounding service can deliver fresh candidates, apply policy, respond within the product's latency budget and recover from failure. Define candidate generation, ranking, filtering, fallback, caching and explanation responsibilities. Decide which rules must remain deterministic outside the model.

Build traceability from a displayed recommendation to the model version, feature data, candidate set and policy filters that produced it. Monitor input drift, catalogue changes, score distributions, coverage, latency and product outcomes. Observability should let the team distinguish a model issue from stale data, a broken feature pipeline or a product integration defect.

Plan degradation before launch. When features are late, the model service is unavailable or a safety threshold is breached, return a tested baseline rather than an empty or unpredictable experience. Assign operational ownership and define incident response, rollback and model retirement.

Control training and release separately. A successful pipeline run should not automatically make a model the live decision-maker. Require evaluation evidence, versioned approval and a progressive rollout appropriate to the risk. Preserve the previous model and baseline until rollback has been tested.

Run a bounded pilot that can answer a decision

A useful pilot is not a miniature production platform. It is the least expensive experiment that resolves the largest uncertainty. If the risk is data coverage, build a dataset audit and baseline. If the risk is user trust, prototype explanations and controls with users. If the risk is online value, integrate one surface with a controlled experiment and a safe fallback.

Set the decision before starting: scale, revise or stop. Give each outcome evidence thresholds and include guardrails. Preserve experiment definitions, data windows, code, model artefacts and results so the conclusion can be reviewed. Avoid presenting an offline accuracy improvement as a production business case.

A senior data scientist can accelerate problem framing, evaluation design and the path from Python or SQL analysis to a governed cloud service. The organisation still needs product, data, engineering, legal and operational owners. Recommendation quality is a cross-functional product responsibility, not a model handed over at the end.

Include a stop option in the business case. If the baseline performs well enough, data risk remains high or user research rejects the experience, stopping can be the responsible result. The pilot has created value when it enables a sound decision, even if that decision is not to scale machine learning.

Seven decisions at the readiness gate

Answer these questions before committing to a full recommendation platform. A negative answer can lead to discovery work or a simpler product solution. It should not be hidden inside a model-development estimate.

• Which user choice and product surface will improve?
• What simple baseline must the system beat?
• Does available data represent preference rather than only exposure?
• How are cold start, privacy and sensitive proxies handled?
• Which value metric and guardrails decide success?
• Can every recommendation be observed, explained and safely replaced by a fallback?
• Who owns product, data, model risk and production operation after launch?