Generative Recommenders: Foundations, Engineering, and a Path to Production
Generative Recommenders: Foundations, Engineering, and a Path to Production
How Hierarchical Sequential Transduction Units (HSTU) turn recommendation into sequential transduction — trillion-parameter models, 15x faster training, up to 285x more FLOPs served at similar latency — and what it takes to actually run this in production.
Most recommendation systems in production today are Deep Learning Recommendation Models — DLRMs. They are the workhorse behind a decade of progress: YouTube DNN, Wide&Deep, DIN, DCN, DHEN. And they share a quiet, structural problem: they scale with data, not with compute. Throw more GPUs at a DLRM and, past a point, nothing happens. Quality plateaus.
The paper behind this post proposes a different starting point:
- Authors
- Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, Yu Shi (Meta)
- Venue
- ICML 2024 (PMLR 235)
- Posted
- 27 Feb 2024, arXiv:2402.17152
- Code
- github.com/facebookresearch/generative-recommenders
The authors' central claim is that most industry DLRMs stop improving as more compute is added, despite being trained on huge feature sets and enormous volumes of data — and that Transformer-style architectures, which do keep improving with compute in language and vision, have been held back in recommendation by a few specific design choices rather than anything fundamental. Their fix is to stop treating recommendation as feature engineering and start treating it as sequential transduction: reformulate ranking and retrieval as one generative modeling problem over an interleaved stream of content and actions, and design a purpose-built architecture — HSTU — for the high-cardinality, non-stationary, streaming nature of recommendation data. The headline result is a recommender shown, for the first time, to sit on the same compute-scaling curve as GPT-3 and LLaMA-2.
This post follows that claim down three threads: the foundations — why the reframing works and what the architecture actually computes; the engineering — how the paper's ideas become a production codebase and the fused kernels that make it fast; and, as a concrete test of whether any of this holds up outside a research paper, an industry case study — how it would map onto a real rewarded-advertising ranking problem at a company I will refer to here as X, and what it would take to actually build and ship that on GCP.
Notation
A few symbols recur throughout — worth having in one place before the math starts.
| Symbol | Meaning |
|---|---|
| Φᵢ | Content token at sequence position i — the item, ad, or piece of content shown |
| aᵢ | Action token at position i — the engagement (click, install, like…) or ∅ for no engagement |
| uᵢ | User state after position i — the sequence encoded up to that point |
| p(·) | A probability distribution the model is trained to predict |
| N, L, n | Sequence length (used interchangeably across the paper's sections — N in complexity bounds, L for an input tensor's length dimension, n inside the M-FALCON formulas) |
| d | Embedding / hidden dimension |
| X | The input tensor to one STU layer, shape L×D. Unrelated to X, the pseudonym used later for the ad-tech company case study — same letter, different meaning, kept apart by context |
| U, V, Q, K | The gate, value, query, and key tensors produced by HSTU's fused pointwise projection |
| ϕ₁, ϕ₂, f₁, f₂ | Pointwise nonlinearities and transformations inside one HSTU block (ϕ₂ is SiLU-normalized attention, replacing softmax) |
| A(X) | The attention-weight matrix computed from Q(X) and K(X) |
| rabp,t | Relative attention bias — positional (p) and timestamp-bucket (t) — added before the attention nonlinearity |
| W_uvqk, W_out | The fused U/V/Q/K projection matrix and the output projection matrix |
| α | Stochastic Length's sparsity exponent, α ∈ (1,2] — higher α samples more aggressively |
| b, m | Batch size and candidate-set size in M-FALCON's serving-cost formulas |
| R | Number of sampled negatives in the sampled-softmax training loss (R=128 in production) |
Glossary of terms and abbreviations
Acronyms and industry jargon used throughout this post, defined on first reference and collected here for quick lookup.
| Term | Definition |
|---|---|
| HSTU | Hierarchical Sequential Transduction Units — the architecture this post is about |
| CUDA | NVIDIA's low-level programming platform for writing code that runs directly on the GPU |
| Triton | A higher-level, Python-embedded language that compiles to efficient GPU code, used for most of the custom kernels in this codebase |
| GEMM | General Matrix Multiply — the basic matrix-multiplication operation GPUs are optimized to run at high throughput |
| SRAM | The GPU's small, fast on-chip memory — kernel fusion keeps data here instead of round-tripping to slower main memory |
| HBM / DRAM | The GPU's larger, slower main memory — moving data to and from it is usually the real bottleneck, not the arithmetic itself |
| DLRM | Deep Learning Recommendation Model — the incumbent industry approach HSTU is compared against |
| GR | Generative Recommender(s) — the modeling framework HSTU implements |
| STU | Sequential Transduction Unit — one HSTU layer; stacked layers form the STUStack |
| SL | Stochastic Length — the sub-sampling technique that keeps long user histories tractable |
| M-FALCON | Microbatched-Fast Attention Leveraging Cacheable OperatioNs — HSTU's inference-serving algorithm |
| FLOPs | Floating-Point Operations — a standard measure of computational cost; used here as a proxy for model complexity |
| QPS | Queries Per Second — a throughput metric for a serving system |
| NE | Normalized Entropy — the production loss and evaluation metric used in the paper; a 0.001 drop is considered a topline-significant win at scale |
| NDCG@k | Normalized Discounted Cumulative Gain at cutoff k — an offline ranking-quality metric that rewards placing relevant items near the top |
| HR@k | Hit Rate at cutoff k — the fraction of cases where the correct next item appears in the top k predictions |
| MRR | Mean Reciprocal Rank — an offline ranking-quality metric based on the position of the first relevant result |
| E-Task | An engagement-prediction task in the model's multi-task head (e.g., click, watchtime, or like); the 12.4% figure is the online metric lift measured on that task in the paper's production A/B test |
| KV cache | Key/Value cache — stores previously computed attention keys and values so they do not need to be recomputed for each new request |
| IPM | Installs Per Mille — installs generated per thousand ad impressions, a standard rewarded-advertising monetization metric |
| SDK | Software Development Kit — the integration a host app uses to request and display a rewarded offer |
| SLO | Service-Level Objective — a target latency or reliability bound a serving system is expected to meet |
| GCP | Google Cloud Platform — the cloud environment used in the deployment plan |
| GKE | Google Kubernetes Engine — GCP's managed Kubernetes service |
| GCS | Google Cloud Storage — GCP's object storage service |
| NCCL | NVIDIA Collective Communications Library — handles GPU-to-GPU communication during distributed training |
| DDP | Distributed Data Parallel — PyTorch's standard multi-GPU training strategy |
| MLPerf | An industry-standard benchmark suite for measuring machine learning training and inference performance |
| CI/CD | Continuous Integration / Continuous Deployment — automated build, test, and release pipelines |
| SCCs | Standard Contractual Clauses — the EU-approved legal mechanism for transferring personal data outside the EU |
| DPO | Data Protection Officer — the role responsible for an organization's data-protection compliance |
| ICML | International Conference on Machine Learning — the venue where this paper was published |
| PMLR | Proceedings of Machine Learning Research — the publication series that indexes ICML's accepted papers |
Why DLRMs stopped scaling with compute
DLRMs rely on thousands of hand-engineered features feeding shallow feature-interaction layers. That works, but it has three structural ceilings:
- No explicit structure in the feature space. A DLRM's feature space is 1,000–10,000 handcrafted signals, versus roughly one unified sequence in the sequential-transduction setting.
- A vocabulary that breaks static-LM assumptions. Recommendation vocabularies are billion-scale and non-stationary — nothing like a fixed 100K-token language vocabulary.
- Naive self-attention over user history is compute-prohibitive at O(N³d + N²d²), which is exactly why nobody just bolted a vanilla Transformer onto a DLRM and called it done.
The empirical consequence, documented by Zhao et al. (2023): DLRM quality plateaus regardless of how much additional compute you throw at it. This is the gap generative recommenders are built to close.
Approximate positions after Figure 1 (Zhai et al., 2024) — DLRM points from Mudigere et al., 2022. Redrawn here as an illustrative log-scale scatter, not a digitization of exact published figures.
Reframing ranking as sequential transduction
The paper's core move is right there in its title: instead of tokenizing text, generative recommenders (GRs) tokenize the interleaved stream of content shown to a user and the actions they take on it — modeling the full joint distribution p(Φ₀,a₀,Φ₁,a₁,…), where Φᵢ is a content token and aᵢ is the action token that follows it.
That interleaving buys two things at once:
- Retrieval becomes learning p(Φᵢ₊₁ | uᵢ) over the content vocabulary — with supervision that differs from vanilla autoregression, since negative actions map to ∅ and non-engagement tokens carry no target.
- Ranking becomes target-aware cross-attention inside a single causal pass — p(aᵢ₊₁ | Φ₀,a₀,…,Φᵢ₊₁) — instead of the O(N) impression-based re-encoding a DLRM requires every time a new candidate needs scoring.
This closes the gap with the sequential sub-modules DLRMs already bolt on (DIN, BST, TWIN) while staying fully generative, with a single objective: arg maxΦ p(Φ | uᵢ) to maximize reward.
Inside HSTU: pointwise attention, fused into three sub-layers
HSTU blocks run three fused operations per layer:
- Pointwise projection — U,V,Q,K = Split(ϕ₁(f₁(X))), one fused matmul instead of four separate linear layers.
- Spatial aggregation — A(X)V(X) = ϕ₂(Q(X)K(X)ᵀ + rab^{p,t})V(X), where pointwise (SiLU-normalized) attention replaces softmax.
- Pointwise transformation — Y(X) = f₂(Norm(A(X)V(X)) ⊙ U(X)), with gating folding the MLP-style transformation directly into attention.
The choice to drop softmax in favor of pointwise attention is not cosmetic — it preserves signal intensity, which matters when predicting engagement magnitude, not just ranking order, and it cuts linear layers from six to two relative to a standard Transformer block. On a synthetic Dirichlet-process streaming benchmark, that shows up directly:
| Architecture | HR@10 | HR@50 |
|---|---|---|
| Transformers | .0442 | .2025 |
| HSTU (softmax attention) | .0617 | .2496 |
| HSTU (pointwise attention) | .0893 | .3170 |
And at industrial scale — 100 billion examples of streaming ranking and retrieval — Transformers do not just underperform, they diverge:
| Architecture | Retrieval logppl | Ranking NE (E/C) |
|---|---|---|
| Transformers | 4.069 | NaN / NaN (loss explosion) |
| HSTU | 3.978 | .4937 / .7805 |
A 0.001 drop in Normalized Entropy (NE) is considered a significant, topline-moving win at billion-user scale — so the gap between "converges" and "explodes" here is not a rounding error.
One STULayer, end to end. A single fused matmul (W_uvqk) produces the gate, values, queries and keys together; triton_hstu_mha applies causal masking and relative timestamp bias inside one fused kernel with no materialized L×L matrix; and the residual path (x_out = x + W_out(u ⊙ Norm(Attn))) means the layer only ever learns Δx, so gradients reach early layers unattenuated across 12+ stacked STULayers.
Building blocks: kernels, fusion, ragged tensors, and caching
The remainder of this post leans on four engineering concepts repeatedly. They are defined once here, in plain terms, before they are used.
GPU kernels: CUDA versus Triton
A kernel is a single function that runs on the GPU, executing the same instruction across thousands of parallel threads at once — a matrix multiply, a normalization, an activation function. CUDA is NVIDIA's low-level programming platform for writing kernels directly against the GPU hardware: maximum control, but verbose and hard to get right. Triton is a higher-level, Python-embedded language that compiles down to efficient GPU code without hand-writing raw CUDA; most of the custom operators in this codebase are written in Triton, with a thin CUDA/C++ layer underneath for the handful of pieces that need lower-level control. When this post refers to a "custom" or "modified CUDA" kernel, it means exactly this: code written at the Triton or CUDA level, below PyTorch's standard operators, purpose-built for HSTU's specific computation pattern.
Fusion: why combining operations is the real speedup
Every standalone PyTorch operation — a matmul, then a LayerNorm, then a projection — is its own kernel launch, and each one reads its input from the GPU's slower main memory (HBM) and writes its output back before the next operation can start. On modern GPUs, that memory movement, not the arithmetic itself, is usually the bottleneck: the chip can do far more math per second than it can move data. Fusion means hand-writing a kernel that performs several of those steps back to back inside one kernel launch, keeping intermediate results in the GPU's fast on-chip memory (SRAM) the whole time instead of round-tripping to HBM between each step. hstu_preprocess_and_attention() fusing LayerNorm with the U/V/Q/K projection into a single matmul is exactly this: same math, one memory round-trip instead of several. Fusion does not change an algorithm's asymptotic complexity — it removes wasted memory bandwidth, which is why it is described elsewhere in this post as the multiplier sitting on top of every algorithmic win.
Ragged (jagged) tensors: why padding is wasteful
Real user histories vary enormously in length — ten interactions for one user, ten thousand for another. A standard batched matmul needs a fixed-size rectangular tensor, so a naive implementation pads every sequence in a batch out to the length of the longest one, filling the gap with zeros. Those zeros still get multiplied through every layer — wasted computation, and often the majority of a batch's FLOPs when even one sequence in the batch is unusually long. A ragged or jagged tensor avoids this by concatenating every sequence end to end into one long, unpadded array, with a small offsets array recording where each individual sequence starts and ends. The custom ragged attention kernel then computes attention as a set of grouped matrix multiplications ("grouped GEMMs") sized to each user's actual sequence length, rather than one large padded batch matmul — this is what eliminates roughly 90% of padding FLOPs.
Caching for inference: not recomputing what has not changed
In causal attention, a token's key and value vectors are a function only of that token and everything before it — they never change once computed. A KV cache takes advantage of this by storing the key/value tensors for a user's fixed interaction history once, then reusing them for every new candidate item scored against that same history, instead of recomputing the full history's attention from scratch each time. This is the mechanism that makes it affordable to score many candidates per user — the cost of encoding history is paid once, and each additional candidate only costs the marginal compute needed for that one new item.
Two O(N) tricks that make long histories tractable
Two algorithmic choices are doing most of the efficiency work, independent of kernel fusion:
Generative (streaming) training. Impression-level DLRM training re-encodes the user for every candidate. GRs instead emit one training example per session and let a single encoder pass supervise every position in the sequence — sampling the i-th user at rate proportional to 1/nᵢ drops total training cost from O(N³d + N²d²) to O(N²d + Nd²), an O(N) reduction. In practice, that means emitting the training example at the end of a user's session or request.
Stochastic Length (SL). User behavior is temporally repetitive across scales, so long histories can be sub-sampled without losing signal. For a user with nc,j interactions and Nc = maxj nc,j, SL keeps the full sequence when nc,j ≤ Nc(α/2), and otherwise samples a length-L subsequence with probability 1 − Ncα/nc,j² (or keeps the full sequence with the complementary probability) — cutting attention complexity to O(Ncαd) for α ∈ (1, 2]. Length extrapolation via SL beats RoPE zero-shot and fine-tuned baselines outright (a 0.098% NE gap versus 1.6–10.4% for the alternatives).
How aggressively that sub-sampling can cut sequence length — while holding quality essentially intact (ΔNE < 0.002) — depends on both α and the maximum sequence length being modeled:
| α | 1,024 | 2,048 | 4,096 | 8,192 |
|---|---|---|---|---|
| 1.6 | 71.5% | 76.1% | 80.5% | 84.4% |
| 1.7 | 56.1% | 63.6% | 69.8% | 75.6% |
| 1.8 | 40.2% | 45.3% | 54.1% | 66.4% |
| 1.9 | 17.2% | 21.0% | 36.3% | 64.1% |
| 2.0 (base) | 3.1% | 6.6% | 29.1% | 64.1% |
Combined, HSTU plus Stochastic Length trains 5.3x–15.2x faster than FlashAttention-2 Transformers at 8K sequence length. Underneath both of those algorithmic wins sits the custom ragged attention kernel described above — in the spirit of Rabe & Staats (2021) and Dao et al. (2022) — which turns naive dense attention (costing Θ(Σᵢ nᵢ²) over a batch, and memory-bound at Θ(Σᵢ nᵢ² d²qk R⁻¹), where nᵢ is the sequence length for sample i, dqk is the attention dimension, and R is GPU register size) into grouped GEMMs sized to each sequence's actual length. That kernel alone delivers 2–5x throughput gains before Stochastic Length is even applied — and it is applied to training only, since training already costs far more than inference, so this is where cheap sparsity pays for itself fastest.
The engineering: one kernel layer, two stacks
The production codebase is organized as one package, four directories, and a single kernel layer everything else calls into:
modules/— production-grade HSTU components:hstu_attention.py,hstu_compute.py,jagged_tensors.py,layer_norm.py,position.py.ops/— the fused Triton/CUDA/C++ kernels every other folder ultimately routes through.research/— an experimental suite for paper prototyping (trainer/train.py,modeling/sequential/hstu.py,autoregressive_losses.py).dlrm_v3/— production training plus the MLPerf benchmark suite (train/train_ranker.py,inference/main.py).
The production pipeline itself is four stages — assemble → preprocess → transduce → post-process — with a handful of files doing the real work: dlrm_hstu.py wires embeddings, preprocessors, the transducer, heads and loss into one module; contextual_interleave_preprocessor.py interleaves actions and items; stu.py implements the fused STULayer/STUStack; and postprocessors.py handles the L2/LayerNorm/timestamp-aware output heads.
Reading the stack top to bottom: an EmbeddingCollection does sparse table lookup for raw user/item IDs; a ContextualPreprocessor plus HSTUPositionalEncoder enriches item embeddings with action history, context, and bucketed timestamps; an N-layer STUStack (e.g. 12 layers) processes the full interleaved sequence; and a TimestampLayerNormPostprocessor feeds multi-task heads predicting click, watchtime, and like.
Whatever the entry point — action_encoder.py, content_encoder.py, the transducer itself — every tensor routes through jagged_tensors.py before it reaches a Triton or CUDA kernel. hstu_preprocess_and_attention() fuses LayerNorm with the U/V/Q/K projections into a single matmul, and the attention kernel never materializes the full L×N matrix in GPU DRAM. Jagged packing, KV-caching, and sampled softmax all still have to pass through this layer to become wall-clock speed. Fusion, more than any single algorithmic trick, is the multiplier sitting on top of everything else.
Execution flow: five phases, raw CSVs to a GPU-bound benchmark
| Phase | Compute | What happens |
|---|---|---|
| Data preprocessing | CPU | pandas-only ETL: download → extract → chronological sort → ID remap. Runs on a laptop, zero GPU. |
| Research training | GPU | main.py → train_fn(): DDP + NCCL, HSTU(num_blocks=8, heads=2), SampledSoftmaxLoss. |
| Research evaluation | GPU | Interleaved into the training loop every eval_interval batches; NDCG@k, HR@k, MRR. |
| DLRM-v3 training | GPU × 4 | train_ranker.py: production training script, same fused HSTU core, torchrec sharding. |
| DLRM-v3 inference | GPU × 4 | inference/main.py: MLPerf loadgen benchmark runner, M-FALCON micro-batched serving. |
The blocking dependency: fbgemm_gpu (jagged_to_padded_dense, dense_to_jagged, cumsum) is GPU-only — it is the one library that makes CPU prototyping of training or inference impossible past the data-preprocessing phase.
Seven optimizations, same model, a fraction of the compute
| Technique | What it does | Complexity / impact |
|---|---|---|
| Batched Jagged Tensors (1D packing) | Removes zero-padding waste from variable-length sequences | ~90% of padding FLOPs eliminated |
| Fused Triton / CUDA kernels | Fuses LayerNorm + U/V/Q/K projection into one matmul; online softmax in SRAM | the multiplier on every other technique |
| Incremental KV-caching | Only new candidates attend to cached K/V at inference | O(N²) → O(ΔN·N) |
| Sampled softmax loss | Scores against R negatives instead of the full catalog | O(|V|·D) → O(R·D), R=128 → >99% cut |
| Short-to-Long L2 STU (L2STU) | Splits memory into short- and long-term representations | trims redundant long-range attention |
| Stochastic Depth (SDSTU) | Randomly drops entire STU layers during training | cheaper training, regularization for free |
| Quantitative feature & target masking | Masks seen items / invalid targets before the loss | removes wasted supervision signal |
M-FALCON: serving 285x more compute without paying for it
Training efficiency is only half the story. Inference has its own bottleneck, and it is a different one: target-aware ranking (the mechanism described earlier that lets HSTU attend to a candidate item directly, rather than re-encoding the whole user for it afterward) means the model has to score dozens or hundreds of candidate items per user on every request. Run naively, that is m entirely separate forward passes through the full history for m candidates — the history gets re-encoded from scratch for every single item under consideration, which is by far the most expensive part of the computation and almost all of it is redundant work.
M-FALCON — Microbatched-Fast Attention Leveraging Cacheable OperatioNs — is the serving algorithm that removes that redundancy, through three mechanisms working together:
| Mechanism | What it does | Why it helps |
|---|---|---|
| Shared-history attention mask | All candidates for one user are scored in a single forward pass; the attention mask lets every candidate attend to the shared history, but not to each other | The expensive part — encoding history — is paid once per user instead of once per candidate |
| Microbatching | Very large candidate sets (tens of thousands, for retrieval-scale scoring) are split into microbatches sized around the history length itself | Keeps each batch small enough to stay efficient on GPU without losing the shared-history benefit |
| KV caching across requests | The key/value tensors for a user's history are computed once and reused across microbatches — and across separate requests in the same session, if the history has not changed | Only the new candidates' own K/V need computing; the history's K/V are pure cache lookups |
The net effect on complexity: scoring m candidates against an n-token history in a batch of b users costs O(bm·n²d) run naively — every candidate pays for the full history's attention from scratch. The shared-history mask reduces that to roughly O(n²d) per user, independent of how many candidates are being scored, because the history is only encoded once; microbatching then lets that same trick scale to candidate sets far larger than would otherwise fit in one GPU pass; and KV caching removes even the cost of the history's forward pass on every subsequent request within a session, leaving only the marginal cost of whatever candidates are new.
Put together, this is what lets a 285x more computationally complex model serve at 1.5x–3x the queries-per-second of the production DLRM baseline, on the same inference budget — the model got far more expensive per candidate, but the cost of the part that dominated the naive computation (re-encoding history) was engineered away.
Together, this lets a 285x more computationally complex model serve at 1.5x–3x the queries-per-second of the production DLRM baseline, on the same inference budget.
HSTU's latency advantage widens with sequence length: 21.5ms vs. 121.3ms at 8K tokens.
At larger candidate counts, M-FALCON lets a far more expensive model close the QPS gap with — and eventually exceed — the DLRM baseline.
A concrete alignment: X's Playtime
Architecture papers are one thing; the more useful test is whether the design maps onto a real system. This case study uses a mobile ad-tech platform referred to as X as the test case — a rewarded-advertising business built around one core loop: show the right rewarded offer to the right user, at the right moment, out of a catalog that changes every day. Its flagship product, referred to here as Playtime, is a rewarded-ad unit inside host apps where users earn in-app currency for installing or engaging with advertised games, matched by first-party data and machine learning.
Four scale characteristics make this a strong fit for a GR-style approach rather than an accident of terminology:
- Catalog — advertiser campaigns are added and paused continuously: a non-stationary vocabulary, not a fixed SKU list.
- Signal — installs, task completions, in-app-purchase events, video-ad views, and time-spent thresholds are naturally multi-hot "actions," exactly the token type HSTU expects.
- Volume — billions of rewarded-ad impressions and engagement events flow through the network daily.
- Objective — genuinely multi-task: match probability, expected playtime or IPM, and downstream retention all matter simultaneously.
The concept mapping is close to one-to-one: content tokens Φᵢ become the offer shown in the catalog at position i; action tokens aᵢ become install, task-complete, IAP, video-view, watchtime-threshold, or skip; the live campaign catalog is the high-cardinality, non-stationary vocabulary; and M-FALCON's micro-batched inference is exactly what's needed to score tens of thousands of live campaigns per catalog refresh, in real time.
A proposed ranking layer follows the same four-stage shape as the reference architecture: merge per-user events into one interleaved, timestamped action stream; use an HSTU retrieval head to shortlist candidates from the live catalog, replacing rule- or embedding-based candidate generation; score the shortlist for match probability and expected value with interleaved causal attention in a single pass; and serve it all through micro-batched, KV-cached inference within the existing latency budget. The expected payoff is straightforward: better offer matching should lift IPM and playtime completion, unifying features into one sequence should cut feature-engineering overhead, M-FALCON should keep serving cost flat as the catalog grows, and the compute-quality scaling law offers a lever for future gains that does not depend on discovering new handcrafted signals.
Rolling it out without betting the infrastructure budget upfront
None of this should ship as a day-one leap to production traffic. A staged plan with real go/no-go gates makes more sense: offline replay against historical logs first (zero production risk), then shadow deployment behind the live ranker on real traffic, then a limited A/B on a single market with explicit fraud-rate and latency guardrails, then retrieval and scale-out once the earlier gates have cleared, and only then the open research extensions — cold-start blending, fraud-aware sequence modeling, multi-market variants.
Two of those open questions are worth naming directly. Cold start is real: new users and new advertiser campaigns arrive with no action history, and the paper's own "content-only GR" baseline underperforms sharply in exactly this regime (11.6% versus 36.9% HR@100 in Table 6). The proposed fix is to blend content and metadata priors — app category, creative features — with the sequential signal early in a session, warm-start from cohort-level sequences, and let the content prior decay as real interaction volume accrues. Concretely, that means multilingual sentence embeddings over store descriptions and ad copy, a CLIP-style vision encoder over app icons and creatives, and keyframe pooling for video — validated offline on the cold-start slice before it ever goes near an online test.
Fraud is the second open question, and it is structural rather than incidental: rewarded engagement is a known abuse surface — install farms, emulator or bot completions, IPM gaming — which is exactly why a platform like this runs a dedicated anti-fraud function in the first place. An auxiliary anomaly head on the same action-sequence representations, scoped jointly with that team rather than bolted on after ranking ships, is the more defensible starting point than treating fraud as someone else's problem downstream.
Putting it on GCP: right-sized, not day-one hyperscale
The training stack is CUDA-bound end to end — fbgemm_gpu, Triton, NCCL — so the deployment has to be GPU-native from the start; there is no CPU fallback path past data preprocessing. A reference architecture on GCP breaks cleanly into four layers: ingest and storage (Pub/Sub for the raw event stream, Dataflow to sequentialize it into jagged interaction logs, GCS for sharded Parquet and checkpoints, BigQuery for analytics); training (GKE or Vertex AI Training on A2/A3 node pools, Vertex AI Pipelines to orchestrate preprocess → train → eval → export); serving (GKE with an L4/A100 pool running the HSTU transducer behind Triton Inference Server, M-FALCON micro-batching at request time, and an optional Memorystore Redis tier for cross-request KV-caching); and ops (Cloud Monitoring for latency/QPS/GPU utilization, Vertex AI Model Monitoring for NE and offline-metric drift, Cloud Build feeding Artifact Registry and GKE for CI/CD).
The critical point is that none of this needs to start at Meta scale. The paper's headline numbers describe Meta's traffic, not a required starting point, and every phase can be validated at single-digit GPU-hours per day before the next is justified:
| Stage | Purpose | GCP config | Approx. cost |
|---|---|---|---|
| 0 — Offline validation | Reproduce the paper's setup at small scale on internal data | 1× a2-highgpu (A100 40GB) | ~$3.7 / GPU-hr |
| 1–2 — Shadow / pilot A/B | Streaming training, single market | 4–8× A100 80GB (a2-ultragpu) | ~$5.1 / GPU-hr |
| 3 — Scale-out | Multi-market, longer sequences | A100 pool, or A3 (H100) if throughput demands it | ~$9–$11 / GPU-hr (H100) |
| Serving (all stages) | M-FALCON micro-batched inference | L4 pool, autoscaled on QPS | ~$0.70 / GPU-hr |
Latency deserves particular care here: rewarded offers are typically served synchronously inside a host app's SDK call, which is a tighter budget than a typical web ranking service. M-FALCON's micro-batching and KV-caching exist precisely to keep a 285x-more-complex model inside that existing SLO — but the real figure needs validating against the specific system, not assumed from the paper.
And because this is EU-regulated ad-tech, compliance has to be designed in rather than retrofitted — this is not legal advice, just engineering considerations to route to legal and a DPO early, before a pilot scales. GRs make some things easier: sequence tokens are pseudonymous action/content IDs rather than thousands of handcrafted, PII-adjacent features, which narrows the surface area for data-minimization and erasure obligations, and one unified interaction log is simpler to reason about than dozens of derived DLRM feature tables. But plenty still needs deliberate design: EU data residency (training and serving confined to European regions, cross-border transfers gated behind SCCs), a genuine right-to-erasure path through the jagged interaction store — not just the online feature store — and an early flag to legal on whether a system shaping monetized user behavior at this scale draws EU AI Act transparency obligations.
What it actually costs — and where it loses
None of this is free, and it's worth being honest about the trade-offs rather than only the upside:
| Advantages | Limitations & risks |
|---|---|
| Unifies thousands of handcrafted DLRM features into one sequence — less feature engineering, more privacy-friendly | Hard CUDA dependency (fbgemm_gpu, Triton, NCCL) — no CPU path past data preprocessing; real infra lock-in to GPU fleets |
| 5.3x–15.2x faster training and up to 5.6x faster inference vs. FlashAttention-2 Transformers | Custom fused kernels are a genuine maintenance burden — this is systems engineering, not just modeling |
| 12.4% online topline lift and up to 65.8% offline NDCG lift over strong baselines | At low compute or low data volume, DLRMs with handcrafted features can still outperform GRs |
| First demonstrated power-law scaling of recommendation quality with compute, up to GPT-3 / LLaMA-2 scale | No LLM-style world knowledge — cold start is target-aware but not solved; still needs real interaction history |
| M-FALCON serves a 285x more complex model at 1.5x–3x the inference throughput of the DLRM baseline | Migrating an existing DLRM stack, plus orchestrating trillion-parameter checkpoints, is a significant one-off cost |
Appendix 1 — SOTA positioning
How HSTU sits relative to sequential recommenders, DLRM sequential sub-modules, and language models applied to RecSys.
Eight eras of sequential recommendation
| Era | Representative models | Key idea |
|---|---|---|
| Markov models | FPMC, MC | Next item depends on previous item(s) |
| RNN | GRU4Rec, NARM, STAMP | Model user behavior as a sequence |
| CNN | Caser, NextItNet | Temporal convolutions over interaction sequences |
| Self-Attention | SASRec, BERT4Rec, TiSASRec | Attention over past interactions |
| Graph + Sequence | SR-GNN, GCSAN, DGSR | Combine session graphs with sequential modeling |
| Contrastive Learning | CL4SRec, DuoRec, ICLRec | Learn robust sequential representations |
| State Space Models | Mamba4Rec, MambaRec, SSMRec | Linear-complexity sequence modeling |
| Foundation Models | TIGER, LLM4Rec, TALLRec, RecGPT | Leverage pretrained LLMs for recommendation |
| Sequential Transduction (this work) | HSTU / Generative Recommenders | Actions AND items as tokens; pointwise attention; generative, streaming training |
Each step mostly fixes one limitation of the one before it: Markov chains only see one step back → RNNs see the whole sequence but forget long-range dependencies → CNNs and attention parallelize and extend range → graphs capture non-linear session structure → contrastive learning fixes data sparsity → state-space models fix attention's quadratic cost → LLMs bring world knowledge. HSTU is the first to fold nearly all of these motivations — long-range, generative, near-linear cost, action-level granularity — into a single architecture, at industrial scale.
The lineage, one model at a time
The throughline runs: MC → FPMC (matrix factorization plus first-order transitions) → GRU4Rec (the first major deep sequential recommender, handling variable-length sessions) → NARM / STAMP (adds attention to weight the most relevant interactions) → Caser / NextItNet (treats interaction history like an image, or applies WaveNet-style dilated causal convolutions) → SASRec (the general Transformer building block adapted for recommendation — still one of the strongest baselines) → BERT4Rec / TiSASRec (bidirectional cloze training, or explicit time-gap-aware attention) → SR-GNN / GCSAN (session-as-graph, capturing non-linear navigation) → the CL4SRec family (augmentation-invariant representations, more robust under sparse data) → the Mamba4Rec family (linear-time state-space recurrence for very long histories) → LLM4Rec / RecGPT (pretrained world knowledge, strong in low-data settings) → HSTU / GR, which is the first to fold nearly all of those motivations into one architecture at industrial scale.
Architectures at a glance
Classical, recurrent & convolutional. Markov Chains model item(t−1) → P(item t): only the last interaction determines the next item, simple but still useful for session-based recommendation. FPMC combines matrix factorization (long-term) with Markov transitions (short-term) — a classic baseline. GRU4Rec runs item embeddings through a GRU to a hidden state and a next-item prediction; NARM adds an attention layer on top of that same GRU backbone to weight the most relevant interactions. STAMP replaces recurrence with attention over a session embedding to emphasize current intent. Caser treats interaction history like an image, applying horizontal and vertical convolutions; NextItNet applies WaveNet-style dilated causal convolutions with residual connections to model long sequences without recurrence.
Attention & graph-based. The basic Transformer block — embedding, positional encoding, self-attention, FFN — is the general building block every attention-based recommender adapts. SASRec applies that block causally over item embeddings and remains one of the most influential and strongest sequential-recommendation baselines. BERT4Rec instead uses bidirectional self-attention with masked (Cloze-style) item prediction — richer representations, but non-causal and not streaming-friendly. TiSASRec extends SASRec with a time-interval-aware attention bias, using real gaps between interactions rather than just their order. SR-GNN represents a session as a directed graph processed by a GNN to capture non-linear navigation paths; GCSAN combines that session graph with self-attention for local-plus-global signal. The CL4SRec family (with DuoRec, ICLRec) augments sequences — cropping, masking, reordering — and encodes them with a contrastive loss to learn representations that are more robust under sparse data.
Efficiency, paradigms & foundation models. Five ideas here matter most for why HSTU looks the way it does: the classic seq2seq encoder-decoder (encoder RNN → context vector → decoder RNN → output) is the conceptual precursor to attention-based sequence models; autoregressive modeling (predicting xt from x<t, causal-masked) is the training paradigm HSTU, SASRec, GRU4Rec, and GPT-style LMs all share; FlashAttention-2 (tiled Q/K/V fused into online softmax computed in SRAM, with an IO-aware backward pass) is exact attention with no algorithmic change — it wins purely on GPU memory-IO efficiency, and it is HSTU's own baseline for the Part B efficiency comparisons above; Mamba and other state-space models replace quadratic attention with linear-time selective-state recurrence, attractive for very long histories; and foundation models (LLM4Rec, TIGER) map item text or semantic IDs through a pretrained LLM for in-context or fine-tuned recommendation — bringing world knowledge and strong low-data performance, but not yet SOTA at industrial scale.
Cheat sheet: the shortlist to know cold
| Category | Models |
|---|---|
| Classical | Markov Chains, FPMC |
| RNN | GRU4Rec, NARM |
| CNN | Caser, NextItNet |
| Transformers | SASRec, BERT4Rec, TiSASRec |
| Graphs | SR-GNN, GCSAN |
| Contrastive | CL4SRec, DuoRec |
| Latest (SSM) | Mamba4Rec and other SSM-based recommenders |
The throughline: the field moved from simple transition models to architectures that model very long user histories efficiently — exactly the progression that motivates HSTU's design choices.
Where HSTU sits among production ranking systems
| Approach | Architecture | Training procedure | Key limitation vs. HSTU |
|---|---|---|---|
| GRU4Rec / SASRec | RNN / self-attention (Transformers) | Target-independent, multi-pass | No target-aware attention; small feature set |
| BERT4Rec / S3Rec | Bidirectional self-attention | Sequential multi-pass (Cloze) | Not causal / streaming-friendly |
| DIN / BST / TWIN / TransAct | Pairwise or self-attention inside a DLRM | Pointwise, streaming | Target-aware, but bolted onto a DLRM — limited capacity, O(N) training cost |
| DLRM (DIN+DCN+MMoE) | Feature interaction + MoE heads | Pointwise, streaming | Thousands of handcrafted features; plateaus with compute |
| Generative Recommenders (HSTU) | Pure sequential transduction | Causal autoregressive, streaming, single-pass | Target-aware AND generative; scales as a power law of compute |
Empirical positioning: the gap widens with scale
Across three orders of magnitude of training compute, GR quality — Hit Rate, Normalized Entropy — follows a power law, while equivalent DLRM baselines plateau early even as embedding and non-embedding parameters keep growing. At the largest published comparison, the DLRM parameter ceiling sits around 200B, against 1.5T reached by production GR — compute territory that overlaps with GPT-3 and LLaMA-2 training compute, the first time a recommendation system has been shown to sit on the same scaling curve as frontier LLMs. The same pattern shows up directly on public benchmarks:
NDCG@10 on public benchmarks (multi-pass, full-shuffle). HSTU and HSTU-large consistently beat SASRec, and the gap widens with scale — the scaling-law result is the headline, not any single benchmark number.
Appendix 2 — Code & project walkthrough
The repository, mapped to how you'd actually run it — from a laptop CSV job to a 4-GPU MLPerf benchmark.
How to actually run it
Underneath the paper, the open-source repository (facebookresearch/generative-recommenders) is a reasonably approachable path from a laptop CSV job to a multi-GPU MLPerf benchmark:
pip install -r requirements.txt # torch, fbgemm_gpu, torchrec, gin_config, triton, pandas, tensorboard
python preprocess_public_data.py # ML-1M / ML-20M / Amazon Books -> tmp/<dataset>/output_format.csv
python run_fractal_expansion.py # optional: synthetically grow a dataset for scaling studies
# configs/*.gin — hyperparameters, model shape, dataset selection
python main.py # research training: DDP via mp.spawn -> train_fn()
python dlrm_v3/train/train_ranker.py # production training: torchrec-sharded, multi-GPU
python dlrm_v3/inference/main.py # inference benchmark: MLPerf loadgen, exercises M-FALCONDependency reality check: what each phase actually requires
| Phase | CUDA/GPU | fbgemm_gpu | torchrec | Triton | Multi-GPU | MLPerf loadgen |
|---|---|---|---|---|---|---|
| Data preprocessing | · | · | · | · | · | · |
| Fractal expansion | ✓ | · | · | · | · | · |
| Research training | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| Research evaluation | ✓ | ✓ | ✓ | ✓ | ✓ | · |
| DLRM-v3 training | ✓ | ✓ | ✓ | · | ✓ | · |
| DLRM-v3 inference | ✓ | ✓ | ✓ | · | ✓ | ✓ |
Only data preprocessing is CPU-only — every training or inference phase needs CUDA plus fbgemm_gpu's GPU-only jagged-tensor operations, which is the one dependency that rules out CPU prototyping past the first step. Core external dependencies: torch ≥ 2.6.0, fbgemm_gpu ≥ 1.1.0, torchrec ≥ 1.1.0, gin_config ≥ 0.5.0, plus Triton, pandas, tensorboard, and MLPerf loadgen for the benchmark step.
Recap: seven takeaways
- Foundations — recasting ranking and retrieval as sequential transduction closes the DLRM/sequential-recommender gap.
- Engineering — one kernel layer (
ops/) powers a research stack and a production stack; fusion is the multiplier. - X alignment — a textbook GR fit for Playtime, staged through a five-phase roadmap with cold-start and fraud as named research questions.
- GCP path — a right-sized, cost-estimated, EU-compliant pipeline, with a 16-week Foundation → Deploy → A/B plan and pre-registered gates.
- Trade-offs — 15x training speed and a compute scaling law, against real GPU lock-in and kernel-maintenance cost.
- SOTA position — traces all eight eras of sequential recsys (~19 architectures) and is the first shown to scale like an LLM.
- Code — a clear, runnable path: preprocess → configure → train (research or production) → MLPerf-benchmark.
The short version
Recasting ranking and retrieval as sequential transduction closes the long-standing gap between DLRMs and sequential recommenders — and it does it with one kernel layer powering both a research stack and a production stack, where fusion turns out to be the real multiplier on top of every algorithmic win. Applied to a live rewarded-advertising ranking problem, it is a genuinely close conceptual fit, staged through gated phases rather than a leap of faith, with cold start and fraud named as open research questions rather than glossed over. On GCP, it is a right-sized, cost-estimated, EU-compliant path rather than a Meta-scale fantasy. The trade-off is real: 15x training speed and a compute scaling law, against real GPU lock-in and kernel-maintenance cost that should not be hand-waved away. And in the broader field, it is the architecture that traces all eight eras of sequential recommendation research and is the first shown to scale like a large language model.
Comments
Post a Comment