How DLRM Trades Expressiveness for Structure at Scale This article is Part 2 of a two-part series on the structural and engineering trade-offs behind modern recommendation models such as DLRM. Part 1: https://medium.com/@np123greatest/why-recommendation-systems-are-structurally-different-from-deep-learning-1-2-62e9130acc6e The reference paper: https://arxiv.org/pdf/1906.00091 📌 TL;DR Once embeddings are in place, the real modeling question becomes how features should interact. This article explains why leaving all interactions to deep networks is often inefficient, and how Factorization Machines introduce structured second-order interactions. It then shows why DLRM succeeds by making deliberate engineering trade-offs between expressiveness, scalability, and controllability — rather than by being deeper or more complex. 4. The Core Idea of Factorization Machines: Making Second-Order Interactions Explicit and Controllable This chapter responds to a key question left open in the previous chapter: In the embedding space, how should relationships between features be modeled? The answer given by Factorization Machines (FM) is: Do not leave feature interactions entirely to a black-box network to “figure out”, but instead explicitly and structurally model second-order relationships between features . 4.1 From Matrix Factorization to CTR: Why Are More General Interaction Forms Needed? In classical recommendation systems, matrix factorization provides an extremely elegant answer. For the user–item rating prediction problem, the prediction takes the form: Source: Image by the author. The meaning of this formulation is very clear: Each user and each item is mapped to an embedding The predicted value comes from the inner product of two embeddings The inner product characterizes the “matching degree” in the latent factor space This works extremely well in binary user–item interaction scenarios. The Problem in CTR Scenarios: More Than One Pair of Interactions However, in CTR prediction, the problem becomes more complex: Features are no longer limited to user_id and item_id They also include: Context (hour, device, location) Ad attributes (campaign, creative) Various statistical features In other words, CTR is a multi-entity, multi-feature matching problem . If we try to generalize matrix factorization to CTR, a natural thought arises: Should we model all pairwise interactions between features? For example: Source: Image by the author. But this immediately leads to an engineering disaster : Let the number of features be N, The number of second-order interactions becomes O(N²) . Each interaction pair requires independent parameters As a result, both parameter size and computation cost explode quadratically . This means the problem is not whether second-order interactions are needed, but rather: How to model second-order interactions without introducing quadratic parameter growth? 4.2 The Key Insight of Factorization Machines Faced with the quadratic parameter and computation costs brought by second-order feature interactions in CTR scenarios, FM does not abandon interaction modeling. Instead, it redefines what “interaction” means . The core insight of FM can be summarized in one sentence: Explicitly model second-order interactions, while avoiding quadratic growth in parameters. Under this idea, FM parameterizes second-order interactions as follows: Source: Image by the author. where: x_i: the value of the i-th feature (one-hot / multi-hot / numerical) v_i: the embedding corresponding to the i-th feature : inner product between embeddings, representing their matching degree in latent factor space From Formula to Structure: How Is FM Actually Computed? Structurally, Factorization Machines can be decomposed into three clear stages(As shown in Figure): Core computation structure of Factorization Machines: all sparse features are first mapped to embedding vectors, then explicit second-order interaction features are constructed via pairwise inner products. The interaction results are concatenated with dense features and fed into an MLP for final CTR prediction: Source: Image by the author. 1. Embedding Representation Stage Each sparse feature is mapped to a low-dimensional vector (e.g., 64 dimensions) and no longer directly participates in high-dimensional computation. 2. Explicit Second-Order Interaction Stage (Interaction Layer) All embedding vectors corresponding to sparse features (in later models such as DLRM, dense features are usually first mapped into the same embedding space before participating in interactions) are paired and processed via dot-products . Each feature pair produces a scalar interaction feature : This layer introduces no new interaction parameters , but provides the model with a clear second-order interaction structure . 3. Prediction Stage (Top MLP) All second-order interaction features are concatenated with dense features and fed into an MLP, which models higher-order nonlinear combinations and outputs the click probability. Why Is This a “Structured” Second-Order Interaction? Compared with brute-force enumeration of all second-order parameters, FM has several advantages: 1.Parameter Sharing: All interactions share the same embedding space. Parameter size is O(N * d), rather than O(N²). 2.Controllable Computation The number of second-order interactions is N(N-1)/2. Each interaction only involves a single dot-product. 3.Preservation of Latent Factor Semantics: Inner products naturally capture similarity / matching degree. This is consistent with the modeling assumptions of matrix factorization. The essence of FM is not “adding more features”, but rather: Using stronger structural constraints to compress and express relationships between features. 4.3 Summary: FM Introduces “Controllable Relational Structure” into the Embedding Space Reviewing the derivation in this chapter, we can see that Factorization Machines are not introduced to increase complexity , but to add a long-missing piece of structural capability to embedding representations. After embeddings have solved “how to represent features” , FM further answers: Can relationships between features be explicitly modeled without introducing quadratic parameter and computation costs? By re-parameterizing each feature-pair interaction as an inner product between corresponding embedding vectors, FM achieves balance in several dimensions: Explicit modeling of second-order feature relationships Linearly controllable parameter scale Preservation of latent factor similarity semantics From a modeling perspective, FM does not provide a “complete model”, but rather a structured interaction template. It specifies which relationships should be computed, and in what form they should be computed. But it does not take responsibility for complex nonlinear aggregation. At this point, we have obtained all the key components required for recommendation system modeling: Using embeddings to represent discrete entities Explicitly modeling feature relationships in a structured way Using MLPs to handle higher-order nonlinear combinations The next chapter will show how these components are systematically combined in DLRM to form an efficient and scalable industrial-grade recommendation model . 5. DLRM Architecture Breakdown: Putting Structured Interactions Together with Deep Networks Before diving into the DLRM architecture itself, it is worth noting that most of the core ideas here have already been discussed. If the structure feels familiar, that is expected. What this chapter does is organize those ideas — embeddings, structured interactions, and modular responsibilities — explicitly around the DLRM architecture described in the original paper . 5.1 Design Overview: The Core of DLRM Is Not “Depth”, but “Division of Labor” Contrary to intuition, DLRM is not a “deeper” or “more complex” neural network. Its core design philosophy is: Decomposing representation learning, relationship modeling, and nonlinear aggregation into independent modules, each with a clear responsibility. From a system architecture perspective, DLRM is explicitly divided into four parts: Representation learning for dense features (Bottom MLP) Embedding lookup for sparse features (EmbeddingBag) Explicit second-order interaction modeling (Interaction Layer) Higher-order nonlinear aggregation (Top MLP) This decomposition directly corresponds to different types of features and modeling requirements in recommendation systems. Overall DLRM architecture: the model separately processes dense and sparse features. Dense features are mapped into the embedding space through the Bottom MLP, while sparse features are looked up via EmbeddingBag to obtain embedding representations. All embeddings (including dense embeddings) perform explicit second-order interactions (pairwise dot-products) in the Interaction Layer, producing structured interaction features. These interaction features are concatenated with dense embeddings and fed into the Top MLP to output the final click probability. Source: Image by the author 5.2 Dense Feature Path: The Role of the Bottom MLP As shown in the lower-left part of the architecture diagram, numerical (dense) features first enter the Bottom MLP : x_dense -> Bottom MLP -> e_dense The role of the Bottom MLP is not to make the final prediction, but to complete two key tasks: Representation Enhancement: Mapping raw numerical features (with different scales and complex distributions) into representations with more stable semantics . Dimensional Alignment: Mapping dense features to the same dimensionality ( d ) as sparse embeddings, so that they can be treated as equivalent embedding nodes in the subsequent interaction layer. From this perspective, the Bottom MLP essentially: “Creates an embedding” for dense features. 5.3 Sparse Feature Path: EmbeddingBag as System Memory The lower-right part of the architecture diagram shows how sparse features are processed. Each sparse feature (such as user_id, item_id, etc.) corresponds to an embedding table; In engineering implementations, these embedding tables are uniformly encapsulated within the EmbeddingBag module: Each feature has its own independent embedding table Lookup and updates are sparse They do not participate in dense matrix computation In DLRM, embeddings play the role of long-term memory : They store latent representations formed from historical interactions of users and items, They are not generic function parameters used for complex nonlinear transformations 5.4 Interaction Layer: A Direct Inheritance of FM Ideas The most critical layer in the DLRM architecture is the Interaction Layer . This layer takes as input: v1, v2, …, v_S, e_dense and performs pairwise dot-products over all embeddings: Source: Image by the author. This design directly inherits the core idea of Factorization Machines : No independent parameters are learned for each feature pair All interactions share the same embedding space Only the effect of interactions (scalar values) is retained, not interaction parameters themselves This brings three key advantages: Explicit modeling of second-order relationships: User–item, item–context, and user–context relationships are all explicitly computed. Linearly controllable parameter scale: Parameter size is O(S * d), rather than O(S²). Preservation of latent factor semantics: Dot-products still characterize “matching degree / similarity”. 5.5 Feature Concatenation and Top MLP: Only Doing the “Final Integration” The Interaction Layer outputs a set of scalar interaction features . These features are concatenated with the dense embedding to form the final input vector: z = [interaction features; e_dense] with dimensionality: Source: Image by the author. This vector is then fed into the Top MLP : Source: Image by the author. At this point, the responsibility of the Top MLP is very clear: It no longer “discovers relationships”, but nonlinearly combines relationships that have already been explicitly computed . 5.6 The Success of DLRM Comes from Structural Clarity, Not Model Depth The core value of DLRM does not lie in deeper networks or more parameters, but in: Respect for the structure of sparse problems A restrained design of parameter scale A clear division of what is handled by structure and what is handled by learning From Matrix Factorization → Factorization Machine → DLRM , models did not become increasingly complex, but instead: Modeling assumptions became clearer, and the structure became more aligned with the nature of recommendation systems. 6. Consequences of Design Choices: Differences Between DLRM and Other Deep Recommendation Models 6.1 A Key Point of Divergence Behind Different Models When comparing the structural differences between DLRM and models such as DeepFM and Deep & Cross , their divergence lies in the most fundamental understanding of embeddings . 6.1.1 The Choice of Other Models: Embeddings as a “Set of Continuous Features” In most deep recommendation models (such as Deep & Cross and some variants of DeepFM), the mapping from discrete features to embeddings itself is not fundamentally different from DLRM: Each discrete feature (such as user_id, item_id) Corresponds to an embedding vector with fixed dimensionality For example, in CTR scenarios: user_id → user_embedding ∈ R^d item_id → item_embedding ∈ R^d The real difference does not occur at the embedding generation stage, but rather at how embeddings are used to model interactions . In these models, embedding vectors are implicitly treated during the interaction stage as: A collection of continuous numerical features, rather than an indivisible semantic whole. Specifically, suppose: user_embedding = [u1, u2, u3, …, ud] item_embedding = [v1, v2, v3, …, vd] In models such as Deep & Cross : u_1, u_2, u_3, … are treated as independent feature dimensions Interactions are no longer restricted to whole-embedding interactions Finer-grained combinations are allowed, such as: u1 × v3 u2 × v7 In some architectures, even: u1 × u2 (interactions within the same embedding vector) are allowed.) That is, in these models, the basic interaction unit is not an embedding vector , but a single dimension within an embedding. From a modeling perspective, embeddings are no longer treated as semantic entities , but as freely combinable continuous features . This design directly leads to: The ability to construct more complex and higher-order feature combinations Significantly enhanced expressive power But at the same time, it also introduces: Higher dimensionality Increased computational cost Greater engineering complexity 6.1.2 DLRM’s Choice: Embeddings as a “Semantic Whole” DLRM adopts a completely different stance . In DLRM, an embedding vector: Corresponds to a discrete entity (user, item, or feature) Is treated as an indivisible semantic whole Interactions occur only between embeddings The interaction form is explicitly constrained to dot-product (consistent with FM) This means that DLRM explicitly models: Relationships between entities, rather than combinations between numerical dimensions. 6.2 The Direct Trade-offs Brought by This Choice Treating embeddings as a “set of continuous features” does indeed bring stronger expressive power, but this choice also comes with clear and direct costs . ❌ Cost 1: Rapid Growth in Dimensionality and Computation In large-scale recommendation systems: Number of sparse features S: dozens to hundreds Embedding dimension d: dozens to hundreds If dimension-level interactions within embeddings are allowed: Interaction scale introduces an additional d² amplification factor Parameter size, computation cost, and communication overhead all increase rapidly In contrast, DLRM only retains embedding-level dot-product interactions : O(S * d) The parameter scale does not grow exponentially with interaction order. ❌ Cost 2: Significantly Reduced Engineering Controllability In industrial systems, model design must account for: Distributed training Large-scale parallelism Strict latency and resource constraints Embedding dimension–level interactions lead to: More complex computation graphs Higher memory and communication pressure Less predictable system behavior This is also the explicit engineering judgment stated in the DLRM paper: “Higher-order interactions beyond second-order may not necessarily be worth the additional computational/memory cost.” In other words, this is not a question of “can it be done” , but rather “is it worth doing.” 6.3 DLRM’s Restrained Choice DLRM does not deny the potential value of more complex interactions. Instead, under the specific constraints of large-scale recommendation systems , it makes a highly restrained trade-off : Accepting a limited upper bound on expressive power In exchange for: Structural clarity Controllable parameter scale System scalability Its core judgment can be summarized as: In industrial recommendation systems, not all expressible relationships are worth being explicitly modeled. Conclusion: What DLRM Teaches Us Is More Than Just a Model Reviewing the entire article, we can see that DLRM is not merely an upgrade of a deep learning model , but rather a systematic clarification of modeling assumptions under industrial constraints : Discrete identities are not ordinary numerical features, but must be modeled as entities The relational structure between features should not rely entirely on black-box functions to learn implicitly In large-scale systems, expressive power must yield to engineering controllability From Matrix Factorization → FM → DLRM , models did not become deeper, but rather became increasingly clear about: What should be handled by structure, and what should be handled by learning. What Can You Take Away from This Series? After reading this series, you may not immediately retain a specific model architecture in detail. Instead, what is more likely to remain is a transferable industrial modeling perspective : Framing complex systems in terms of structural trade-offs Decomposing problems into representation, interaction, and aggregation Building intuition around expressiveness, scalability, and controllability When approaching industrial-scale problems, the guiding question naturally shifts from “Can performance be improved?” to “Is the improvement justified by its parameter cost, computational overhead, and system complexity?” More importantly, this perspective provides a reusable analysis framework. When encountering new recommendation, ranking, or advertising models in the future, it becomes easier to identify which structural problem a model is actually addressing — whether it is representation, interaction modeling, nonlinearity, or engineering constraints — leading to faster paper reading, more grounded design reviews, and clearer system documentation. Why Recommendation Systems Are Structurally Different from Deep Learning [2/2] was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.