Implementation:Recommenders team Recommenders SUM Cells

Knowledge Sources	Recommenders
Domains	Sequential Recommendation, Deep Learning, Recurrent Neural Networks
Last Updated	2026-02-10 00:00 GMT

Overview

This module implements custom RNN cells for the SUM (Sequential User Matrix) model, providing multi-channel memory-based user state tracking with attention-driven reading and writing, novelty-aware gating, and a highway slot mechanism.

Description

The module provides two RNN cell implementations (SUMCell and SUMV2Cell) that extend Keras's LayerRNNCell to maintain multiple memory slots representing distinct user interest channels, along with utility functions for dtype validation.

SUMCell maintains a state composed of s memory slots of hidden size h plus a highway slot and an input-sized buffer for tracking the previous input. The state size is computed as (slots * real_units) + real_units + input_size, where real_units = (num_units - input_size) / slots. The cell operates through the following mechanism at each time step:

Reading: Attention weights are computed via dot product between the input and learned head vectors (one per slot), passed through softmax with a learnable temperature parameter beta. A weighted sum of the memory slots combined with the highway slot produces the read vector h_hat.

Novelty detection: A cosine similarity distance between the current input and the previous input (stored in the state buffer) is computed. This distance is modulated by a learnable alpha parameter via exponentiation (alpha^dist), producing a novelty score that scales write intensity -- novel inputs produce stronger writes.

Writing: GRU-style gating with reset, erase, and add gates processes the input and read vector. Memory slots are updated through attention-weighted erase-and-add operations: state = state * (1 - att_weights * erase) + att_weights * erase * add. The highway slot is updated similarly but uses the novelty distance directly instead of attention weights.

SUMV2Cell extends SUMCell by replacing the reading-based attention with a learned writing-specific attention mechanism. Instead of reusing the softmax reading attention for writing, it computes separate writing attention weights through a two-layer fully connected network: the concatenation of input and h_hat is passed through a linear layer with ReLU activation, then through a second linear layer, and finally softmax-normalized. This allows the cell to learn distinct attention patterns for reading versus writing.

Both cells include _basic_build which creates the shared parameters: erase weights/bias, reset weights/bias, add weights/bias, head vectors, and the learnable beta (initialized to 1.02) and alpha (initialized to 0.98) scalars. Parameters named with _no_reg suffix are excluded from regularization.

Usage

Use SUMCell or SUMV2Cell as the RNN cell within the SUM sequential recommendation model when you need to model multiple user interests simultaneously. SUMCell is appropriate when reading and writing attention can share the same distribution. SUMV2Cell is preferred when the model benefits from learning separate attention mechanisms for reading (aggregating memory) and writing (updating memory), which provides more flexibility in how user interests are tracked and updated.

Code Reference

Source Location

• Repository: Recommenders
• File: recommenders/models/deeprec/models/sequential/sum_cells.py
• Lines: 1-381

Signature

class SUMCell(LayerRNNCell):
    """Cell for Sequential User Matrix"""

    def __init__(
        self, num_units, slots, attention_size, input_size,
        activation=None, reuse=None, kernel_initializer=None,
        bias_initializer=None, name=None, dtype=None, **kwargs
    ):
        """Initialize SUMCell with multi-slot memory configuration."""

    def build(self, inputs_shape):
        """Create parameters for the SUM cell."""

    def call(self, inputs, state):
        """Process user behavior at time T, update and return user state.
        Args:
            inputs: (batch of) user behaviors at time T
            state: (batch of) user states at time T-1
        Returns:
            Tuple of (new_state, new_state).
        """

class SUMV2Cell(SUMCell):
    """A variant of SUM cell with upgraded writing attention."""

    def build(self, inputs_shape):
        """Create parameters including writing-specific attention layers."""

    def call(self, inputs, state):
        """Process user behavior with learned writing attention.
        Args:
            inputs: (batch of) user behaviors at time T
            state: (batch of) user states at time T-1
        Returns:
            Tuple of (new_state, new_state).
        """

Import

from recommenders.models.deeprec.models.sequential.sum_cells import SUMCell, SUMV2Cell

I/O Contract

Inputs

Name	Type	Required	Description
num_units	int	Yes	Total state size: must equal (slots * real_units) + input_size, where real_units is the per-slot hidden dimension
slots	int	Yes	Number of memory slots (the last slot is reserved as the highway slot, so the usable slots are slots - 1)
attention_size	int	Yes	Attention mechanism size (used in model configuration)
input_size	int	Yes	Dimension of the input features; the last portion of the state stores the previous input
activation	str or callable	No	Activation function; defaults to tanh
kernel_initializer	initializer	No	Initializer for weight matrices
bias_initializer	initializer	No	Initializer for bias vectors; defaults to constant 1.0
inputs (call)	tf.Tensor	Yes	Tensor of shape [batch_size, input_size] representing user behaviors at time T
state (call)	tf.Tensor	Yes	Tensor of shape [batch_size, num_units] representing the full user state at time T-1

Outputs

Name	Type	Description
state (call)	tf.Tensor	Updated user state of shape [batch_size, num_units] containing updated memory slots, highway slot, and current input as the new "last input"
state (call)	tf.Tensor	Same as above (returned twice as both output and new state, since the full state is the output)

Usage Examples

Basic Usage

import tensorflow as tf
from recommenders.models.deeprec.models.sequential.sum_cells import SUMCell, SUMV2Cell

# Configuration
input_dim = 64
num_slots = 4
real_units = 32
num_units = num_slots * real_units + input_dim  # total state size

# Create a SUM cell
cell = SUMCell(
    num_units=num_units,
    slots=num_slots,
    attention_size=32,
    input_size=input_dim,
)

# Use with tf.nn.dynamic_rnn for processing a sequence of user behaviors
inputs = tf.placeholder(tf.float32, [None, None, input_dim])
initial_state = tf.zeros([batch_size, num_units])
outputs, final_state = tf.nn.dynamic_rnn(
    cell, inputs, initial_state=initial_state, dtype=tf.float32
)

# For the SUMV2 variant with learned writing attention
cell_v2 = SUMV2Cell(
    num_units=num_units,
    slots=num_slots,
    attention_size=32,
    input_size=input_dim,
)
outputs_v2, final_state_v2 = tf.nn.dynamic_rnn(
    cell_v2, inputs, initial_state=initial_state, dtype=tf.float32
)