Implementation:Recommenders team Recommenders Notebook Memory Management
| Knowledge Sources | |
|---|---|
| Domains | Profiling, Jupyter, Memory Management |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
IPython/Jupyter notebook utility that profiles memory usage and execution time for each cell execution, reporting RAM consumption deltas interactively.
Description
This module provides interactive memory profiling for IPython/Jupyter notebooks by hooking into IPython's event system. Based on the ipython_memory_usage project, it tracks RAM consumption changes after each cell execution.
The start_watching_memory function registers two IPython event callbacks: pre_run_cell (which captures the current time before cell execution) and watch_memory (which runs after each cell completes). The watch_memory callback calculates the memory delta between the current memory usage (via memory_profiler.memory_usage()) and the previous cell's usage, computes the elapsed time, and prints a formatted summary showing the cell identifier, memory delta in MB, current RAM usage, and total system RAM (via psutil.virtual_memory()).
The stop_watching_memory function unregisters both callbacks from the IPython event system. Global variables maintain state across calls, including the previous_call_memory_usage reference point and a watching_memory flag.
At module import time, the module initializes by reading the current memory usage and attempting to access the IPython namespace, issuing a warning if not running in a notebook environment.
Usage
Use this module during notebook-based experimentation to monitor memory consumption, particularly when loading large datasets or training memory-intensive recommendation models. Start profiling at the beginning of a session and stop when no longer needed.
Code Reference
Source Location
- Repository: Recommenders
- File: recommenders/utils/notebook_memory_management.py
- Lines: 1-98
Signature
def start_watching_memory()
def stop_watching_memory()
def watch_memory()
def pre_run_cell()
Import
from recommenders.utils.notebook_memory_management import start_watching_memory, stop_watching_memory
I/O Contract
Inputs
start_watching_memory
| Name | Type | Required | Description |
|---|---|---|---|
| (none) | - | - | No parameters; registers IPython event callbacks |
stop_watching_memory
| Name | Type | Required | Description |
|---|---|---|---|
| (none) | - | - | No parameters; unregisters IPython event callbacks |
Outputs
| Name | Type | Description |
|---|---|---|
| console output | str | After each cell execution, prints a formatted string showing the cell identifier, memory delta (MB), elapsed time (seconds), current RAM usage (MB), and total system RAM (MB) |
Usage Examples
Basic Usage
# In a Jupyter notebook cell:
from recommenders.utils.notebook_memory_management import start_watching_memory, stop_watching_memory
# Start profiling
start_watching_memory()
# Output after each cell:
# In [3] used 125.4531 Mb RAM in 2.35s, total RAM usage 1024.55 Mb, total RAM memory 16384.00 Mb
# When done, stop profiling
stop_watching_memory()