Implementation:Iterative Dvc Rwlock

Knowledge Sources	Iterative_Dvc
Domains	Concurrency, File_Locking
Last Updated	2026-02-10 10:00 GMT

Overview

Rwlock implements a process-aware reader-writer file lock for DVC. It is implemented in dvc/rwlock.py (221 lines) and provides a context manager that coordinates concurrent read and write access to file paths across separate DVC processes.

from dvc.rwlock import rwlock

The lock state is persisted as a JSON file (rwlock) in the DVC temporary directory, protected by a lower-level file lock (rwlock.lock). Stale entries from dead processes are automatically detected and cleaned via PID validation.

Public Function

rwlock()

A context manager that acquires reader-writer locks on specified file paths.

Signature:

@contextmanager
def rwlock(tmp_dir, fs, cmd, read, write, hardlink):

Parameters:

Parameter	Type	Default	Description
tmp_dir	str	required	Directory where the rwlock file is stored
fs	FileSystem	required	Filesystem instance for tmp_dir
cmd	str	required	Command name acquiring the lock (stored for diagnostics)
read	list[str]	required	File paths to acquire read locks on
write	list[str]	required	File paths to acquire write locks on
hardlink	bool	required	Whether to use hardlink-based locking for the guard lock

Exceptions:

• dvc.lock.LockError -- raised when requested paths are blocked by another active process
• RWLockFileCorruptedError -- raised when the JSON lock file cannot be parsed
• RWLockFileFormatError -- raised when the lock file structure does not pass schema validation

Usage:

with rwlock(tmp_dir, fs, "dvc push", read=["/data/train.csv"], write=[], hardlink=True):
    # Safe to read /data/train.csv here
    pass

Lock File Schema

The JSON lock file follows this schema (validated by voluptuous):

INFO_SCHEMA = {Required("pid"): int, Required("cmd"): str}

SCHEMA = Schema({
    Optional("write", default={}): {str: INFO_SCHEMA},
    Optional("read", default={}): {str: [INFO_SCHEMA]},
})

Structure:

• write: A dictionary mapping file paths to a single {"pid": int, "cmd": str} entry (exclusive lock).
• read: A dictionary mapping file paths to a list of {"pid": int, "cmd": str} entries (shared lock).

Lock Semantics

Requested	Blocked By	Reason
Read on path P	Active writer on any overlapping path	Cannot read while another process is writing
Write on path P	Active writer on any overlapping path	Cannot write while another process is writing
Write on path P	Active reader on any overlapping path	Cannot write while another process is reading

Path overlap is determined by localfs.overlaps(), which checks whether one path is a prefix of the other.

Internal Functions

_edit_rwlock()

A context manager that provides safe read-modify-write access to the JSON lock file. Acquires the guard lock (rwlock.lock), reads and validates the JSON structure, yields the parsed lock dict for modification, and writes the updated dict back on exit.

@contextmanager
def _edit_rwlock(lock_dir, fs, hardlink):

_check_blockers()

Scans the lock file for entries that would block the requested operation. For each potential blocker:

1. If the blocker PID matches the current process info, it is skipped (self-overlap is allowed).
2. If psutil.pid_exists(pid) returns True, the blocker is active and a LockError is raised.
3. If the PID no longer exists, the stale entry is logged as a warning and auto-removed from the lock file.

_acquire_read()

Adds the current process info to the read list for each requested path, unless already present.

_acquire_write()

Sets the current process info as the writer for each requested path, unless already set.

_release_write()

Removes write entries for the specified paths, asserting they match the current process info.

_release_read()

Removes the current process info from the read list for the specified paths.

_infos_to_str()

Formats a list of blocker info dicts into a human-readable string for error messages.

Execution Flow

1. rwlock() creates an info dict with the current PID and command name.
2. _edit_rwlock() acquires the guard file lock and loads the JSON state.
3. _check_blockers() is called twice: first checking writers against all requested paths (read + write), then checking readers against write paths only.
4. Stale entries (dead PIDs) are automatically removed during blocker checking.
5. If no blockers remain, _acquire_read() and _acquire_write() register the current process.
6. The lock file is written back and the guard lock is released.
7. The context body executes.
8. On exit, _edit_rwlock() is re-acquired and _release_write() / _release_read() remove the current process's entries.

Error Classes

RWLockFileCorruptedError

Raised when the lock file exists but contains invalid JSON.

RWLockFileFormatError

Raised when the lock file is valid JSON but does not conform to the expected schema.

Module Constants

Constant	Value	Description
RWLOCK_FILE	"rwlock"	Name of the JSON lock state file
RWLOCK_LOCK	"rwlock.lock"	Name of the guard lock file

Key Design Decisions

• JSON-based persistence: The lock state is stored as a human-readable JSON file, making it easy to inspect and manually recover from stuck locks. The error message explicitly tells users they can remove the file manually.
• PID-based stale detection: Using psutil.pid_exists() to detect dead processes allows automatic recovery from crashes without requiring manual intervention.
• Two-level locking: The guard lock (rwlock.lock) protects the JSON file from concurrent modification, while the JSON content tracks logical read/write ownership across paths.
• Path overlap semantics: Using localfs.overlaps() for path matching means that a write lock on /data blocks reads on /data/train.csv and vice versa, preventing partial-directory conflicts.
• Non-thread-safe design: The context manager is explicitly described as non-thread-safe. It coordinates between separate processes, not threads within the same process.

Dependencies

• psutil -- process existence checking for stale lock detection
• voluptuous -- JSON schema validation (Schema, Required, Optional, Invalid)
• dvc.lock.make_lock -- file-based guard lock creation
• dvc.lock.LockError -- error raised when paths are blocked
• dvc.fs.localfs -- local filesystem instance for path overlap checking
• dvc.exceptions.DvcException -- base exception class for custom errors