Implementation:Pola rs Polars Buffer

Knowledge Sources	Pola_rs_Polars Polars Internals
Domains	Memory_Management, Data_Structures
Last Updated	2026-02-09 09:00 GMT

Overview

Concrete tool for thread-safe, reference-counted contiguous memory buffers provided by the polars-buffer crate.

Description

Buffer<T> is the fundamental memory container in Polars' Arrow implementation. It provides a thread-safe, reference-counted, sliceable contiguous memory region equivalent to an Arc<Vec<T>> with several key improvements: O(1) slicing and cloning via shared ownership of the underlying SharedStorage<T>, copy-on-write semantics for converting back to a Vec<T>, and support for externally allocated memory (e.g., from FFI). The buffer tracks a pointer and length into the backing storage, enabling zero-copy slicing operations. It also includes a global pool of 8 MiB of zeroed memory for efficient allocation of zero-initialized buffers.

Usage

Import this type when working with Polars' internal Arrow array implementations that require shared, immutable memory regions. Buffer<T> is used throughout the Polars engine as the backing memory for Arrow arrays, bitmaps, and other columnar data structures. Use it when you need to share data across threads without copying, or when building custom Arrow-compatible data structures.

Code Reference

Source Location

Repository: Pola_rs_Polars
File: crates/polars-buffer/src/buffer.rs
Lines: 1-502

Signature

pub struct Buffer<T> {
    storage: SharedStorage<T>,
    ptr: *const T,
    length: usize,
}

impl<T> Buffer<T> {
    pub const fn new() -> Self;
    pub const fn from_storage(storage: SharedStorage<T>) -> Self;
    pub fn from_static(data: &'static [T]) -> Self;
    pub fn from_vec(data: Vec<T>) -> Self;
    pub fn from_owner<O: Send + AsRef<[T]> + 'static>(owner: O) -> Self;
    pub fn with_slice<R, F: FnOnce(Buffer<T>) -> R>(slice: &[T], f: F) -> R;
    pub fn with_vec<R, F: FnOnce(Buffer<T>) -> R>(vec: &mut Vec<T>, f: F) -> R;
    pub fn into_storage(self) -> SharedStorage<T>;
    pub fn len(&self) -> usize;
    pub fn is_empty(&self) -> bool;
    pub fn is_sliced(&self) -> bool;
    pub fn expand_end_to_storage(self) -> Self;
    pub fn as_slice(&self) -> &[T];
    pub fn sliced<R: RangeBounds<usize>>(mut self, range: R) -> Self;
    pub unsafe fn sliced_unchecked<R: RangeBounds<usize>>(mut self, range: R) -> Self;
    pub fn slice_in_place<R: RangeBounds<usize>>(&mut self, range: R);
    pub unsafe fn slice_in_place_unchecked<R: RangeBounds<usize>>(&mut self, range: R);
    pub fn storage_ptr(&self) -> *const T;
    pub fn offset(&self) -> usize;
    pub unsafe fn set_len(&mut self, len: usize);
    pub fn into_mut(mut self) -> Either<Self, Vec<T>>;
    pub fn get_mut_slice(&mut self) -> Option<&mut [T]>;
    pub fn storage_refcount(&self) -> u64;
    pub fn is_same_buffer(&self, other: &Self) -> bool;
}

impl<T: Pod> Buffer<T> {
    pub fn try_transmute<U: Pod>(mut self) -> Result<Buffer<U>, Self>;
}

impl<T: Clone> Buffer<T> {
    pub fn to_vec(self) -> Vec<T>;
}

impl<T: Zeroable> Buffer<T> {
    pub fn zeroed(length: usize) -> Self;
}

Import

use polars_buffer::Buffer;

I/O Contract

Inputs

Name	Type	Required	Description
data (from_vec)	Vec<T>	Yes	A vector to take ownership of as buffer backing
data (from_static)	&'static [T]	Yes	A static slice to reference without copying
owner (from_owner)	O: Send + AsRef<[T]> + 'static	Yes	An external owner whose memory the buffer references
range (sliced)	R: RangeBounds<usize>	Yes	Range for zero-copy slicing
length (zeroed)	usize	Yes	Number of zero-initialized elements to allocate

Outputs

Name	Type	Description
Buffer<T>	Buffer<T>	Thread-safe shared memory region with O(1) clone and slice
into_mut returns	Either<Buffer<T>, Vec<T>>	Left if shared (cannot get exclusive), Right if exclusive ownership recovered
get_mut_slice returns	Option<&mut [T]>	Mutable slice if buffer has exclusive ownership, None otherwise
as_slice returns	&[T]	Immutable slice view into the buffer
try_transmute returns	Result<Buffer, Buffer<T>>	Transmuted buffer on success, original on alignment/size mismatch

Usage Examples

Basic Buffer Creation and Slicing

use polars_buffer::Buffer;

// Create from a Vec
let buffer: Buffer<u32> = vec![1, 2, 3, 4, 5].into();
assert_eq!(buffer.as_slice(), &[1, 2, 3, 4, 5]);

// O(1) clone (data is shared)
let cloned = buffer.clone();
assert_eq!(cloned.len(), 5);

// Zero-copy slicing
let sliced = buffer.sliced(1..4);
assert_eq!(sliced.as_slice(), &[2, 3, 4]);

Copy-on-Write Semantics

use polars_buffer::Buffer;

// Single-owner buffer can be converted back to Vec
let buffer: Buffer<u32> = vec![10, 20, 30].into();
let vec = buffer.into_mut().right().unwrap();
assert_eq!(vec, vec![10, 20, 30]);

// Shared buffer cannot be converted
let buffer: Buffer<u32> = vec![10, 20, 30].into();
let _clone = buffer.clone();
assert!(buffer.into_mut().is_left());

Zero-Initialized Buffers

use polars_buffer::Buffer;

// Efficiently create zero-filled buffers (uses global zero pool for small sizes)
let zeros: Buffer<u64> = Buffer::zeroed(1024);
assert_eq!(zeros.len(), 1024);
assert!(zeros.iter().all(|&x| x == 0));

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment