Implementation:Pola rs Polars SharedStorage

Knowledge Sources	Pola_rs_Polars Polars Internals
Domains	Memory_Management, Data_Structures
Last Updated	2026-02-09 09:00 GMT

Overview

Concrete tool for reference-counted shared memory storage with multiple ownership models provided by the polars-buffer crate.

Description

SharedStorage<T> is the low-level reference-counted memory backing used by Buffer<T>. It manages the lifecycle of contiguous memory regions through atomic reference counting (similar to Arc<T>) and supports four distinct ownership models via the BackingStorage enum: Vec (owned Rust vector with original capacity tracking), ForeignOwner (externally owned memory via a boxed Any + Send trait object), External (externally managed memory that still needs refcounting), and Leaked (intentionally leaked memory that skips refcounting entirely). The struct uses a VecVTable mechanism to safely handle type-erased vector deallocation, enabling transmutation between Pod types while preserving correct drop behavior.

Usage

Import this type when building low-level buffer primitives that need direct control over memory ownership semantics. SharedStorage<T> is primarily used internally by Buffer<T> and should be preferred through the Buffer API for most use cases. Direct use is appropriate when implementing custom Arrow array types, integrating with FFI boundaries, or when fine-grained control over memory sharing and ownership transfer is required.

Code Reference

Source Location

Repository: Pola_rs_Polars
File: crates/polars-buffer/src/storage.rs
Lines: 1-496

Signature

pub struct SharedStorage<T> {
    inner: NonNull<SharedStorageInner<T>>,
    phantom: PhantomData<SharedStorageInner<T>>,
}

impl<T> SharedStorage<T> {
    pub const fn empty() -> Self;
    pub fn from_static(slice: &'static [T]) -> Self;
    pub unsafe fn from_slice_unchecked(slice: &[T]) -> Self;
    pub fn with_slice<R, F: FnOnce(SharedStorage<T>) -> R>(slice: &[T], f: F) -> R;
    pub fn with_vec<R, F: FnOnce(SharedStorage<T>) -> R>(vec: &mut Vec<T>, f: F) -> R;
    pub unsafe fn from_slice_with_owner<O: Send + 'static>(slice: &[T], owner: O) -> Self;
    pub fn from_owner<O: Send + AsRef<[T]> + 'static>(owner: O) -> Self;
    pub fn from_vec(v: Vec<T>) -> Self;
    pub fn leak(&mut self);
    pub unsafe fn transmute_unchecked<U>(self) -> SharedStorage<U>;
    pub const fn len(&self) -> usize;
    pub const fn is_empty(&self) -> bool;
    pub const fn as_ptr(&self) -> *const T;
    pub fn is_exclusive(&self) -> bool;
    pub fn refcount(&self) -> u64;
    pub fn try_as_mut_slice(&mut self) -> Option<&mut [T]>;
    pub fn try_take_vec(&mut self) -> Option<Vec<T>>;
    pub fn try_as_mut_vec(&mut self) -> Option<SharedStorageAsVecMut<'_, T>>;
    pub fn try_into_vec(mut self) -> Result<Vec<T>, Self>;
}

impl<T: Pod> SharedStorage<T> {
    pub fn try_transmute<U: Pod>(self) -> Result<SharedStorage<U>, Self>;
}

impl SharedStorage<u8> {
    pub fn bytes_from_pod_vec<T: Pod>(v: Vec<T>) -> Self;
}

Import

use polars_buffer::SharedStorage;

I/O Contract

Inputs

Name	Type	Required	Description
v (from_vec)	Vec<T>	Yes	Vector whose memory is taken over by the storage
slice (from_static)	&'static [T]	Yes	Static slice to reference without ownership transfer
owner (from_owner)	O: Send + AsRef<[T]> + 'static	Yes	External object whose memory is referenced

Outputs

Name	Type	Description
SharedStorage<T>	SharedStorage<T>	Reference-counted shared memory with atomic refcounting
try_into_vec returns	Result<Vec<T>, Self>	Ok(Vec) if exclusive Vec-backed, Err(self) otherwise
try_as_mut_slice returns	Option<&mut [T]>	Mutable slice if exclusively owned Vec-backed storage
try_transmute returns	Result<SharedStorage, Self>	Transmuted storage if alignment and size constraints are met

Usage Examples

Creating SharedStorage from Various Sources

use polars_buffer::SharedStorage;

// From a Vec (most common)
let ss = SharedStorage::from_vec(vec![1u32, 2, 3, 4]);
assert_eq!(ss.len(), 4);

// From a static slice (no allocation)
let ss = SharedStorage::from_static(&[10u8, 20, 30]);
assert_eq!(ss.len(), 3);

// Empty storage (const, no allocation)
let ss = SharedStorage::<f64>::empty();
assert!(ss.is_empty());

Exclusive Access and Vec Recovery

use polars_buffer::SharedStorage;

let mut ss = SharedStorage::from_vec(vec![1i64, 2, 3]);

// Exclusive access allows mutation
assert!(ss.is_exclusive());
let slice = ss.try_as_mut_slice().unwrap();
slice[0] = 100;

// Recover the original Vec
let vec = ss.try_into_vec().unwrap();
assert_eq!(vec, vec![100, 2, 3]);

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment