Implementation:Pola rs Polars Credential Provider Configuration

Knowledge Sources	polars Polars User Guide - Cloud Storage
Domains	Cloud_Computing, Credential_Management, Data_Engineering
Last Updated	2026-02-09 10:00 GMT

Overview

Concrete APIs for configuring authentication credentials for cloud object storage (S3, Azure Blob, GCS) in Polars, supporting static options, named profiles, role assumption, and custom credential functions.

Description

The Credential Provider Configuration APIs allow users to authenticate against cloud storage providers when performing scan or read operations on remote data. Polars provides dedicated credential provider classes for AWS and Azure, a storage_options dictionary for static credentials, and a global configuration mechanism for setting default providers across all I/O operations.

Usage

Import polars and configure credentials before calling any scan/read function that references a cloud URI. Use storage_options for quick static configuration, or instantiate a CredentialProviderAWS / CredentialProviderAzure for managed credential lifecycle. Set a global default with pl.Config.set_default_credential_provider for session-wide configuration.

Code Reference

Source Location

Repository: polars
File: docs/source/src/python/user-guide/io/cloud-storage.py
Lines: 47-112

Signature

# Static storage options (passed as dict to scan/read functions)
storage_options = {
    "aws_access_key_id": str,
    "aws_secret_access_key": str,
    "aws_region": str,
}

# AWS credential provider
pl.CredentialProviderAWS(
    profile_name: str = None,
    assume_role: dict = None,  # {"RoleArn": str, "RoleSessionName": str}
) -> CredentialProviderAWS

# Azure credential provider
pl.CredentialProviderAzure(...) -> CredentialProviderAzure

# Global default configuration
pl.Config.set_default_credential_provider(
    provider: CredentialProvider,
) -> None

Import

import polars as pl

I/O Contract

Inputs

Name	Type	Required	Description
storage_options	dict	No	Dictionary with cloud provider credentials (aws_access_key_id, aws_secret_access_key, aws_region)
credential_provider	CredentialProvider or Callable	No	A credential provider instance or callable returning CredentialProviderFunctionReturn
profile_name	str	No	Named AWS profile from ~/.aws/credentials
assume_role	dict	No	Role assumption configuration with RoleArn and RoleSessionName keys

Outputs

Name	Type	Description
CredentialProviderAWS	CredentialProviderAWS	Configured AWS credential provider with managed token lifecycle
CredentialProviderAzure	CredentialProviderAzure	Configured Azure credential provider with managed token lifecycle
None	None	Return value of set_default_credential_provider (configures global state)

Usage Examples

import polars as pl

# Method 1: Using storage_options dict for static credentials
df = pl.scan_parquet(
    "s3://bucket/*.parquet",
    storage_options={
        "aws_access_key_id": "<key>",
        "aws_secret_access_key": "<secret>",
        "aws_region": "us-east-1",
    },
).collect()

# Method 2: Using CredentialProviderAWS with role assumption
lf = pl.scan_parquet(
    "s3://bucket/data.parquet",
    credential_provider=pl.CredentialProviderAWS(
        profile_name="my_profile",
        assume_role={
            "RoleArn": "arn:aws:iam::role/...",
            "RoleSessionName": "polars",
        },
    ),
)

# Method 3: Set global default credential provider
pl.Config.set_default_credential_provider(
    pl.CredentialProviderAWS(profile_name="default")
)

# All subsequent operations use the default provider automatically
df = pl.scan_parquet("s3://bucket/data.parquet").collect()

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment