Implementation:Neuml Txtai Storage Cloud

Knowledge Sources	Neuml_Txtai
Domains	Cloud Storage, Object Storage, Infrastructure
Last Updated	2026-02-10 01:00 GMT

Overview

Concrete tool for syncing embeddings indexes to and from cloud object storage provided by txtai.

Description

The ObjectStorage class is a Cloud provider implementation backed by Apache libcloud. It enables saving and loading txtai embeddings indexes from any object storage service that libcloud supports, including AWS S3, Google Cloud Storage, Azure Blob Storage, MinIO, and many others. On load, it can download either a single archive file or all objects in a container (optionally filtered by prefix). On save, it uploads all local files to the container, with optional prefix-based namespacing. The class automatically creates containers if they do not exist during save operations.

Usage

Use ObjectStorage when you need to persist txtai embeddings indexes in cloud object storage. The provider is configured via the txtai cloud configuration by specifying a libcloud-supported provider name (e.g., "s3", "google_storage"), along with access credentials, container name, and optional prefix. This enables durable and scalable storage of embeddings indexes across cloud providers.

Code Reference

Source Location

• Repository: Neuml_Txtai
• File: src/python/txtai/cloud/storage.py

Signature

class ObjectStorage(Cloud):
    @staticmethod
    def isprovider(provider)
    def __init__(self, config)
    def metadata(self, path=None)
    def load(self, path=None)
    def save(self, path)
    def objectname(self, name)

Import

from txtai.cloud.storage import ObjectStorage

I/O Contract

Inputs

Name	Type	Required	Description
config	dict	Yes	Cloud configuration dictionary with keys: "provider" (libcloud driver name), "container" (bucket/container name), optional "key"/"secret" (or ACCESS_KEY/ACCESS_SECRET env vars), optional "host", "port", "region", "token", "prefix"
provider	str	Yes (isprovider)	Provider name to check against libcloud DRIVERS registry
path	str	No	Local file path; if an archive file, targets that specific object; otherwise targets the entire container

Outputs

Name	Type	Description
isprovider result	bool	True if the provider name is found in the libcloud drivers registry
metadata result	object or None	Object metadata (for archive paths) or container metadata; None if not found
load result	str	Local path where files were downloaded

Usage Examples

from txtai.cloud.storage import ObjectStorage

# Configure S3-compatible object storage
config = {
    "provider": "s3",
    "container": "my-txtai-indexes",
    "key": "AKIAIOSFODNN7EXAMPLE",
    "secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "region": "us-east-1",
    "prefix": "production/v1"
}

storage = ObjectStorage(config)

# Check if this is a valid object storage provider
if ObjectStorage.isprovider("s3"):
    print("S3 provider is available")

# Check if container exists
if storage.exists():
    # Load all files from container to local path
    local_path = storage.load("/tmp/index")

# Save local index files to cloud storage
storage.save("/path/to/local/index")

Related Pages

• Environment:Neuml_Txtai_Python_Core_Dependencies