Implementation:Turboderp org Exllamav2 Shard

Knowledge Sources	Turboderp_org_Exllamav2
Domains	Model_Management, Utilities
Last Updated	2026-02-15 00:00 GMT

Overview

CLI utility for splitting a single safetensors model file into multiple shards of a specified size, generating a corresponding index JSON file with a weight map.

Description

shard.py is a standalone script that takes a large safetensors file and splits it into smaller shard files, following the Hugging Face safetensors sharding convention.

Processing pipeline:

1. Tensor scanning: Opens the input file using safe_open and iterates over all tensor keys. For each tensor, the helper function _tsize(st, key) computes the byte size by multiplying the number of elements by the dtype size (supports I32, I16, F16, F32). Tensors are grouped into shards: a new shard is started whenever adding the next tensor would exceed the specified shard size.

2. Shard writing: For each shard, the script re-opens the source file, reads the assigned tensors via f.get_tensor(key), collects them in a dictionary, and writes them to a new safetensors file using save_file. Output filenames follow the pattern <input_base>-NNNNN-of-NNNNN.safetensors.

3. Index generation: After all shards are written, the script creates an index JSON file at <input_file>.index.json containing:

• • metadata.total_size -- Total byte size of all tensors.
  • weight_map -- Dictionary mapping each tensor key to its shard filename.

The _tsize helper calculates tensor sizes without loading the full tensor data, using get_slice to access shape and dtype metadata efficiently.

Usage

This tool is used when a model's safetensors file is too large for certain storage or transfer constraints. It is also useful for converting single-file models to the multi-shard format expected by some loading frameworks. The generated index file allows loaders to find which shard contains each weight tensor.

Code Reference

Source Location

• Repository: Turboderp_org_Exllamav2
• File: util/shard.py
• Lines: 1-84

Signature

# CLI argument parser
parser = argparse.ArgumentParser(description="Split .safetensors file into shards")
parser.add_argument("input_file", type=str, help="Path to input file")
parser.add_argument("shard_size", type=int, help="Shard size in megabytes")

# Internal helper
def _tsize(st, key) -> int:
    ...

Import

# Script executed directly via CLI
python util/shard.py model.safetensors 4096

I/O Contract

Argument	Type	Required	Description
input_file	str (positional)	Yes	Path to the input .safetensors file
shard_size	int (positional)	Yes	Maximum shard size in megabytes (MB)

Output File	Pattern	Description
Shard files	<base>-00001-of-NNNNN.safetensors	Individual shard files containing subsets of tensors
Index file	<input_file>.index.json	JSON with total_size metadata and weight_map

Supported Dtypes	Bytes per Element
I32	4
I16	2
F16	2
F32	4

Usage Examples

# Split a 20GB model file into 4GB shards
# python util/shard.py /models/llama-7b/model.safetensors 4096
#
# Output:
#   /models/llama-7b/model-00001-of-00005.safetensors
#   /models/llama-7b/model-00002-of-00005.safetensors
#   /models/llama-7b/model-00003-of-00005.safetensors
#   /models/llama-7b/model-00004-of-00005.safetensors
#   /models/llama-7b/model-00005-of-00005.safetensors
#   /models/llama-7b/model.safetensors.index.json

# Split into 2GB shards
# python util/shard.py /models/mixtral/model.safetensors 2048

# The generated index.json structure:
# {
#   "metadata": { "total_size": 14483456000 },
#   "weight_map": {
#     "model.embed_tokens.weight": "model-00001-of-00005.safetensors",
#     "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00005.safetensors",
#     ...
#   }
# }

Related Pages

• Turboderp_org_Exllamav2_Model_Diff -- Model comparison tool that may work with sharded models
• Turboderp_org_Exllamav2_FPx_Quantization -- Quantization that produces model files needing sharding