Principle:Predibase Lorax Runtime Adapter Merging

Knowledge Sources	TIES-Merging Model Soups
Domains	Model_Merging, Model_Serving
Last Updated	2026-02-08 02:00 GMT

Overview

A runtime composition mechanism that loads multiple LoRA adapters, validates their compatibility, and merges their weight tensors per-layer using a configured merge strategy before inference.

Description

Runtime Adapter Merging enables on-the-fly combination of multiple fine-tuned LoRA adapters at inference time. The process:

1. Load: Download and load each adapter's weights and LoRA configuration
2. Validate: Ensure all adapters have compatible ranks and valid target modules
3. Merge: Apply the selected merge strategy (linear, TIES, DARE) to each weight tensor across all adapters
4. Result: Produce a single merged ModuleMap and LoraConfig that behaves as a single adapter

This allows users to create task-specific adapter ensembles without any additional training.

Usage

Use when a request specifies merged_adapters with multiple adapter IDs. The server loads each adapter, merges them according to the strategy, and runs inference with the merged result.

Theoretical Basis

Pseudo-code:

# Runtime merge pipeline
adapters = [(load(id), load_config(id)) for id in adapter_ids]
validate_configs([config for _, config in adapters])  # Same rank required
strategy = strategy_registry[merge_strategy](density, majority_sign_method)
for weight_name in all_weight_names:
    tensors = [adapter[weight_name] for adapter in adapters]
    merged[weight_name] = strategy.merge(tensors, weights)
merged_config = merge_lora_configs([c for _, c in adapters])

Related Pages

Implemented By

• Implementation:Predibase_Lorax_Merge_Adapters_Function