Implementation:Predibase Lorax Merge Adapters Function

Knowledge Sources	LoRAX
Domains	Model_Merging, Model_Serving
Last Updated	2026-02-08 02:00 GMT

Overview

Concrete tool for merging multiple loaded LoRA adapters into a single adapter provided by the merge_adapters function.

Description

The merge_adapters() function takes a list of (ModuleMap, LoraConfig) tuples and an AdapterParameters protobuf message containing merge settings. It validates that all LoRA configs have the same rank, selects the merge strategy from the registry, and applies it to each weight tensor across all adapters. The result is a single merged ModuleMap and merged LoraConfig.

The load_and_merge_adapters() function in adapter.py orchestrates the full pipeline: loading each adapter's weights, then calling merge_adapters().

Usage

Called internally when the server processes a request with merged_adapters. Not called directly by users.

Code Reference

Source Location

• Repository: LoRAX
• File: server/lorax_server/utils/merges/strategies.py (Lines: 108-158)
• File: server/lorax_server/utils/adapter.py (Lines: 58-73)

Signature

def merge_adapters(
    adapters: List[Tuple["ModuleMap", "LoraConfig"]],
    merge_params: AdapterParameters,
) -> Tuple["ModuleMap", "LoraConfig"]:
    """
    Merge multiple loaded LoRA adapters into a single adapter.

    Args:
        adapters: List of (module_map, lora_config) tuples
        merge_params: Protobuf message with merge strategy, weights, density

    Returns:
        Tuple of (merged_module_map, merged_lora_config)
    """

def load_and_merge_adapters(
    model_id: str,
    adapter_parameters: AdapterParameters,
    adapter_source: str,
    adapter_index: int,
    weight_names: Tuple[str],
    api_token: str,
) -> Tuple[ModuleMap, AdapterConfig, Set[str], PreTrainedTokenizer]:
    """Load multiple adapters and merge them."""

Import

from lorax_server.utils.merges.strategies import merge_adapters
from lorax_server.utils.adapter import load_and_merge_adapters

I/O Contract

Inputs

Name	Type	Required	Description
adapters	List[Tuple[ModuleMap, LoraConfig]]	Yes	Loaded adapter weights and configs
merge_params	AdapterParameters	Yes	Protobuf with weights, strategy, density

Outputs

Name	Type	Description
merged_module_map	ModuleMap	Combined weight tensors for all layers
merged_lora_config	LoraConfig	Merged config with union of target modules

Usage Examples

Internal Merge Pipeline

from lorax_server.utils.merges.strategies import merge_adapters

# After loading two adapters
adapters = [
    (module_map_a, lora_config_a),  # adapter A
    (module_map_b, lora_config_b),  # adapter B
]

# AdapterParameters from protobuf with merge settings
merge_params = AdapterParameters(
    weights=[0.6, 0.4],
    merge_strategy=MergeStrategyEnum.TIES,
    density=0.7,
    majority_sign_method=MajoritySignMethodEnum.TOTAL,
)

merged_map, merged_config = merge_adapters(adapters, merge_params)
# merged_map contains combined weight tensors
# merged_config has union of target modules from both adapters

Related Pages

Implements Principle

• Principle:Predibase_Lorax_Runtime_Adapter_Merging