Implementation:Hiyouga LLaMA Factory Misc Utils

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Utilities, Device Management
Last Updated	2026-02-06 19:00 GMT

Overview

Concrete miscellaneous utility functions for device management, version checking, and model helpers provided by LLaMA Factory.

Description

This module provides a collection of utility functions and classes used throughout the LLaMA Factory codebase. The utilities fall into several categories:

Device Management:

• get_current_device() -- Returns the current device (CUDA, XPU, NPU, MPS, or CPU) based on LOCAL_RANK
• get_device_name() -- Returns the device name string
• get_device_count() -- Returns the number of available accelerator devices
• get_torch_device() -- Returns the torch device namespace for the detected backend
• get_current_memory() -- Returns (available, total) memory in bytes
• get_peak_memory() -- Returns (allocated, reserved) peak memory
• is_accelerator_available() -- Checks if any hardware accelerator is available
• torch_gc() -- Forces garbage collection and empties device cache

Version and Dependency Checking:

• check_version(requirement) -- Validates a package version requirement
• check_dependencies() -- Validates all required package versions (transformers, datasets, accelerate, peft, trl)

Model Utilities:

• count_parameters(model) -- Returns (trainable, total) parameter counts with 4-bit quantization awareness
• infer_optim_dtype(model_dtype) -- Infers optimal dtype (bf16, fp16, or fp32) based on hardware
• get_logits_processor() -- Returns a LogitsProcessorList that removes NaN/Inf logits
• calculate_tps(dataset, metrics, stage) -- Calculates effective tokens per second
• numpify(inputs) -- Converts torch tensors to numpy arrays with bfloat16 handling

Hub Utilities:

• try_download_model_from_other_hub(model_args) -- Downloads models from ModelScope or OpenMind hubs
• use_modelscope(), use_openmind(), use_ray(), use_kt() -- Environment variable checks

Other Utilities:

• AverageMeter class -- Running average tracker for metrics
• has_tokenized_data(path) -- Checks if a pre-tokenized dataset exists
• skip_check_imports() -- Patches transformers to skip flash attention import checks
• find_available_port() -- Finds a free TCP port
• fix_proxy() -- Fixes proxy settings for Gradio UI
• is_env_enabled(env_var) -- Checks if an environment variable is truthy

Usage

This module is imported by virtually every other module in the framework. Device management functions are used during model loading, version checking runs at startup, and hub utilities are used for model downloading.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: src/llamafactory/extras/misc.py
• Lines: 1-365

Signature

class AverageMeter:
    def __init__(self) -> None: ...
    def reset(self) -> None: ...
    def update(self, val: float, n: int = 1) -> None: ...

def check_version(requirement: str, mandatory: bool = False) -> None: ...
def check_dependencies() -> None: ...
def calculate_tps(dataset: list[dict[str, Any]], metrics: dict[str, float], stage: Literal["sft", "rm"]) -> float: ...
def count_parameters(model: "torch.nn.Module") -> tuple[int, int]: ...
def get_current_device() -> "torch.device": ...
def get_device_name() -> str: ...
def get_torch_device() -> Any: ...
def get_device_count() -> int: ...
def get_logits_processor() -> "LogitsProcessorList": ...
def get_current_memory() -> tuple[int, int]: ...
def get_peak_memory() -> tuple[int, int]: ...
def has_tokenized_data(path: "os.PathLike") -> bool: ...
def infer_optim_dtype(model_dtype: Optional["torch.dtype"]) -> "torch.dtype": ...
def is_accelerator_available() -> bool: ...
def is_env_enabled(env_var: str, default: str = "0") -> bool: ...
def numpify(inputs: Union["NDArray", "torch.Tensor"]) -> "NDArray": ...
def skip_check_imports() -> None: ...
def torch_gc() -> None: ...
def try_download_model_from_other_hub(model_args: "ModelArguments") -> str: ...
def use_modelscope() -> bool: ...
def use_openmind() -> bool: ...
def use_ray() -> bool: ...
def use_kt() -> bool: ...
def find_available_port() -> int: ...
def fix_proxy(ipv6_enabled: bool = False) -> None: ...

Import

from llamafactory.extras.misc import (
    get_current_device,
    count_parameters,
    check_dependencies,
    infer_optim_dtype,
    torch_gc,
    AverageMeter,
)

I/O Contract

Inputs (key functions)

Name	Type	Required	Description
requirement	str	Yes	Version requirement string for check_version (e.g., "transformers>=4.51.0")
model	torch.nn.Module	Yes	Model for count_parameters
model_dtype	torch.dtype	No	Model dtype for infer_optim_dtype (None triggers auto-detection)
model_args	ModelArguments	Yes	Model arguments for try_download_model_from_other_hub
env_var	str	Yes	Environment variable name for is_env_enabled

Outputs (key functions)

Name	Type	Description
get_current_device	torch.device	Current accelerator device (e.g., cuda:0, npu:0, cpu)
count_parameters	tuple[int, int]	(trainable_params, all_params) with 4-bit quantization awareness
infer_optim_dtype	torch.dtype	Optimal dtype: bfloat16, float16, or float32
get_current_memory	tuple[int, int]	(available_bytes, total_bytes) for current device
get_peak_memory	tuple[int, int]	(peak_allocated_bytes, peak_reserved_bytes)

Usage Examples

from llamafactory.extras.misc import (
    get_current_device,
    count_parameters,
    infer_optim_dtype,
    torch_gc,
    AverageMeter,
    check_dependencies,
)

# Check all required dependencies at startup
check_dependencies()

# Get the device for model placement
device = get_current_device()  # e.g., torch.device("cuda:0")

# Count model parameters
trainable, total = count_parameters(model)
print(f"Trainable: {trainable:,} / Total: {total:,}")

# Infer optimal training dtype
dtype = infer_optim_dtype(model.dtype)  # torch.bfloat16 if available

# Track training metrics
meter = AverageMeter()
for loss in losses:
    meter.update(loss.item())
print(f"Average loss: {meter.avg:.4f}")

# Free GPU memory after inference
torch_gc()

Related Pages

• Hiyouga_LLaMA_Factory_Constants - Companion module with constants and enums
• Hiyouga_LLaMA_Factory_Data_Loader - Uses has_tokenized_data and check_version
• Hiyouga_LLaMA_Factory_HfChatEngine - Uses device management and memory utilities