Implementation:Hiyouga LLaMA Factory V1 Kernel Base

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	Machine Learning, Hardware Optimization
Last Updated	2026-02-06 19:00 GMT

Overview

BaseKernel is an abstract base class that defines the contract for all hardware-specific kernel implementations in the LLaMA-Factory v1 kernel plugin system.

Description

The BaseKernel ABC provides the foundational interface for custom kernel replacements that optimize model operations on specific hardware accelerators. It defines class attributes for kernel identification (_kernel_id) and target device type (_device), along with methods for dependency checking and kernel application. Subclasses must implement the abstract apply method to perform the actual model patching. The check_deps method validates that the current hardware accelerator matches the kernel's target device type before allowing application.

Usage

Use BaseKernel as a parent class when implementing new hardware-specific kernel optimizations. Subclass it, set _kernel_id and _device, then implement apply to monkey-patch model modules with optimized forward methods. This class should not be instantiated directly.

Code Reference

Source Location

Repository: Hiyouga_LLaMA_Factory
File: src/llamafactory/v1/plugins/model_plugins/kernels/base.py
Lines: 1-87

Signature

class BaseKernel(ABC):
    _kernel_id: Any = ""
    _device: DeviceType = DeviceType.CPU

    @classmethod
    def get_kernel_id(cls) -> str: ...

    @classmethod
    def get_device(cls) -> str: ...

    @classmethod
    def check_deps(cls) -> bool: ...

    @classmethod
    @abstractmethod
    def apply(cls, **kwargs) -> HFModel: ...

Import

from llamafactory.v1.plugins.model_plugins.kernels.base import BaseKernel

I/O Contract

Inputs

Name	Type	Required	Description
_kernel_id	Any (hashable)	Yes (class attr)	Unique identifier for the kernel implementation
_device	DeviceType	Yes (class attr)	Target device type (e.g., DeviceType.CPU, DeviceType.NPU, DeviceType.CUDA)
**kwargs (apply)	dict	Yes	Keyword arguments containing the model instance and kernel configuration

Outputs

Name	Type	Description
get_kernel_id	str	The unique kernel identifier string
get_device	str	The device type string associated with the kernel
check_deps	bool	True if the kernel's required device matches the current accelerator
apply	HFModel	The model with the kernel optimization applied

Usage Examples

# Defining a custom kernel subclass
from llamafactory.v1.plugins.model_plugins.kernels.base import BaseKernel
from llamafactory.v1.accelerator.helper import DeviceType
from llamafactory.v1.plugins.model_plugins.kernels.registry import register_kernel

@register_kernel
class MyCustomKernel(BaseKernel):
    _kernel_id = "my_custom_op"
    _device = DeviceType.CUDA

    @classmethod
    def apply(cls, **kwargs):
        super().apply(**kwargs)  # checks deps, raises if not available
        model = kwargs.get("model")
        # Apply custom kernel optimizations to the model
        return model

# Checking kernel availability
if MyCustomKernel.check_deps():
    model = MyCustomKernel.apply(model=model)

Related Pages

Hiyouga_LLaMA_Factory_V1_Kernel_Registry - Registry that manages kernel registration and lookup
Hiyouga_LLaMA_Factory_V1_NPU_RMSNorm - Concrete kernel implementation for NPU RMSNorm
Hiyouga_LLaMA_Factory_V1_Types - Type definitions including HFModel

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment