Implementation:Langchain ai Langchain HuggingFaceEndpoint

Knowledge Sources	Langchain_ai_Langchain
Domains	LLM Framework, Hugging Face, Text Completion, Inference API
Last Updated	2026-02-11 00:00 GMT

Overview

The HuggingFaceEndpoint class provides a LangChain LLM wrapper for Hugging Face's Inference API and Inference Endpoints, supporting text generation with synchronous, asynchronous, and streaming interfaces.

Description

This module implements HuggingFaceEndpoint as a subclass of LangChain's LLM base class in the langchain-huggingface partner package. It provides a text-in/text-out interface to any model hosted on Hugging Face that supports text generation tasks (text-generation, text2text-generation, summarization, conversational). The class uses the huggingface_hub library's InferenceClient and AsyncInferenceClient for API communication, supporting both Hugging Face's serverless Inference API (via repo_id) and dedicated Inference Endpoints (via endpoint_url). It also supports third-party inference providers (e.g., Novita, Cerebras) through the provider parameter.

Usage

Import HuggingFaceEndpoint from langchain_huggingface and instantiate with either a repo_id (for Hugging Face hosted models) or an endpoint_url (for dedicated endpoints). For chat-style interactions, wrap with ChatHuggingFace.

Code Reference

Source Location

Repository: Langchain_ai_Langchain
File: libs/partners/huggingface/langchain_huggingface/llms/huggingface_endpoint.py
Lines: 1-455

Signature

class HuggingFaceEndpoint(LLM):
    endpoint_url: str | None = None
    repo_id: str | None = None
    provider: str | None = None
    huggingfacehub_api_token: str | None = Field(
        default_factory=from_env("HUGGINGFACEHUB_API_TOKEN", default=None)
    )
    max_new_tokens: int = 512
    top_k: int | None = None
    top_p: float | None = 0.95
    typical_p: float | None = 0.95
    temperature: float | None = 0.8
    repetition_penalty: float | None = None
    return_full_text: bool = False
    truncate: int | None = None
    stop_sequences: list[str] = Field(default_factory=list)
    seed: int | None = None
    timeout: int = 120
    streaming: bool = False
    do_sample: bool = False
    watermark: bool = False
    server_kwargs: dict[str, Any] = Field(default_factory=dict)
    model_kwargs: dict[str, Any] = Field(default_factory=dict)
    model: str
    client: Any = None
    async_client: Any = None
    task: str | None = None

Import

from langchain_huggingface import HuggingFaceEndpoint
# or
from langchain_huggingface.llms.huggingface_endpoint import HuggingFaceEndpoint

I/O Contract

Inputs

Name	Type	Required	Description
repo_id	str or None	No*	Hugging Face model repository ID (e.g., `"mistralai/Mistral-Nemo-Base-2407"`)
endpoint_url	str or None	No*	URL of a deployed Inference Endpoint
model	str	No*	Model identifier (resolved from `repo_id`, `endpoint_url`, or env `HF_INFERENCE_ENDPOINT`)
provider	str or None	No	Third-party inference provider name (e.g., `"novita"`, `"cerebras"`). Defaults to auto-selection
huggingfacehub_api_token	str or None	No	API token. Env: `HUGGINGFACEHUB_API_TOKEN` or `HF_TOKEN`
max_new_tokens	int	No	Maximum generated tokens (default: 512)
top_k	int or None	No	Top-k filtering for token selection
top_p	float or None	No	Nucleus sampling threshold (default: 0.95)
typical_p	float or None	No	Typical decoding mass (default: 0.95)
temperature	float or None	No	Logits distribution temperature (default: 0.8)
repetition_penalty	float or None	No	Repetition penalty (1.0 = no penalty)
return_full_text	bool	No	Whether to prepend prompt to output (default: False)
truncate	int or None	No	Truncate input tokens to this size
stop_sequences	list[str]	No	Stop sequences for generation (default: empty)
seed	int or None	No	Random sampling seed
timeout	int	No	Request timeout in seconds (default: 120)
streaming	bool	No	Whether to stream tokens (default: False)
do_sample	bool	No	Whether to activate logits sampling (default: False)
watermark	bool	No	Apply watermarking (default: False)
task	str or None	No	Task type (e.g., `"text-generation"`)
server_kwargs	dict	No	Additional kwargs for `InferenceClient` constructor
model_kwargs	dict	No	Additional kwargs for model calls

* At least one of repo_id, endpoint_url, or model must be provided (or HF_INFERENCE_ENDPOINT env var).

Outputs

Name	Type	Description
result	str	Generated text string (for `_call` and `_acall`)
chunk	GenerationChunk	Individual token chunks (for `_stream` and `_astream`)

Key Mechanisms

Model Resolution

The build_extra model validator resolves the model field from multiple sources (only one allowed):

Explicit model parameter
endpoint_url parameter
repo_id parameter
HF_INFERENCE_ENDPOINT environment variable

Client Initialization

The validate_environment validator creates both synchronous (InferenceClient) and asynchronous (AsyncInferenceClient) clients from huggingface_hub. It introspects the client constructors to pass only supported kwargs from server_kwargs, logging warnings for unsupported parameters.

Stop Sequence Handling

Stop sequences are accumulated from both the class-level stop_sequences and runtime stop parameters. In the _call method, any stop sequence found at the end of the generated text is stripped. In the _stream and _astream methods, generation stops immediately when a stop sequence is detected in a token.

Extra Parameter Handling

The build_extra model validator uses extra="forbid" combined with dynamic parameter extraction: parameters not in the class fields are transferred to model_kwargs with a warning, enabling flexible pass-through of provider-specific parameters.

Usage Examples

Basic Usage

from langchain_huggingface import HuggingFaceEndpoint

model = HuggingFaceEndpoint(
    repo_id="mistralai/Mistral-Nemo-Base-2407",
    max_new_tokens=512,
    temperature=0.01,
)

response = model.invoke("What is Deep Learning?")
print(response)

With Third-Party Provider

from langchain_huggingface import HuggingFaceEndpoint

model = HuggingFaceEndpoint(
    repo_id="mistralai/Mistral-Nemo-Base-2407",
    provider="novita",
    max_new_tokens=100,
    do_sample=False,
)

response = model.invoke("Explain quantum computing briefly.")
print(response)

Streaming

from langchain_huggingface import HuggingFaceEndpoint

model = HuggingFaceEndpoint(
    repo_id="mistralai/Mistral-Nemo-Base-2407",
    streaming=True,
    max_new_tokens=256,
)

for chunk in model.stream("Write a poem about AI."):
    print(chunk, end="")

With Dedicated Endpoint

from langchain_huggingface import HuggingFaceEndpoint

model = HuggingFaceEndpoint(
    endpoint_url="http://localhost:8010/",
    max_new_tokens=512,
    top_k=10,
    top_p=0.95,
    temperature=0.01,
    repetition_penalty=1.03,
)

response = model.invoke("What is Deep Learning?")
print(response)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment