Implementation:Hpcaitech ColossalAI ColossalCloudLLM

Knowledge Sources	Hpcaitech_ColossalAI
Domains	NLP, LLM Integration, Cloud Services
Last Updated	2026-02-09 00:00 GMT

Overview

ColossalCloudLLM is a LangChain-compatible LLM wrapper that integrates with language models hosted on the ColossalCloud Platform via REST API calls.

Description

This class extends LangChain's LLM base class to provide access to LLMs running on the ColossalCloud Platform. It handles authentication configuration via URL and Host environment variables, manages generation parameters (max_new_tokens, top_k, top_p, temperature, repetition_penalty), and sends text completion requests via HTTP POST. The response text is post-processed with optional stop-word truncation before being returned.

Usage

Use ColossalCloudLLM when you need to integrate a ColossalCloud-hosted language model into a LangChain-based pipeline for ColossalQA applications. You must set the URL and HOST environment variables or pass them explicitly through set_auth_config.

Code Reference

Source Location

• Repository: Hpcaitech_ColossalAI
• File: applications/ColossalQA/colossalqa/local/colossalcloud_llm.py
• Lines: 1-118

Signature

class ColossalCloudLLM(LLM):
    n: int
    gen_config: dict = None
    auth_config: dict = None
    valid_gen_para: list = ["max_new_tokens", "top_k", "top_p", "temperature", "repetition_penalty"]

    def __init__(self, gen_config=None, **kwargs):
        ...

    def set_auth_config(self, **kwargs):
        ...

    def _call(self, prompt: str, stop=None, **kwargs: Any) -> str:
        ...

    def text_completion(self, prompt, gen_config, auth_config):
        ...

Import

from colossalqa.local.colossalcloud_llm import ColossalCloudLLM

I/O Contract

Inputs

Name	Type	Required	Description
n	int	Yes	Instance identifier parameter
gen_config	dict	No	Generation configuration dictionary; must include "max_new_tokens" if provided (default: {"max_new_tokens": 50})
prompt	str	Yes	The text prompt to send to the model (used in _call)
stop	list	No	List of stop words to truncate the generated response
max_new_tokens	int	No	Maximum number of tokens to generate (passed via gen_config or kwargs)
top_k	int	No	Top-k sampling parameter (passed via gen_config or kwargs)
top_p	float	No	Top-p nucleus sampling parameter (passed via gen_config or kwargs)
temperature	float	No	Sampling temperature (passed via gen_config or kwargs)
repetition_penalty	float	No	Penalty for repeated tokens (passed via gen_config or kwargs)

Outputs

Name	Type	Description
return	str	The text generated by the ColossalCloud-hosted model, optionally truncated at stop words

Environment Variables

Variable	Description
URL	The endpoint URL for the ColossalCloud API
HOST	The host header value for the ColossalCloud API

Usage Examples

import os
from colossalqa.local.colossalcloud_llm import ColossalCloudLLM

# Set environment variables
os.environ["URL"] = "https://api.colossalcloud.example.com/v1/completions"
os.environ["HOST"] = "api.colossalcloud.example.com"

# Configure generation parameters
gen_config = {
    "max_new_tokens": 100,
    "top_p": 0.9,
    "temperature": 0.5,
    "repetition_penalty": 2,
}

# Create and configure the LLM
llm = ColossalCloudLLM(n=1, gen_config=gen_config)
llm.set_auth_config()

# Generate text
response = llm(prompt="What is ColossalAI?")
print(response)

Related Pages

• Environment:Hpcaitech_ColossalAI_ColossalQA_RAG_Environment