Implementation:Run llama Llama index LLM Utils

Overview

This module provides utility functions for LLM resolution and JSON parsing in LlamaIndex. The primary function resolve_llm converts various LLM specifications (strings, LLM instances, LangChain models) into a unified LLM object. The module also provides parse_partial_json for parsing incomplete JSON strings into valid Python dictionaries.

Source file: llama-index-core/llama_index/core/llms/utils.py (198 lines)

Type Definitions

LLMType = Union[str, LLM, "BaseLanguageModel"]

The LLMType union type represents all accepted input formats for LLM specification:

• str -- string identifiers like "default" or "local:model_path"
• LLM -- a LlamaIndex LLM instance
• BaseLanguageModel -- a LangChain language model instance (v0.x or v1.x)

resolve_llm Function

def resolve_llm(
    llm: Optional[LLMType] = None,
    callback_manager: Optional[CallbackManager] = None
) -> LLM:

Resolves an LLM specification into a concrete LLM instance. The resolution logic follows this decision tree:

Case 1: llm == "default"

1. If the IS_TESTING environment variable is set, returns a MockLLM instance.
2. Otherwise, attempts to create an OpenAI instance from llama_index.llms.openai.
3. Validates the OpenAI API key.
4. Raises ImportError if the llama-index-llms-openai package is not installed.
5. Raises ValueError with a descriptive message if the API key is invalid.

Case 2: llm is a string (not "default")

The string must follow the format "local[:model_path]":

1. Splits the string on ":" to extract the prefix and optional model path.
2. Validates that the prefix is "local"; raises ValueError otherwise.
3. Creates a LlamaCPP instance from llama_index.llms.llama_cpp with:

• • The specified model path
  • Default messages_to_prompt and completion_to_prompt functions
  • n_gpu_layers=1 in model kwargs

1. Raises ImportError if the llama-index-llms-llama-cpp package is not installed.

Case 3: llm is a LangChain BaseLanguageModel

1. Wraps the model in a LangChainLLM adapter from llama_index.llms.langchain.
2. Raises ImportError if the llama-index-llms-langchain package is not installed.

Case 4: llm is None

1. Prints a message that LLM is explicitly disabled.
2. Returns a MockLLM instance.

Final Steps

After resolution, the function:

1. Asserts that the result is an LLM instance.
2. Sets the callback_manager using the provided argument, the LLM's existing callback manager, or the global Settings.callback_manager (in that priority order).
3. Returns the configured LLM.

parse_partial_json Function

def parse_partial_json(s: str) -> Dict:

Parses an incomplete JSON string into a valid Python dictionary. This is adapted from the Open Interpreter project and is useful for handling streaming LLM outputs that may produce partial JSON.

Algorithm

1. Direct parse attempt: First tries json.loads(s). If successful, returns the result.
2. Character-by-character processing: Iterates through each character, tracking:

• • A stack of expected closing characters (} and ])
  • Whether currently inside a string literal
  • Whether the current character is escaped

1. String handling: Newlines inside strings are converted to \n escape sequences.
2. Incomplete key detection: If still inside a string at the end and no colon follows the last quote, the incomplete key is removed.
3. Trailing cleanup:

• • If the string ends with ":", appends null as a default value.
  • If the string ends with ",", removes the trailing comma.

1. Stack closure: Any remaining open structures are closed in reverse order.
2. Final parse: Attempts json.loads on the repaired string.
3. Error handling: Raises ValueError("Malformed partial JSON encountered.") if both parse attempts fail, or if mismatched closing characters are encountered.

Dependencies

• llama_index.core.llms.llm.LLM -- the core LLM class
• llama_index.core.llms.callbacks.CallbackManager -- callback management
• llama_index.core.settings.Settings -- global settings (lazy import)
• llama_index.core.llms.mock.MockLLM -- mock LLM for testing and disabled states (lazy import)
• llama_index.llms.openai -- OpenAI LLM integration (optional, lazy import)
• llama_index.llms.llama_cpp -- LlamaCPP integration (optional, lazy import)
• llama_index.llms.langchain -- LangChain adapter (optional, lazy import)
• LangChain's BaseLanguageModel (v0.x or v1.x, optional, conditional import)