Implementation:Langfuse Langfuse ChatML Types

Knowledge Sources	Langfuse
Domains	ChatML, Validation, Multimodal
Last Updated	2026-02-14 00:00 GMT

Overview

This module defines Zod v4 schemas and TypeScript types for validating and transforming ChatML (Chat Markup Language) messages, supporting OpenAI-compatible multimodal content including text, images, audio, tool calls, and thinking/reasoning output.

Description

The types.ts file in the ChatML directory is the canonical type definition module for chat message structures used throughout Langfuse's I/O representation layer. It defines a hierarchy of Zod schemas that validate messages conforming to the OpenAI Chat Completions API format while also accommodating extensions from other providers (LangChain, Anthropic, etc.).

The module is organized into several schema groups:

Tool Schemas:

• ToolDefinitionSchema - Validates tool/function definitions with name, optional description, and parameters.
• ToolCallSchema - Validates tool call invocations with id, name, JSON-string arguments, optional type and index.

Media Reference Schemas:

• ParsedMediaReferenceSchema - Represents a parsed Langfuse media reference with type, id, source, and referenceString fields.
• MediaReferenceStringSchema - A transform schema that parses the Langfuse magic string format (@@@langfuseMedia:type=X|id=Y|source=Z@@@) into a ParsedMediaReferenceType.

Content Part Schemas:

• ThinkingContentPartSchema - Validates model reasoning/thinking output with optional signature (Anthropic) and summary (OpenAI) fields.
• RedactedThinkingContentPartSchema - Validates encrypted thinking content (Anthropic safety feature).
• OpenAITextContentPart - Validates text content parts with type "text", "input_text", or "output_text".
• OpenAIImageContentPart - Validates image content parts supporting URL, base64-encoded, and Langfuse media reference images.
• OpenAIInputAudioContentPart - Validates audio input content parts with Langfuse media reference data.
• OpenAIContentParts - An array schema combining text, image, and audio content parts.

Message Schemas:

• BaseChatMlMessageSchema - The full message schema with all OpenAI API fields plus thinking/reasoning, tool calls, and passthrough for framework-specific metadata.
• SimpleChatMessageSchema - A lightweight, permissive schema for backend content extraction that avoids expensive media/tool validation.
• ChatMlMessageSchema - The full schema with frontend transforms: spreads additional_kwargs into the message and moves unknown fields into a json property for PrettyJsonView rendering.
• ChatMlArraySchema / SimpleChatMlArraySchema - Array wrappers requiring at least one message.

Type Guards:

• isOpenAITextContentPart and isOpenAIImageContentPart - Runtime type guard functions using safeParse.

Usage

Use these schemas when you need to:

• Validate incoming chat messages from various LLM providers for storage or display.
• Parse and render multimodal chat content (text, images, audio) in the Langfuse UI.
• Extract simple content from chat messages in backend pipelines without expensive validation.
• Handle Langfuse media references embedded in chat message content fields.
• Support tool calling / function calling message formats.

Code Reference

Source Location

• Repository: Langfuse
• File: packages/shared/src/utils/IORepresentation/chatML/types.ts
• Lines: 1-326

Signature

// Tool schemas
export const ToolDefinitionSchema: z.ZodObject<{ name: z.ZodString; description: z.ZodOptional<z.ZodString>; parameters: z.ZodOptional<z.ZodRecord<z.ZodString, z.ZodAny>> }>;
export const ToolCallSchema: z.ZodObject<{ id: z.ZodString; name: z.ZodString; arguments: z.ZodString; type: z.ZodOptional<z.ZodString>; index: z.ZodOptional<z.ZodNumber> }>;

// Media reference schemas
export const ParsedMediaReferenceSchema: z.ZodObject<{ type: z.ZodString; id: z.ZodString; source: z.ZodString; referenceString: z.ZodString }>;
export type ParsedMediaReferenceType = z.infer<typeof ParsedMediaReferenceSchema>;
export const MediaReferenceStringSchema: z.ZodPipeline<...>;

// Thinking schemas
export const ThinkingContentPartSchema: z.ZodObject<{ type: z.ZodLiteral<"thinking">; content: z.ZodString; signature: z.ZodOptional<z.ZodString>; summary: z.ZodOptional<z.ZodString> }>;
export type ThinkingContentPart = z.infer<typeof ThinkingContentPartSchema>;
export const RedactedThinkingContentPartSchema: z.ZodObject<{ type: z.ZodLiteral<"redacted_thinking">; data: z.ZodString }>;
export type RedactedThinkingContentPart = z.infer<typeof RedactedThinkingContentPartSchema>;

// Content schemas
export type OpenAITextContentPartType = z.infer<typeof OpenAITextContentPart>;
export const OpenAIUrlImageUrl: z.ZodString;
export type OpenAIImageContentPartType = z.infer<typeof OpenAIImageContentPart>;
export const OpenAIContentParts: z.ZodArray<...>;
export type OpenAIOutputAudioType = z.infer<typeof OpenAIOutputAudioSchema>;
export const OpenAIContentSchema: z.ZodNullable<z.ZodUnion<[z.ZodString, typeof OpenAIContentParts]>>;

// Type guards
export const isOpenAITextContentPart: (content: any) => content is OpenAITextContentPartType;
export const isOpenAIImageContentPart: (content: any) => content is OpenAIImageContentPartType;

// Message schemas
export const BaseChatMlMessageSchema: z.ZodObject<...>;
export const SimpleChatMlArraySchema: z.ZodArray<...>;
export const ChatMlMessageSchema: z.ZodPipeline<...>;
export const ChatMlArraySchema: z.ZodArray<typeof ChatMlMessageSchema>;

Import

import {
  ChatMlMessageSchema,
  ChatMlArraySchema,
  BaseChatMlMessageSchema,
  SimpleChatMlArraySchema,
  OpenAIContentSchema,
  OpenAIContentParts,
  ThinkingContentPartSchema,
  RedactedThinkingContentPartSchema,
  ToolDefinitionSchema,
  ToolCallSchema,
  MediaReferenceStringSchema,
  isOpenAITextContentPart,
  isOpenAIImageContentPart,
} from "@langfuse/shared/src/utils/IORepresentation/chatML/types";

I/O Contract

Inputs

Name	Type	Required	Description
role	string	No	The message role (e.g. "system", "user", "assistant", "tool")
content	array | object | null	No	The message content; can be a simple string, structured content parts array, or null for tool-call-only messages
name	string	No	Optional name identifier for the message sender
audio	{ data: string; transcript?: string }	No	Audio output from the model with media reference and optional transcript
tools	ToolDefinition[]	No	Array of tool/function definitions available for the model
tool_calls	ToolCall[]	No	Array of tool call invocations made by the model
tool_call_id	string	No	ID of the tool call this message is responding to
thinking	ThinkingContentPart[]	No	Model reasoning/thinking output (Anthropic, OpenAI)
redacted_thinking	RedactedThinkingContentPart[]	No	Encrypted thinking content (Anthropic safety)
additional_kwargs	Record<string, any>	No	Extra framework-specific fields (LangChain pattern)

Outputs

Name	Type	Description
ChatMlMessageSchema output	object	Transformed message with additional_kwargs spread in and unknown fields moved to a json property
SimpleChatMlArraySchema output	object[]	Minimally validated array of chat messages (at least one)
ParsedMediaReferenceType	{ type: string; id: string; source: string; referenceString: string }	Parsed Langfuse media reference extracted from magic string format

Usage Examples

import { ChatMlArraySchema, SimpleChatMlArraySchema, MediaReferenceStringSchema } from "@langfuse/shared/src/utils/IORepresentation/chatML/types";

// Validate and transform a full chat message array for frontend rendering
const result = ChatMlArraySchema.safeParse([
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "Hello!" },
  {
    role: "assistant",
    content: null,
    tool_calls: [
      { id: "call_1", name: "get_weather", arguments: '{"city":"Berlin"}' },
    ],
  },
]);

// Use simple schema for backend extraction (faster, more permissive)
const simpleResult = SimpleChatMlArraySchema.safeParse([
  { role: "user", content: "What is the weather?" },
]);

// Parse a Langfuse media reference magic string
const mediaRef = MediaReferenceStringSchema.safeParse(
  "@@@langfuseMedia:type=image/jpeg|id=abc-123|source=base64@@@"
);
// mediaRef.data => { type: "image/jpeg", id: "abc-123", source: "base64", referenceString: "..." }

// Validate multimodal content with image
const multimodalMessage = ChatMlArraySchema.safeParse([
  {
    role: "user",
    content: [
      { type: "text", text: "What is in this image?" },
      { type: "image_url", image_url: { url: "https://example.com/img.png" } },
    ],
  },
]);