Principle:Langchain ai Langchain Standard Integration Testing

Overview

A standardized integration test suite that validates chat model implementations against real API endpoints to ensure correct end-to-end behavior.

Description

The Standard Integration Testing principle defines how LangChain partner integrations verify that their chat model implementations work correctly with real API services. While unit tests validate interface compliance in isolation, integration tests exercise the full request/response cycle against live endpoints.

LangChain provides the ChatModelIntegrationTests base class in the langchain-tests package with 35+ pre-built test methods that make real API calls. These tests require valid API credentials and network access.

How it works:

1. The integration author creates a test class that inherits from ChatModelIntegrationTests.
2. The author implements chat_model_class and chat_model_params (same pattern as unit tests).
3. API credentials are provided via environment variables (e.g., DEEPSEEK_API_KEY).
4. The inherited tests exercise the full range of model capabilities against the live API.

What the integration tests validate:

• Basic operations: test_invoke, test_stream, test_batch verify that the model can process messages and return valid responses.
• Async operations: test_ainvoke, test_astream validate asynchronous variants.
• Tool calling: test_tool_calling, test_tool_calling_async, test_tool_calling_with_no_arguments verify function/tool integration.
• Structured output: test_structured_output, test_structured_output_pydantic, test_structured_output_json_schema validate typed output generation.
• Token metadata: test_usage_metadata, test_usage_metadata_streaming verify token counting.
• Agent loops: test_agent_loop runs a multi-turn agent tool-use loop.
• Message types: test_tool_message_histories_string_content, test_tool_message_histories_list_content verify various message formats.
• Edge cases: test_stop_sequence, test_long_messages, test_json_mode verify provider-specific behavior.

Feature flags: Same as unit tests, integration test behavior is controlled by properties like has_tool_calling, supports_json_mode, supports_image_inputs, etc. Unsupported tests are automatically skipped.

Test configuration: Integration tests run with a 30-second timeout per test to handle slow API responses gracefully.

Usage

Apply this principle when:

• Validating that a chat model integration works correctly against the live provider API.
• Running pre-release checks to ensure API compatibility has not regressed.
• Debugging failures that only occur with real network requests (authentication, rate limiting, response format changes).

Theoretical Basis

Integration tests complement unit tests by validating the full system boundary. The Abstract Test Case pattern ensures every integration is tested against the same set of real-world scenarios.

from langchain_tests.integration_tests import ChatModelIntegrationTests
from my_package.chat_models import ChatMyProvider


class TestChatMyProviderIntegration(ChatModelIntegrationTests):
    @property
    def chat_model_class(self) -> type[ChatMyProvider]:
        return ChatMyProvider

    @property
    def chat_model_params(self) -> dict:
        return {"model": "model-001", "temperature": 0}

Running with:

# Requires API credentials in environment
export MYPROVIDER_API_KEY="..."
uv run --group test --group test_integration pytest --timeout=30 tests/integration_tests/

Related Pages

• Implementation:Langchain_ai_Langchain_ChatModelIntegrationTests_Subclass