Implementation:Diagram of thought Diagram of thought Format Compliance Validator

Knowledge Sources	On the Diagram of Thought Diagram of Thought
Domains	Testing, Validation, Prompt_Engineering
Last Updated	2026-02-14

Overview

Concrete pattern for validating that Diagram of Thought (DoT) prompt output conforms to expected XML tag structure, typed record format, and reasoning quality standards. This is a pattern doc -- there is no library to install. Users implement this validation suite themselves in their own codebase.

Description

A validation suite that checks DoT output across multiple dimensions:

• Tag alternation: Verifies that XML role tags (<proposer>, <critic>, <summarizer>) appear in the correct order, starting with a proposer and ending with a summarizer.
• Record format: Confirms that @node, @edge, and @status typed records use valid roles, valid edge kinds, and satisfy structural constraints.
• Acyclicity: Ensures the DAG constraint holds by verifying src < dst for every @edge record.
• Status coverage: Checks that proposer nodes receive validation decisions via @status records.

Usage

Apply this validation pattern after prompt customization and run it on representative test examples that span the expected input distribution. The validation results indicate whether the customized prompt produces structurally sound and protocol-compliant output before the prompt is deployed in production.

Code Reference

Source Location

• Repository: Diagram of Thought
• File: README.md
• Lines: L99-103 (tips for effective use), L39-48 (quick demo with examples)

Signature

def check_tag_alternation(output: str) -> bool: ...
def check_record_format(output: str) -> dict: ...
def validate_prompt(prompt: str, test_examples: list, llm_client) -> list: ...

Import

import re

I/O Contract

Inputs

Name	Type	Required	Description
prompt	str	Yes	Fully configured DoT system prompt to validate
test_examples	list[str]	Yes	Set of test problems to run the prompt against
llm_client	object	Yes	LLM client with a generate(system, user) method

Outputs

Name	Type	Description
results	list[dict]	Validation report: per-example compliance results. Each dict contains the example snippet, tag alternation result (bool), and record format result (dict with role_valid, kind_valid, acyclic, node_count, edge_count, status_count)

Usage Examples

Complete Validation Suite

import re


def check_tag_alternation(output: str) -> bool:
    """Verify XML tags alternate between proposer/critic/summarizer."""
    tags = re.findall(r"<(proposer|critic|summarizer)>", output)
    # Must start with proposer, must end with summarizer
    if not tags or tags[0] != "proposer" or tags[-1] != "summarizer":
        return False
    return True


def check_record_format(output: str) -> dict:
    """Validate typed record format compliance."""
    nodes = re.findall(r"@node id=(\d+) role=(\w+)", output)
    edges = re.findall(r"@edge src=(\d+) dst=(\d+) kind=(\w+)", output)
    statuses = re.findall(r"@status target=(\d+) mark=(\w+)", output)

    # Check valid roles
    valid_roles = {"problem", "proposer", "critic", "summarizer"}
    role_ok = all(role in valid_roles for _, role in nodes)

    # Check valid edge kinds
    valid_kinds = {"use", "critique", "refine"}
    kind_ok = all(kind in valid_kinds for _, _, kind in edges)

    # Check acyclicity: src < dst
    acyclic = all(int(src) < int(dst) for src, dst, _ in edges)

    return {"role_valid": role_ok, "kind_valid": kind_ok, "acyclic": acyclic,
            "node_count": len(nodes), "edge_count": len(edges), "status_count": len(statuses)}


def validate_prompt(prompt: str, test_examples: list, llm_client) -> list:
    """Run full validation suite."""
    results = []
    for example in test_examples:
        output = llm_client.generate(system=prompt, user=example)
        result = {
            "example": example[:50],
            "tag_alternation": check_tag_alternation(output),
            "record_format": check_record_format(output),
        }
        results.append(result)
    return results

Related Pages

Implements Principle

• Principle:Diagram_of_thought_Diagram_of_thought_Prompt_Validation_Testing

Requires Environment

• Environment:Diagram_of_thought_Diagram_of_thought_LLM_API