Workflow:Diagram of thought Diagram of thought DoT Trace Extraction
| Knowledge Sources | |
|---|---|
| Domains | LLM_Reasoning, Trace_Analysis, DAG_Construction |
| Last Updated | 2024-09-15 12:00 GMT |
Overview
End-to-end process for extracting, parsing, and analyzing the structured reasoning DAG from Diagram of Thought output using the typed serialization protocol.
Description
This workflow covers the process of extracting an auditable reasoning graph from raw DoT output. The Diagram of Thought framework embeds typed serialization records (@node, @edge, @status) within the LLM's text output. These records form a formal backbone that can be deterministically parsed to reconstruct the reasoning DAG. The extracted graph reveals which propositions were validated or invalidated, how reasoning steps depend on each other, and how the final conclusion was synthesized. This enables post-hoc analysis, debugging, verification, and quality assessment of the model's reasoning process.
Usage
Execute this workflow after running a DoT iterative reasoning session (see the DoT_Iterative_Reasoning workflow) when you need to verify the reasoning path, debug unexpected conclusions, perform formal analysis of the reasoning structure, or build automated evaluation pipelines for LLM reasoning quality.
Execution Steps
Step 1: Collect Raw DoT Output
Capture the complete text output from a DoT reasoning session. This includes all interleaved role-tagged sections (proposer, critic, summarizer) along with any embedded typed serialization records. The raw output is the input artifact for trace extraction.
Key considerations:
- Preserve the complete output including all XML role tags
- Ensure typed records (@node, @edge, @status) are intact
- Store the raw output for reproducibility
Step 2: Parse Typed Records
Scan the raw output to extract all typed serialization records. Each record type provides different structural information: @node records identify reasoning steps and their roles (problem, proposer, critic, summarizer), @edge records define dependencies between nodes with typed relationships (use, critique, refine), and @status records capture validation decisions (validated or invalidated).
What happens:
- Extract all lines matching the @node pattern to build a node registry
- Extract all @edge records to build the adjacency list
- Extract all @status records to build the validation map
- Validate that edge sources always have smaller IDs than destinations (acyclicity guarantee)
Step 3: Reconstruct the DAG
Assemble the parsed records into a Directed Acyclic Graph data structure. Nodes represent reasoning steps tagged with their role and content. Edges represent dependencies typed as use (builds upon), critique (evaluates), or refine (corrects). The validation status is overlaid on each node to distinguish validated from invalidated propositions.
Key considerations:
- Verify the graph is acyclic (edge constraint: src < dst)
- Associate each node with its full text content from the role-tagged section
- Flag any orphan nodes (nodes without incoming edges except the problem root)
- Identify the summarizer node(s) as terminal node(s)
Step 4: Analyze Reasoning Paths
Traverse the reconstructed DAG to extract reasoning paths and assess quality. Identify the critical path from problem to final synthesis through only validated nodes. Detect branching points where the proposer explored alternative approaches, and backtracking points where the critic invalidated a step.
What happens:
- Trace all paths from the problem node to summarizer node(s)
- Identify the validated path (nodes with status=validated leading to the conclusion)
- Count branching factor (alternative proposals at each step)
- Measure critique effectiveness (ratio of invalidated to validated propositions)
- Detect reasoning depth (longest path length)
Step 5: Generate Trace Report
Produce a structured report summarizing the reasoning trace analysis. The report includes the DAG visualization, the critical validated path, any invalidated branches with their critique reasons, and summary statistics about the reasoning process. This report serves as the auditable artifact for verification, debugging, or evaluation.
Key considerations:
- Include a visual representation of the DAG (graph rendering)
- List all validated propositions in order along the critical path
- Document invalidated branches with their critique rationale
- Provide summary statistics: total nodes, validated/invalidated ratio, depth, branching factor