Principle:Ucbepic Docetl Pipeline Execution Streaming
| Knowledge Sources | |
|---|---|
| Domains | Web_Application, Real_Time_Processing |
| Last Updated | 2026-02-08 01:40 GMT |
Overview
A real-time execution feedback principle that streams pipeline execution progress and results via WebSocket connections.
Description
Pipeline Execution Streaming enables real-time visibility into pipeline execution through WebSocket connections. The backend runs the pipeline in a background thread, captures console output from DSLRunner's ThreadSafeConsole, and streams it to connected clients. Final results, costs, and errors are also streamed as structured JSON messages.
Usage
Use the playground's "Run All" button to trigger pipeline execution. The Output panel displays real-time console output, and final results appear in a paginated table with column inspection.
Theoretical Basis
WebSocket-based execution streaming:
- Connection: Frontend establishes WebSocket to /ws/run_pipeline/{namespace}
- Execution: Backend starts DSLRunner in a thread, polling console output
- Streaming: Console output streamed as "output" messages at regular intervals
- Completion: Final "result" or "error" message sent with cost and status
Related Pages
Implemented By
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment