Principle:Marker Inc Korea AutoRAG Node Line Evaluation
| Knowledge Sources | |
|---|---|
| Domains | Pipeline Orchestration, RAG Pipeline Optimization |
| Last Updated | 2026-02-12 00:00 GMT |
Overview
Node line evaluation executes a sequence of pipeline nodes in order, evaluating module candidates at each node and selecting the best performer before passing results downstream.
Description
A RAG pipeline is composed of one or more node lines, where each node line is an ordered sequence of nodes. Typical nodes include query expansion, retrieval, reranking, passage compacting, and generation. Node line evaluation is the core optimization loop of AutoRAG: it processes each node in sequence, runs all configured module candidates for that node, evaluates them against the specified metrics, selects the best module, and feeds the winning module's output to the next node.
This sequential, greedy optimization approach means that each node is optimized independently given the best output from all preceding nodes. While this does not explore the full combinatorial space of all possible module combinations across nodes, it provides a practical and efficient approximation that scales linearly with the number of nodes and module candidates.
At each node, the evaluation produces a summary.csv file that records every module candidate's performance metrics, execution time, parameter configuration, and whether it was selected as the best. The node line itself also produces a top-level summary.csv that aggregates the best module selection from every node in the line.
Usage
Node line evaluation is used during every optimization trial. It is the workhorse function that drives the actual evaluation and selection process. It is called once per node line defined in the configuration, and each call processes all nodes within that line sequentially.
Theoretical Basis
The node line evaluation algorithm is a greedy sequential optimization:
FUNCTION run_node_line(nodes, node_line_dir, previous_result):
IF previous_result IS NULL:
previous_result = load_qa_data(project_dir)
summary_list = []
FOR EACH node IN nodes:
previous_result = node.run(previous_result, node_line_dir)
node_summary = load_summary(node_line_dir / node.type / "summary.csv")
best_row = node_summary[is_best == True]
summary_list.append({
node_type: node.type,
best_module_filename: best_row.filename,
best_module_name: best_row.module_name,
best_module_params: best_row.module_params,
best_execution_time: best_row.execution_time
})
save_csv(summary_list, node_line_dir / "summary.csv")
RETURN previous_result
Within each node.run() call, the following happens internally:
- All configured module candidates for this node type are executed against the previous_result DataFrame.
- Each module produces an output DataFrame and is timed.
- Evaluation metrics are computed for each module's output.
- The configured strategy (mean, rank, or normalize_mean) selects the best module.
- The best module's output is saved as best_*.parquet and becomes the previous_result for the next node.
The greedy property means that once a module is selected as best for a given node, that decision is final and all subsequent nodes optimize relative to it. This makes the optimization O(N * M) where N is the number of nodes and M is the average number of module candidates per node, rather than O(M^N) for exhaustive search.