Workflow:Vespa engine Vespa Document indexing pipeline
| Knowledge Sources | |
|---|---|
| Domains | Document_Processing, Indexing, Search |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
End-to-end process for transforming incoming document operations into indexed content through Vespa's document processing chain and indexing language execution.
Description
This workflow describes how Vespa processes document operations (puts, updates, removes) through the indexing pipeline. When a document arrives via the Document API, it enters a document processing chain where the IndexingProcessor executes compiled indexing language scripts against the document. The ScriptManager maps document types to their configured indexing expressions, and DocumentScript encapsulates the per-document-type execution logic. The pipeline handles field tokenization, embedding generation, and field value computation as defined in the application's schema.
Usage
Execute this workflow when feeding documents into a Vespa application. This is the core path for all document ingestion, whether via the vespa-feed-client, the document/v1 HTTP API, or the Document API over message bus. Use this when you need to understand how schema-defined indexing statements (the "indexing:" directives) transform raw document fields into indexed, searchable content.
Execution Steps
Step 1: Document operation reception
Receive incoming document operations through the document processing chain. Operations arrive as DocumentPut (new or replacement document), DocumentUpdate (partial field modifications), or DocumentRemove (deletion). Each operation carries a document type that determines which indexing script applies. The Processing batch may contain multiple operations that are processed sequentially.
Key considerations:
- Operations arrive via message bus routing through the container
- Each operation has a deadline for timeout enforcement
- DocumentRemove operations pass through without script execution
Step 2: Script resolution
The ScriptManager resolves the appropriate indexing script for the document's type. Scripts are loaded from the IlscriptsConfig configuration at initialization time. The manager maintains a hierarchical map from document type and field name to compiled ScriptExpression objects. For full document processing, a composite script covering all fields is used. For partial updates, field-specific scripts are selected when a statement depends on only a single input field.
Key considerations:
- Scripts are compiled once at initialization and reused for all documents
- Field-specific scripts enable efficient partial updates
- Document type inheritance is supported for script lookup
- The ScriptManager is initialized with Linguistics, Chunker, Embedder, and FieldGenerator instances
Step 3: Linguistics span tree cleanup
Before executing the indexing script, remove any existing linguistics annotation span trees from the document's string fields. This prevents conflicts when the indexing expressions generate fresh linguistic annotations (tokenization results, stemming output) during execution.
Key considerations:
- Only fields declared as input fields in the script are cleaned
- The cleanup verifies that fields exist in the document type before processing
- This step ensures idempotent behavior when documents are reprocessed
Step 4: Indexing expression execution
Execute the compiled indexing language expressions against the document. For document puts, the full script runs against all fields, producing output values according to the schema's indexing statements. For document updates, field-specific expressions run only for the modified fields. The expression engine invokes linguistics processing (tokenization, stemming), embedding generation, and field generators as configured.
Key considerations:
- Expressions are selected per-field via the ExpressionSelector inner class
- Execution respects the deadline and throws TimeoutException if exceeded
- Invalid input generates descriptive error messages with field context
- The FieldValuesFactory provides access to document field values during execution
Step 5: Error handling and output
Handle execution results and route processed documents to the output. Successful processing returns Progress.DONE with the transformed document operations in the output list. Errors are categorized as InvalidInputException (bad data), OverloadException (system capacity), or TimeoutException (deadline exceeded), each producing an appropriate Progress error code with a descriptive message including the document ID.
Key considerations:
- The processor annotates itself with @Provides("indexedDocument")
- Processing order is enforced: @After("indexingStart") and @Before("indexingEnd")
- Failed documents produce error progress codes rather than exceptions
- The output list may contain transformed versions of the input operations