Implementation:PacktPublishing LLM Engineers Handbook VectorBaseDocument Bulk Insert

Type	API Doc
API	`VectorBaseDocument.bulk_insert(cls, documents: list[VectorBaseDocument]) -> bool`
Source	llm_engineering/domain/base/vector.py:L80-97
Repository	PacktPublishing/LLM-Engineers-Handbook
Implements	Principle:PacktPublishing_LLM_Engineers_Handbook_Vector_Storage

Overview

The bulk_insert class method on VectorBaseDocument persists a list of documents to the Qdrant vector database. It automatically groups documents by their concrete class, creates collections if they do not exist, and performs upsert operations to ensure idempotent writes. This is the primary write API for the vector storage layer in the feature engineering pipeline.

API Signature

@classmethod
def bulk_insert(cls, documents: list["VectorBaseDocument"]) -> bool:

Parameters

Parameter	Type	Description
`documents`	`list[VectorBaseDocument]`	A list of documents to insert into Qdrant. The list may contain documents of mixed concrete types (e.g., `CleanedArticleDocument` and `EmbeddedArticleChunk`). Documents are automatically grouped by class for collection routing.

Return Value

Type	Description
`bool`	`True` if the insertion was successful for all document groups. `False` if the input list is empty or if no document groups could be formed.

Source Code

@classmethod
def bulk_insert(cls, documents: list["VectorBaseDocument"]) -> bool:
    # Groups documents by class, creates collection if missing, upserts via qdrant_client
    grouped_documents = VectorBaseDocument.group_by_class(documents)
    if not grouped_documents:
        return False

    for document_class, class_documents in grouped_documents.items():
        collection_name = document_class.get_collection_name()
        use_vector_index = document_class._has_vector_field()
        # Creates collection if needed
        # Upserts points to Qdrant
        ...
    return True

Import

from llm_engineering.domain.base.vector import VectorBaseDocument

In practice, callers typically work with concrete subclasses:

from llm_engineering.domain.cleaned_documents import CleanedArticleDocument
from llm_engineering.domain.embedded_chunks import EmbeddedArticleChunk

How It Works

Document grouping — VectorBaseDocument.group_by_class(documents) partitions the input list into a dictionary keyed by concrete document class. This ensures that documents of different types are routed to their respective Qdrant collections.
Empty check — If the grouped result is empty (no documents provided), the method returns False immediately.
Per-class processing — For each document class and its corresponding documents:
- Collection name resolution — document_class.get_collection_name() returns the Qdrant collection name (derived from the class's Settings.name attribute).
- Vector index detection — document_class._has_vector_field() checks whether the document class defines an embedding field. This determines whether the collection should be created with a vector index.
- Collection creation — If the collection does not already exist, it is created with the appropriate configuration (vector size and distance metric if vectors are used, or payload-only if not).
- Upsert execution — Documents are converted to Qdrant PointStruct objects (with UUID as the point ID, document fields as payload, and optionally the embedding as the vector) and upserted into the collection.
Success return — Returns True after all document groups have been successfully upserted.

Usage Example

from llm_engineering.domain.base.vector import VectorBaseDocument

# Cleaned documents (no vectors)
cleaned_docs = [cleaned_article_1, cleaned_article_2, cleaned_post_1]
VectorBaseDocument.bulk_insert(cleaned_docs)

# Embedded chunks (with vectors)
embedded_chunks = [embedded_chunk_1, embedded_chunk_2, embedded_chunk_3]
VectorBaseDocument.bulk_insert(embedded_chunks)

# Mixed types — automatically grouped by class
all_documents = cleaned_docs + embedded_chunks
VectorBaseDocument.bulk_insert(all_documents)

External Dependencies

Dependency	Purpose
qdrant_client	Qdrant Python client; provides `upsert()`, `create_collection()`, and collection management APIs
pydantic	Data validation and serialization; `VectorBaseDocument` extends Pydantic's `BaseModel`
loguru	Structured logging for insertion progress and error reporting

Design Notes

The method is a classmethod called on VectorBaseDocument itself (not a subclass), because it handles heterogeneous document lists that may span multiple concrete types.
Upsert semantics ensure that re-running the pipeline with the same data does not create duplicate records. Each document's UUID serves as the idempotency key.
Automatic collection creation means callers never need to pre-provision Qdrant collections — the schema is inferred from the document class at write time.
The vector index detection mechanism allows the same insertion API to handle both payload-only documents (cleaned documents) and vector-indexed documents (embedded chunks) transparently.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment