Principle:Ucbepic Docetl Data Sampling And Selection
| Knowledge Sources | |
|---|---|
| Domains | LLM_Data_Processing, Data_Reduction |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Multi-strategy document sampling enables pipeline size reduction by supporting uniform, stratified, outlier-detection, full-text search, and embedding-based sampling strategies to create representative subsets of data.
Theoretical Basis
LLM-powered data pipelines often process large document collections where applying expensive LLM operations to every document is unnecessary or cost-prohibitive. Intelligent sampling allows users to reduce dataset size while preserving the statistical properties and representativeness of the original collection. The choice of sampling strategy depends on the analytical goal: uniform sampling preserves overall distributions, stratified sampling maintains group proportions, and outlier detection focuses on atypical documents.
DocETL's sampling operation supports six distinct methods. Uniform sampling uses scikit-learn's train_test_split for random selection. First sampling takes the first N records, useful for ordered datasets. Custom sampling allows explicit selection by key values. Outlier detection uses embedding distances from a centroid to identify unusual documents, controlled either by standard deviation cutoff or fixed count. Top-embedding retrieves the K documents most similar to a query using cosine similarity on dense embeddings. Top-FTS uses BM25 full-text search scoring for keyword-based retrieval.
All methods except custom support stratification, which groups documents by one or more keys and applies the sampling method within each group. This ensures that minority groups are represented in the sample. The operation can sample either a fixed number per group or proportionally based on group size, providing flexibility for different analytical needs.
Key Design Decisions
| Decision | Choice | Rationale |
|---|---|---|
| Multiple sampling methods | Six strategies (uniform, first, custom, outliers, top_embedding, top_fts) | Different analytical goals require different sampling approaches; a single method cannot serve all use cases |
| Stratification support | Optional grouping by one or more keys with per-group or proportional allocation | Ensures minority groups are represented; prevents sampling bias in heterogeneous datasets |
| Embedding-based outlier detection | Distance from centroid with configurable std or count cutoff | Identifies atypical documents that may be of special interest or indicate data quality issues |