Principle:Confident ai Deepeval Dataset Publishing
| Knowledge Sources | |
|---|---|
| Domains | |
| Last Updated | 2026-02-14 09:00 GMT |
Overview
Dataset publishing is the process of uploading evaluation datasets to the Confident AI cloud platform. It enables collaborative review, centralized versioning, and team-wide access to evaluation data through a managed cloud service.
Description
While local file exports serve individual use cases, dataset publishing addresses the needs of teams and organizations by:
- Centralizing dataset management -- storing evaluation datasets on a shared cloud platform where all team members can access, review, and iterate on them.
- Enabling collaborative review -- published datasets can be reviewed and annotated by multiple team members through the Confident AI web interface.
- Supporting versioning -- datasets are published under named aliases, allowing teams to track different versions of evaluation data over time.
- Facilitating CI/CD integration -- published datasets can be pulled programmatically in automated evaluation pipelines, ensuring consistent evaluation across environments.
- Controlling finalization state -- the finalized flag indicates whether a dataset is ready for use or still under review, providing a lightweight approval workflow.
In the DeepEval framework, dataset publishing is performed via the push method on EvaluationDataset, which uploads the dataset contents to the Confident AI platform under a specified alias.
Usage
Dataset publishing is used when evaluation data needs to be:
- Shared across a team or organization via a centralized platform
- Reviewed and annotated collaboratively before use in evaluation runs
- Made available for automated evaluation pipelines in CI/CD systems
- Versioned and tracked over time with named aliases
Theoretical Basis
Dataset publishing applies principles from cloud-based data management and collaborative workflows:
- Cloud-based dataset management -- storing datasets in a centralized, accessible platform rather than distributing files manually. This mirrors patterns from ML experiment tracking platforms and collaborative data science tools.
- Collaborative evaluation -- enabling multiple stakeholders (engineers, domain experts, QA teams) to contribute to evaluation data quality through shared access and review capabilities.
The abstract publishing process follows this pattern:
DATASET_PUBLISHING(dataset, alias, finalized):
1. SERIALIZE dataset goldens for API transmission
2. AUTHENTICATE with Confident AI platform (API key)
3. UPLOAD dataset under the given alias
4. SET finalization state (finalized=True marks as ready for use)
5. CONFIRM publication and return platform URL
Key properties:
- Named access -- datasets are addressable by alias, enabling version-independent references in code.
- State management -- the finalized flag provides a simple approval mechanism.
- Platform integration -- published datasets appear in the Confident AI dashboard for visual review and management.