Implementation:DataTalksClub Data engineering zoomcamp Kestra PurgeCurrentExecutionFiles
| Metadata | |
|---|---|
| Knowledge Sources | repo: DataTalksClub/data-engineering-zoomcamp, docs: Kestra Documentation, Kestra PurgeCurrentExecutionFiles Plugin |
| Domains | Resource Management, Storage Cleanup, Pipeline Maintenance |
| Last Updated | 2026-02-09 14:00 GMT |
Overview
Concrete tool for purging all temporary output files from the current Kestra execution's internal storage using the PurgeCurrentExecutionFiles core plugin, preventing storage accumulation across pipeline runs.
Description
This implementation uses io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles to delete all files that were registered as task outputs during the current pipeline execution. The task requires no parameters -- it automatically identifies and removes all output files scoped to the current execution context.
In the 04_postgres_taxi flow, this task is the final step (id: purge_files) and removes the downloaded and decompressed CSV files that were produced by the extract task and consumed by the CopyIn task. By this point, all data has been loaded into PostgreSQL, so the CSV files in Kestra's internal storage are no longer needed.
The task includes a description annotation noting: "This will remove output files. If you'd like to explore Kestra outputs, disable it." This serves as a development hint -- during debugging, this task can be disabled to preserve intermediate files for inspection.
Usage
This task should be the last task in any Kestra flow that produces intermediate files. It runs unconditionally after all data processing branches have completed.
Code Reference
Source Location: 02-workflow-orchestration/flows/04_postgres_taxi.yaml, Lines 261-263
Signature:
- id: purge_files
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
description: This will remove output files. If you'd like to explore Kestra outputs, disable it.
Import: N/A -- io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles is part of the Kestra core plugin set, included in every Kestra installation. No additional plugin dependencies are required.
I/O Contract
Inputs:
| Name | Type | Description |
|---|---|---|
| (implicit) Current execution context | Kestra execution | The task automatically identifies all output files registered by previous tasks in the current execution. No explicit parameters are required. |
Outputs:
| Name | Type | Description |
|---|---|---|
| Purged files | Void | All output files from the current execution are deleted from Kestra's internal storage. No return value. |
| Storage freed | Bytes (implicit) | The disk space consumed by the execution's output files is reclaimed. In this flow, this corresponds to the size of the downloaded CSV file. |
Usage Examples
Minimal usage (as used in the flow):
- id: purge_files
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
With description annotation for development awareness:
- id: purge_files
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
description: This will remove output files. If you'd like to explore Kestra outputs, disable it.
Placement within the full flow task list:
tasks:
- id: set_label
type: io.kestra.plugin.core.execution.Labels
# ...
- id: extract
type: io.kestra.plugin.scripts.shell.Commands
# ... produces CSV files in internal storage
- id: if_yellow_taxi
type: io.kestra.plugin.core.flow.If
# ... DDL, CopyIn, deduplication, merge
- id: if_green_taxi
type: io.kestra.plugin.core.flow.If
# ... DDL, CopyIn, deduplication, merge
- id: purge_files
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
description: This will remove output files. If you'd like to explore Kestra outputs, disable it.
# Final task -- all intermediate files cleaned up
Disabling the task during development (via Kestra UI):
The task can be disabled by toggling its state in the Kestra UI execution form, or by commenting it out in the flow YAML during development:
# - id: purge_files
# type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles
# description: Disabled for debugging -- output files will be preserved