Implementation:Microsoft Autogen Agbench Remove Missing Cmd

Property	Value
Source	https://github.com/microsoft/autogen
Domains	Benchmarking CLI File_Management Data_Cleanup
Last Updated	2026-02-11 17:00 GMT

Overview

A command-line utility that identifies and removes benchmark run folders with incomplete or missing results based on configurable completion criteria.

Description

The remove_missing_cmd.py module provides automated cleanup functionality for AutoGenBench run logs by detecting and removing folders that lack complete benchmark results. The core function default_scorer() evaluates each instance directory by examining the console_log.txt file for three required markers: "FINAL ANSWER:", "SCENARIO.PY COMPLETE !#!#", and "RUN.SH COMPLETE !#!#". It also checks that the last 10 lines do not contain "Error code", ensuring the benchmark completed successfully. The delete_folders_with_missing_results() function iterates through task directories in the run logs, checking each numbered instance subdirectory using the scorer. When missing results are detected, the function prompts for user confirmation (unless --noconfirm is specified) before deleting the entire task folder. The CLI interface remove_missing_cli() uses argparse to handle command-line arguments including the run logs path and an optional -c/--noconfirm flag to bypass confirmation prompts. The module includes a safety mechanism that prompts users to confirm they have modified the default_scorer function to match their expected completion patterns, preventing accidental deletion of valid results.

Usage

Use this module when you need to:

Clean up benchmark run directories with incomplete executions
Remove folders where benchmarks failed or did not complete
Free disk space by deleting invalid benchmark results
Maintain only successfully completed benchmark runs
Automate cleanup of large benchmark result sets
Prepare benchmark results for tabulation by removing incomplete data

Code Reference

Source Location: /tmp/kapso_repo_2mr4n2g4/python/packages/agbench/src/agbench/remove_missing_cmd.py

Signature:

def remove_missing_cli(args: Sequence[str]) -> None:
    """
    CLI entry point for removing folders with missing benchmark results.

    Parses arguments, validates the runlogs directory path, and initiates
    the deletion process with optional confirmation prompts.

    Args:
        args: Command-line arguments where args[0] is invocation command
              and args[1:] contains parsed arguments

    Returns:
        None (exits with status code 1 on error)
    """

def default_scorer(instance_dir: str) -> bool:
    """
    Evaluate if an instance directory has complete benchmark results.

    Checks console_log.txt for required completion markers and absence
    of error messages in the final lines.

    Args:
        instance_dir: Path to the instance directory to evaluate

    Returns:
        True if all completion criteria are met, False otherwise
    """

def delete_folders_with_missing_results(runlogs_path: str, noconfirm: bool = False) -> None:
    """
    Scan and delete task folders containing incomplete benchmark results.

    Iterates through task directories, checks each instance, and removes
    folders with missing results after user confirmation.

    Args:
        runlogs_path: Path to the directory containing benchmark run logs
        noconfirm: If True, delete without confirmation prompts

    Returns:
        None (prints deletion status and total count)
    """

Import:

from agbench.remove_missing_cmd import remove_missing_cli, default_scorer, delete_folders_with_missing_results

I/O Contract

Inputs

Parameter	Type	Required	Description
`args`	`Sequence[str]`	Yes	Command-line arguments list. args[0] is invocation command, args[1] is runlogs path
`runlogs_path`	`str`	Yes	Path to directory containing benchmark run logs organized by task_id/instance
`noconfirm`	`bool`	No	If True, skip confirmation prompts before deletion (default: False)
`instance_dir`	`str`	Yes	Path to specific instance directory for scoring evaluation

Outputs

Output	Type	Description
Return value	`None`	Functions perform side effects (file deletion) and print status
Deleted folders	`int`	Count of task folders removed from the filesystem
Console output	`str`	Status messages for each folder evaluated and deletion summary
Exit code	`int`	1 if runlogs path is invalid, 0 on successful completion

Usage Examples

Command-Line Usage:

# Interactive mode (prompts for confirmation on each folder)
autogenbench remove_missing path/to/runlogs

# Example interactive output:
# Missing Results in : path/to/runlogs/task_123
# Press 1 to delete, anything else to skip...
# [user enters 1]
# Deleted folder: path/to/runlogs/task_123
# Total folders deleted: 1

# Non-interactive mode (auto-delete without prompts)
autogenbench remove_missing path/to/runlogs --noconfirm

# Short form flag
autogenbench remove_missing path/to/runlogs -c

Programmatic Usage:

from agbench.remove_missing_cmd import (
    delete_folders_with_missing_results,
    default_scorer
)

# Delete missing results interactively
delete_folders_with_missing_results(
    runlogs_path="/path/to/benchmark/runlogs",
    noconfirm=False
)

# Delete missing results automatically (no prompts)
delete_folders_with_missing_results(
    runlogs_path="/path/to/benchmark/runlogs",
    noconfirm=True
)

# Check if a specific instance has complete results
instance_path = "/path/to/runlogs/task_456/0"
is_complete = default_scorer(instance_path)
if is_complete:
    print("Instance has complete results")
else:
    print("Instance is missing results")

Custom Scorer Implementation:

import os

def custom_scorer(instance_dir: str) -> bool:
    """
    Custom completion criteria for specific benchmark types.

    Modify this function to match your expected completion patterns.
    """
    console_log = os.path.join(instance_dir, "console_log.txt")
    if not os.path.isfile(console_log):
        return False

    with open(console_log, "rt") as fh:
        content = fh.read()

        # Custom markers for your benchmark
        has_custom_marker = "CUSTOM_COMPLETE" in content
        has_results_file = os.path.exists(
            os.path.join(instance_dir, "results.json")
        )

        return has_custom_marker and has_results_file

# Use custom scorer
# Note: You would need to modify the source to use this custom scorer
# or fork the module for custom implementations

Standalone Script Usage:

# Can also be run directly as a Python script
python remove_missing_cmd.py /path/to/runlogs

# With no-confirm flag
python remove_missing_cmd.py /path/to/runlogs -c

Integration with Benchmark Workflow:

from agbench.remove_missing_cmd import delete_folders_with_missing_results
from agbench.tabulate_cmd import tabulate_cli

# Clean up incomplete runs before tabulation
runlogs_path = "/path/to/benchmark/runlogs"

# Remove incomplete results (no confirmation for automation)
delete_folders_with_missing_results(runlogs_path, noconfirm=True)

# Now tabulate only complete results
tabulate_cli(["autogenbench tabulate", runlogs_path])

Related Pages

Agbench_CLI - Main CLI dispatcher that routes to this command
Agbench_Tabulate_Cmd - Results tabulation (typically used after cleanup)
File_System_Management - Directory traversal and deletion operations
Result_Validation - Benchmark result completeness checking
Scorer_Function - Configurable evaluation criteria pattern

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment