Principle:Sktime Pytorch forecasting Estimator Conformance Testing
| Knowledge Sources | |
|---|---|
| Domains | Time_Series, Forecasting, Deep_Learning, Testing, Software_Architecture |
| Last Updated | 2026-02-08 09:00 GMT |
Overview
An API conformance testing framework that validates whether estimators (models, metrics, and other components) in pytorch-forecasting correctly implement the required interface contracts. Inspired by scikit-learn's check_estimator utility and adapted from sktime, it provides both programmatic and pytest-integrated testing workflows.
Description
The estimator conformance testing framework ensures that all objects in pytorch-forecasting adhere to their declared API contracts. It provides three main entry points:
check_estimator: The primary function for running all conformance tests against a single estimator class or instance. When given a class, it discovers all applicable test suites using get_test_classes_for_obj, generates test instances via the class's create_test_instances_and_names method, and runs every relevant test fixture. When given an instance, tests run on both that specific instance and its class. For packaged objects (neural network models), it also fetches the package class via the pkg attribute and runs package-level tests. Results are returned as a dictionary mapping test fixture names (in pytest notation, e.g., test_name[fixture-id]) to either the string PASSED or the exception that was raised.
The function supports fine-grained test selection through multiple parameters:
- tests_to_run -- restrict to specific test names
- fixtures_to_run -- restrict to specific test-fixture combination codes
- tests_to_exclude -- remove specific tests after selection
- fixtures_to_exclude -- remove specific fixtures after selection
- raise_exceptions -- if True, exceptions propagate immediately rather than being captured in the results dictionary
The test execution is delegated to the test class's run_tests method, which manages fixture generation, test execution, and result collection. Verbosity controls printout: level 0 produces no output, level 1 prints a pass/fail summary, and level 2 prints full test output.
parametrize_with_checks: A pytest decorator that generates parameterized test cases for one or more estimator classes. For each estimator, it discovers all applicable test names via _get_test_names_for_obj and creates pytest parameter tuples of (estimator, test_name). This enables third-party libraries to run pytorch-forecasting conformance checks on their own estimator implementations using standard pytest patterns.
Helper Functions: _get_test_names_from_class extracts all method names starting with test from a test class. _get_test_names_for_obj combines this across all test classes applicable to a given object. These helpers enable dynamic test discovery without hard-coded test lists.
The framework requires pytest as a soft dependency (not included in base installation), verified at runtime via skbase's _check_soft_dependencies utility, with a clear error message directing users to install it.
Usage
Use check_estimator during development to verify that a new model or metric correctly implements all required interface methods. Use parametrize_with_checks in third-party test suites to automatically run the full conformance suite against custom estimators. Run with raise_exceptions=True during development for immediate error feedback, and with raise_exceptions=False in CI pipelines to collect a full report of all passing and failing tests.
Theoretical Basis
Contract Testing Pattern:
Conformance testing implements the Design by Contract principle for object-oriented frameworks. Each base class in the hierarchy defines a contract (set of required methods, expected behaviors, and tag constraints), and the test suite validates that subclasses fulfill these contracts:
# Pseudo-code: conformance test dispatch
def check_estimator(estimator):
# Discover applicable test suites from the class hierarchy
test_classes = get_test_classes_for_obj(estimator)
results = {}
for test_cls in test_classes:
# Each test class covers one level of the hierarchy
# e.g., BaseObject tests, Metric tests, Model tests
class_results = test_cls().run_tests(
obj=estimator,
raise_exceptions=False,
)
results.update(class_results)
return results # {"test_name[fixture]": "PASSED" or Exception}
Test Fixture Generation:
Each test class generates fixtures (specific test configurations) for the object under test. For a model class, this might include different hyperparameter combinations and dataset configurations:
# Pseudo-code: fixture-based parameterization
def parametrize_with_checks(estimators):
params = []
for est in estimators:
test_names = get_all_test_names(est)
for test in test_names:
params.append((est, test))
return pytest.mark.parametrize("obj, test_name", params)
Inheritance-Aware Testing:
Tests are organized hierarchically to match the class hierarchy. A model class inheriting from BaseModel will have tests from both the BaseObject test suite and the BaseModel test suite applied, ensuring that both general object contracts and model-specific contracts are validated.