Heuristic:Confident ai Deepeval Secret Management Best Practices
| Knowledge Sources | |
|---|---|
| Domains | Security, LLM_Evaluation |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
Best practices for managing API keys and secrets in deepeval, avoiding the deprecated plaintext JSON keystore in favor of environment variables and dotenv files.
Description
Deepeval has historically stored API keys in a plaintext JSON file at `.deepeval/.deepeval`. This approach is now deprecated, and the framework actively warns against it. All secret fields in the `Settings` model are wrapped in Pydantic `SecretStr`, which prevents accidental logging. The `KeyFileHandler` now refuses to write secrets to disk and emits deprecation warnings when reading them from the legacy keystore. The recommended approach is to use `.env` files or process environment variables.
Usage
Apply this heuristic when initially setting up deepeval, when migrating from an older version that used the JSON keystore, or when configuring CI/CD pipelines where secrets must be injected securely.
The Insight (Rule of Thumb)
- Action: Store all API keys in `.env` or `.env.local` files (or inject via CI/CD environment variables). Never use the legacy `.deepeval/.deepeval` JSON keystore for secrets.
- Value: All secret fields use `SecretStr` type, preventing `str(key)` from revealing the value.
- Trade-off: Dotenv files require `.gitignore` discipline. Process env vars are ephemeral but require shell configuration.
- Compatibility: Set `DEEPEVAL_DISABLE_LEGACY_KEYFILE=1` to skip legacy keyfile loading entirely. Set `DEEPEVAL_DISABLE_DOTENV=1` in CI to prevent `.env` file loading from interfering with injected secrets.
Reasoning
The legacy JSON keystore was a simple local storage mechanism, but storing API keys in plaintext files creates security risks. The deprecation was implemented in stages: first, a warning on read; then, a hard block on write. The `SecretStr` wrapper ensures that even if a settings object is accidentally serialized or logged, the secret value is masked.
Hard block on secret writes from `deepeval/key_handler.py:199-206`:
# hard stop on secrets: never write to disk
if _is_secret_key(key):
logger.warning(
"%s is a secret setting, refusing to persist. "
"Keep your secrets in .env or .env.local instead.",
_env_key_for_legacy_enum(key),
)
return
Deprecation warning on secret reads from `deepeval/key_handler.py:246-259`:
if (
value is not None
and _is_secret_key(key)
and _env_key_for_legacy_enum(key) not in _WARNED_SECRET_KEYS
):
logger.warning(
"Reading secret '%s' from legacy %s/%s. Persisting API keys in plaintext is deprecated. "
"Move this to your environment (.env / .env.local). This fallback will be removed in a future release.",
_env_key_for_legacy_enum(key),
HIDDEN_DIR,
KEY_FILE,
)
_WARNED_SECRET_KEYS.add(_env_key_for_legacy_enum(key))
SecretStr detection from `deepeval/config/settings.py:94-104`:
def _is_secret_key(env_key: str) -> bool:
field = Settings.model_fields.get(env_key)
if not field:
return False
if field.annotation is SecretStr:
return True
origin = get_origin(field.annotation)
if origin is Union:
return any(arg is SecretStr for arg in get_args(field.annotation))
return False
Legacy keyfile skip flag from `deepeval/config/settings.py:119`:
if parse_bool(os.getenv("DEEPEVAL_DISABLE_LEGACY_KEYFILE"), default=False):
return