Environment:PacktPublishing LLM Engineers Handbook Python 3 11 Poetry Environment

Knowledge Sources	LLM Engineers Handbook pyproject.toml
Domains	Infrastructure, Python
Last Updated	2026-02-08 08:00 GMT

Overview

Python 3.11 development environment managed by Poetry 1.8.3+ with 30+ dependencies for ML pipelines, web crawling, and RAG inference.

Description

This environment provides the core Python runtime and dependency management for the LLM Engineers Handbook project. It uses Poetry as the package manager with a pyproject.toml configuration specifying exact or minimum version constraints for all packages. The project targets Python 3.11.x specifically (not 3.12+) due to compatibility requirements with ZenML, PyTorch, and other ML libraries. Poetry groups separate core dependencies from optional AWS-related packages.

Usage

Use this environment for all local development, pipeline execution, and API serving. It is the mandatory prerequisite for every workflow in the project: Digital Data ETL, Feature Engineering, Dataset Generation, LLM Finetuning, Model Evaluation, and RAG Inference.

System Requirements

Category	Requirement	Notes
OS	Ubuntu 20.04+ / macOS / Windows (WSL2)	Dockerfile uses `python:3.11-slim-bullseye`
Python	3.11.x (exact)	Specified as `python = "~3.11"` in pyproject.toml
Poetry	>= 1.8.3, < 2.0	Dockerfile pins `POETRY_VERSION=1.8.3`
Docker	>= 27.1.1	Required for local infrastructure services
Git	>= 2.44.0	Required for repository operations
pyenv	>= 2.3.36	Optional, for managing Python versions

Dependencies

System Packages

`build-essential` (for compiling native extensions)
`gcc` (C compiler)
`python3-dev` (Python development headers)
`libglib2.0-dev` (GLib library)
`libnss3-dev` (Network Security Services)

Python Packages (Core)

`zenml` = 0.74.0 (with server extras)
`pymongo` >= 4.6.2
`torch` = 2.2.2
`numpy` >= 1.26.4
`datasets` >= 3.0.1
`selenium` >= 4.21.0
`webdriver-manager` >= 4.0.1
`beautifulsoup4` >= 4.12.3
`html2text` >= 2024.2.26
`chromedriver-autoinstaller` >= 0.6.4
`qdrant-client` >= 1.8.0
`langchain` >= 0.2.11
`langchain-openai` >= 0.1.3
`langchain-community` >= 0.2.11
`sentence-transformers` >= 3.0.0
`tiktoken` >= 0.7.0
`fastapi` >= 0.100, <= 0.110
`uvicorn` >= 0.30.6
`opik` >= 0.2.2
`pydantic-settings` >= 2.1.0

Python Packages (AWS - Optional Group)

`sagemaker` >= 2.232.2
`s3fs` > 2022.3.0
`aws-profile-manager` >= 0.7.3
`kubernetes` >= 30.1.0
`sagemaker-huggingface-inference-toolkit` >= 2.4.0

Credentials

The following environment variables must be set in `.env`:

`OPENAI_API_KEY`: OpenAI API key for GPT-4o-mini (dataset generation, RAG query expansion)
`HUGGINGFACE_ACCESS_TOKEN`: HuggingFace Hub token for model/dataset access
`COMET_API_KEY`: Comet ML API key for experiment tracking
`COMET_PROJECT`: Comet ML project name (default: "twin")

Quick Install

# Install Python 3.11 via pyenv
pyenv install 3.11
pyenv local 3.11

# Install Poetry
pip install poetry>=1.8.3

# Install core dependencies
poetry install

# Install AWS dependencies (optional)
poetry install --with aws

Code Evidence

Python version constraint from `pyproject.toml:10`:

python = "~3.11"

Poetry version from `Dockerfile:6`:

ENV POETRY_VERSION=1.8.3

Settings loaded from `.env` via Pydantic in `llm_engineering/settings.py:7-8`:

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

Optional AWS imports with fallback warning in `llm_engineering/infrastructure/aws/deploy/huggingface/config.py:5-8`:

try:
    from sagemaker.compute_resource_requirements.resource_requirements import ResourceRequirements
except ModuleNotFoundError:
    logger.warning("Couldn't load SageMaker imports. Run 'poetry install --with aws' to support AWS.")

Common Errors

Error Message	Cause	Solution
`ModuleNotFoundError: No module named 'sagemaker'`	AWS optional group not installed	Run `poetry install --with aws`
`Python version ~3.11 required`	Wrong Python version active	Use `pyenv local 3.11` or install Python 3.11
`Failed to load settings from the ZenML secret store`	ZenML secrets not configured	Create `.env` file with required variables (falls back automatically)

Compatibility Notes

Python 3.12+: Not supported due to ZenML and torch 2.2.2 compatibility constraints.
macOS: Requires `export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` before starting ZenML server.
Windows: Must use WSL2 for full compatibility.
Poetry 2.0+: Not supported; stick to Poetry 1.x series.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment