Managing AI Dependencies with requirements.txt

Building AI applications today feels like assembling a spaceship from parts made by a thousand different manufacturers. You've got PyTorch or TensorFlow for deep learning, Hugging Face Transformers for NLP, OpenCV for computer vision, NumPy and Pandas for data wrangling, plus a dozen utility libraries that each depend on their own specific versions of CUDA, cuDNN, or system-level binaries. Somewhere in this chaos sits requirements.txt — the humble text file that can either save your project or silently break it six months later.

Let's talk about how to use requirements.txt effectively in the AI/ML world without losing your sanity.

Why AI Dependency Management Is Uniquely Painful

Before diving into solutions, it helps to understand why AI projects suffer from dependency hell more than typical software projects:

1. The CUDA Taxonomy Problem

PyTorch 2.1.0+cu118 is not the same as PyTorch 2.1.0+cu121. One works with CUDA 11.8, the other with CUDA 12.1. If your GPU drivers expect one version and your code expects another, nothing runs. Worse, these variants often can't coexist in the same environment.

2. The Precompiled Wheel Maze

Many AI libraries distribute platform-specific wheels. A package that installs cleanly on Linux might fail entirely on macOS or Windows because the underlying C++ extensions (like FlashAttention or xFormers) weren't compiled for that platform.

3. The Research Code Time Capsule

AI moves fast. A paper from 2022 might depend on Transformers 4.21.0, which breaks with 4.30.0. Meanwhile, newer models require the latest versions. You often need multiple incompatible versions of the same ecosystem on the same machine.

4. Indirect Dependency Explosions

Installing transformers pulls in tokenizers, safetensors, tqdm, requests, numpy, pyyaml, filelock, huggingface-hub, and more. Any one of those can conflict with another project's needs.

The Basics: What requirements.txt Actually Does

At its core, requirements.txt is just a list of packages for pip to install. But the way you specify those packages determines whether your project is reproducible or fragile.

The Naive Approach (Don't Do This)

torch
transformers
numpy
pandas

This says "install whatever the latest version is." It's convenient today and guaranteed to break tomorrow. A teammate running this file next month will get different package versions, potentially incompatible with your code.

The Better Approach

torch==2.1.0
transformers==4.35.0
numpy==1.24.3
pandas==2.0.3

This pins exact versions, making installation deterministic. But it's still incomplete — it doesn't capture transitive dependencies (the packages that these packages depend on). Your environment might work because requests==2.31.0 was already installed, but a fresh environment might pull in requests==2.32.0, which breaks something else.

The Gold Standard: Full Environment Freeze

torch==2.1.0
transformers==4.35.0
numpy==1.24.3
pandas==2.0.3
requests==2.31.0
urllib3==2.0.7
certifi==2023.7.22
charset-normalizer==3.3.2
idna==3.4
filelock==3.13.1
...

You generate this with:

pip freeze > requirements.txt

This is perfectly reproducible but often over-specific. It includes packages you didn't directly choose, and it bakes in your exact OS and Python version, which might not transfer cleanly to another system.

AI-Specific Best Practices

1. Separate Core and Derived Dependencies

Maintain two files:

requirements.in — what you actually care about:

torch==2.1.0+cu118
transformers>=4.30.0,<5.0.0
datasets
accelerate

requirements.txt — the fully resolved, pinned output generated by a tool like pip-compile (from pip-tools):

pip-compile requirements.in --generate-hashes -o requirements.txt

This gives you the best of both worlds: human-readable intent and machine-perfect reproducibility.

2. Use Index URLs for CUDA-Specific Packages

PyTorch and TensorFlow often require custom package indexes. Instead of telling users to "install PyTorch manually first," encode it in the file:

--index-url https://download.pytorch.org/whl/cu118
torch==2.1.0
torchvision==0.16.0
transformers==4.35.0

Or use --extra-index-url if you need packages from both PyPI and a custom index.

3. Handle Platform Differences with Environment Markers

Not every dependency applies to every OS. Use PEP 508 environment markers:

torch==2.1.0; sys_platform == "linux"
torch==2.1.0; sys_platform == "win32"
# macOS often uses the CPU-only version:
torch==2.1.0; sys_platform == "darwin"

For even more control, use requirements/ subdirectories:

requirements/
├── base.txt
├── dev.txt
├── gpu.txt
└── cpu.txt

Then in dev.txt:

-r base.txt
-r gpu.txt
pytest
black

4. Never Check in pip freeze Output Blindly

Before committing a frozen requirements.txt, audit it. Remove packages that are clearly local development tools (like jupyter, ipykernel, or matplotlib if they're only for notebooks). These bloat production containers and create unnecessary conflict surfaces.

5. Containerize Early

If you're serious about reproducibility, requirements.txt is only half the battle. The other half is the system environment — CUDA drivers, cuDNN, Python itself. A Dockerfile combined with a pinned requirements.txt is the closest thing to a guarantee:

FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3-pip
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

Common Pitfalls and How to Avoid Them

When requirements.txt Isn't Enough

For complex AI projects, consider graduating to modern tools:

• pip-tools: Generates pinned requirements from high-level dependencies. Great for teams already using pip.
• Poetry: Manages dependencies and virtual environments together. Its lock file (poetry.lock) is more robust than requirements.txt for capturing the full dependency graph.
• conda: Essential if you need non-Python dependencies (like specific CUDA toolkit versions or MKL libraries). Use environment.yml instead of requirements.txt.
• uv: An extremely fast Python package manager (written in Rust) that's gaining traction for large AI projects with heavy dependency trees.

Example: Poetry for an AI Project

[tool.poetry.dependencies]
python = "^3.10"
torch = {version = "^2.1.0", source = "pytorch-gpu"}
transformers = "^4.35.0"
accelerate = "^0.24.0"

[[tool.poetry.source]]
name = "pytorch-gpu"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

Run poetry install, and you get an isolated, reproducible environment without manually managing transitive dependencies.

A Practical Workflow

Here's a battle-tested workflow for AI projects:

1. Start clean: Create a virtual environment for every project. Never install AI packages globally.
2. Declare intent: Write a minimal requirements.in with only direct dependencies and loose but safe version bounds.
3. Compile: Use pip-compile to generate a locked requirements.txt.
4. Test in isolation: Install from the locked file in a fresh environment to verify.
5. Containerize: Write a Dockerfile that uses the locked requirements.
6. Document the hardware: Add notes in your README about CUDA version, GPU model, and any system-level drivers required.
7. Update carefully: When upgrading, modify requirements.in, recompile, test thoroughly, and commit both files.

Final Thoughts

requirements.txt is deceptively simple — it's just a list of packages, but in the AI world, that list is a contract between your code, your hardware, your team, and time itself. Treat it with respect: pin your versions, separate your concerns, and never assume that "latest" means "working."

The best requirements.txt is one that you can hand to a colleague (or your future self) with confidence, knowing that pip install -r requirements.txt will produce the exact same environment that trained your model, generated your results, and powered your demo.

Because in AI, reproducibility isn't just a nice-to-have. It's science.