Skip to main content

How to Pip Install from Multiple Requirements Files in Python

Managing Python project dependencies often involves using requirements.txt files. In more complex projects, you might have different sets of requirements for different environments (e.g., development, testing, production) or for different features. pip allows you to install packages from multiple requirements files in a single command, or even include one requirements file within another.

This guide explains how to effectively use pip install -r with multiple files to manage your Python dependencies.

The Need for Multiple Requirements Files​

Separating dependencies into multiple files can improve project organization and management:

  • Environment-Specific Packages: Install development tools (linters, test runners) only in development environments, not in production.
    • requirements_dev.txt: Packages for development (e.g., pytest, black, flake8).
    • requirements_prod.txt: Core application packages for production.
  • Base and Feature Packages: Have a base set of requirements and additional ones for specific features.
    • requirements_base.txt: Core dependencies.
    • requirements_feature_x.txt: Packages needed only for "feature X".
  • Layered Dependencies: Shared common libraries and specific application libraries.

Method 1: Specifying Multiple -r Options (Recommended)​

This is the most straightforward and generally recommended way to install from multiple files. You provide the -r (or --requirement) option multiple times in a single pip install command.

Command Syntax​

pip install -r <file1.txt> -r <file2.txt> -r <file3.txt> ...

Or, if pip is not directly in your PATH or you need to specify the Python version:

python -m pip install -r <file1.txt> -r <file2.txt> ...
python3 -m pip install -r <file1.txt> -r <file2.txt> ...
py -m pip install -r <file1.txt> -r <file2.txt> ...         # For Windows

Example Usage​

Let's assume you have the following requirements files:

  • common.txt: 
    requests>=2.25.0
    numpy==1.23.0
  • dev.txt: 
    pytest==7.1.0
    pylint==2.14.0
  • prod.txt: 
    gunicorn==20.1.0

To set up a development environment, you'd install common and development-specific packages:

pip install -r common.txt -r dev.txt

To set up a production environment, you'd install common and production-specific packages:

pip install -r common.txt -r prod.txt

pip will process all specified files and install all the listed packages and their dependencies. If a package is listed in multiple files (perhaps with different version specifiers), pip's dependency resolver will attempt to find a compatible version.

Method 2: Nesting Requirements Files (Using -r Inside a File)​

You can also include one requirements file within another using the -r <included_file.txt> syntax directly inside a requirements file.

How It Works​

When pip processes a requirements file and encounters a line like -r another_file.txt, it will then proceed to install all packages listed in another_file.txt as if they were part of the original file.

Example File Structure​

  • requirements/common.txt: 
    requests>=2.25.0
    numpy==1.23.0
  • requirements/dev.txt: 
    # Include common dependencies
    -r common.txt

    # Development-specific packages
    pytest==7.1.0
    pylint==2.14.0
  • requirements/prod.txt: 
    # Include common dependencies
    -r common.txt

    # Production-specific packages
    gunicorn==20.1.0

Now, to install for development:

pip install -r requirements/dev.txt     # This will also pull in requirements/common.txt

And for production:

pip install -r requirements/prod.txt    # This will also pull in requirements/common.txt

Considerations and Potential Downsides​

  • Path Resolution: Paths in nested -r directives are typically relative to the file containing the directive.
  • pip freeze Complexity: When you run pip freeze > requirements.txt, it will output all installed packages into that single file. It will not preserve your nested -r structure. This makes managing and updating the "source" nested files more manual if you rely on pip freeze to update them.
  • Discoverability: It might be less obvious what the full set of dependencies is without opening multiple files.

For these reasons, Method 1 (multiple -r options on the command line) is often preferred for its explicitness and easier compatibility with pip freeze workflows for regenerating individual files.

Generating Requirements Files (pip freeze)​

To create or update your requirements files based on the currently installed packages in your environment:

# Activate your virtual environment first!
pip freeze > requirements.txt # Creates/overwrites requirements.txt

If managing separate files, you might need a more manual process or tools like pip-tools (pip-compile) for better management. For example, to update only dev.txt with currently installed dev tools:

pip freeze | grep -E "(pytest|pylint|black)" > dev.txt
note

This is a simplified example and might need refinement for complex setups.

Best Practices for Managing Multiple Requirements Files​

  • Use Virtual Environments: Always. This keeps dependencies isolated per project.
  • Prefer Command-Line -r: Use pip install -r file1.txt -r file2.txt for clarity and easier updates with pip freeze.
  • Be Specific with Versions: Pin versions (e.g., requests==2.25.1) or use compatible release specifiers (e.g., numpy>=1.20,<1.24) for reproducible builds.
  • Consider pip-tools: For more advanced dependency management, including compiling dependencies from a higher-level requirements.in file to a fully pinned requirements.txt, look into pip-tools (pip-compile and pip-sync). This can simplify managing nested or layered dependencies.

Troubleshooting: Upgrading pip and setuptools​

If you encounter issues with pip not recognizing multiple -r options correctly or other unexpected behavior, ensure your pip and setuptools are up to date:

python -m pip install --upgrade pip setuptools
# or
python3 -m pip install --upgrade pip setuptools

Conclusion​

pip provides flexible ways to manage dependencies across multiple requirements files.

  • Specifying multiple -r options on the command line (pip install -r dev.txt -r common.txt) is the most common and generally recommended approach for its explicitness and ease of use with pip freeze.
  • Nesting requirements files (placing -r common.txt inside dev.txt) is also possible but can make pip freeze workflows more complex to manage accurately.

Choose the method that best suits your project's structure and maintenance workflow, and always use virtual environments for robust dependency management.

Table of Contents