Skip to main content

How to Resolve Python Protobuf Error "TypeError: Descriptors cannot not be created directly"

When working with Python libraries that utilize Google's Protocol Buffers (protobuf), you might encounter the error TypeError: Descriptors cannot not be created directly. This typically arises after updating the protobuf library to version 4.x or higher, often as a dependency of another package you installed. The error indicates an incompatibility between the newer protobuf library internals and older generated code files (_pb2.py) used by other packages.

This guide explains the cause related to recent protobuf changes and provides the recommended solutions to fix this error.

Understanding the Error: Protobuf v4+ Breaking Changes​

The protobuf library underwent significant changes starting with version 4.21.0. It began using a C extension based on the upb library for faster message parsing and serialization, replacing the previous pure-Python or slower C++ implementation.

This new implementation requires that the Python code generated from .proto definition files (the _pb2.py files) must be created using protoc (the protobuf compiler) version 3.19.0 or newer.

The error message itself often provides clues:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

The Root Cause: Outdated Generated Code (_pb2.py)​

The core issue is that you have protobuf version 4.x+ installed, but another package in your environment (which depends on protobuf) is still using _pb2.py files that were generated with an older protoc version (< 3.19.0). These older generated files are incompatible with the new C-based implementation in protobuf 4.x+.

Even if you didn't explicitly install protobuf, it's often installed as a dependency for packages like tensorflow, google-cloud-*, tensorboard, and many others.

The most common and generally recommended solution is to downgrade and pin your protobuf installation to the latest version in the 3.20.x series. This version uses the older implementation compatible with most existing generated code.

  1. Uninstall existing Protobuf (if any > 3.20.x):

    pip uninstall protobuf
    # or pip3 uninstall protobuf
    # or python -m pip uninstall protobuf
    note

    Answer 'y' if prompted.

  2. Install the latest 3.20.x version:

    # Use pip or pip3 or python -m pip as appropriate for your setup
    pip install "protobuf==3.20.*"
    • The "protobuf==3.20.*" syntax installs the latest available patch release within the 3.20 series.

  3. Update requirements.txt (if used): Add or modify the line in your requirements.txt file to explicitly require this version:

    protobuf==3.20.*

  4. (Optional) Verify Installation:

    pip show protobuf

    Check that the installed version starts with 3.20.. If you still have issues or the wrong version installed (due to dependency conflicts), you might need to force the reinstall:

    pip install "protobuf==3.20.*" --force-reinstall

Solution 2: Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION Env Var (Workaround)​

As mentioned in the error message, you can force protobuf (even version 4+) to use its older, pure-Python implementation by setting an environment variable. Note: This will significantly decrease performance, especially for parsing large messages.

  • Linux / macOS:

    export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
    # Now run your Python script
    note

    This setting is temporary for the current shell session.

  • Windows Command Prompt:

    set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
    :: Now run your Python script
    note

    Temporary for the current CMD session.

    Use setx for a more permanent setting (requires new CMD window):

    setx PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION python

  • Windows PowerShell:

    $Env:PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="python"
    # Now run your Python script
    note

    Temporary for the current PowerShell session.

While this avoids the error, the performance impact makes pinning to 3.20.x (Solution 1) generally preferable until dependent packages update their generated code.

Solution 3: Upgrade the Dependent Package​

The ideal long-term solution is for the maintainers of the package causing the error (the one with the outdated _pb2.py files) to release a new version generated with a newer protoc. You can try upgrading that specific package:

# Replace <dependent-package-name> with the actual package (e.g., tensorflow, google-api-core)
pip install --upgrade <dependent-package-name>

Check the changelog or release notes for that package to see if they mention compatibility with Protobuf 4+. If upgrading the dependent package doesn't fix it, you'll likely need to revert to Solution 1 (pinning Protobuf).

Special Case: Google App Engine and Cloud Libraries​

This error became particularly prominent for users of Google Cloud client libraries and Google App Engine because many of Google's own libraries contained older generated code. While Google is updating its libraries, the most reliable immediate fix remains pinning protobuf==3.20.* in your requirements.txt file.

Debugging: Identifying the Dependency​

If you're unsure which package is causing the issue:

  • Examine the full traceback of the error message; it often originates within the files of a specific installed package.
  • Check your requirements.txt or pip freeze output for packages known to use Protobuf (Google libraries, TensorFlow, ONNX, etc.).
  • Try uninstalling potential candidates one by one (in a safe test environment!) to see if the error disappears, which helps isolate the problematic dependency.

Conclusion​

The TypeError: Descriptors cannot not be created directly error is a direct consequence of the Protobuf library's version 4+ upgrade, which requires dependent packages to regenerate their Protobuf Python files (_pb2.py) using protoc >= 3.19.0.

  • The most reliable immediate solution is to downgrade and pin the protobuf package to the latest 3.20.x version: pip install "protobuf==3.20.*". Remember to update your requirements.txt.
  • Setting the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python environment variable works but negatively impacts performance.
  • Upgrading the specific package that uses Protobuf might resolve the issue if its maintainers have released a compatible version.

Monitor the packages you depend on; eventually, most will update to support Protobuf 4+, allowing you to remove the version pin.

Table of Contents