Skip to main content

Python NumPy: How to Fix "MemoryError / _ArrayMemoryError: Unable to allocate array with shape and data type"

When working with large datasets or creating massive arrays in NumPy, you might encounter a daunting error: numpy.core._exceptions._ArrayMemoryError: Unable to allocate X GiB for an array with shape (Y, Z, ...) and data type W (or a similar MemoryError). This error signifies a fundamental problem: your system does not have enough available contiguous memory to create the NumPy array as requested with the specified shape and data type.

This guide will thoroughly explain the common reasons behind this memory allocation failure, covering scenarios on both Linux (related to overcommit settings) and Windows (related to paging file size). We'll also explore crucial solutions such as ensuring a 64-bit Python installation, optimizing data types (e.g., using uint8), and strategies for processing large data in chunks.

Understanding the "Unable to Allocate Array" Error​

Memory Requirements of NumPy Arrays​

NumPy arrays require a contiguous block of memory. When you request an array of a certain shape and data type, NumPy asks the operating system for that amount of memory. If the OS can not find a large enough single block of free RAM (and potentially swap/page file space, depending on OS settings), it denies the request, leading to the memory allocation error.

How Array Size is Calculated​

The error message often indicates the amount of memory NumPy tried to allocate. This is calculated as: Total Elements * Bytes per Element

  • Total Elements: Product of all dimensions in the shape (e.g., for shape (1000, 1000, 100), total elements = 100,000,000).
  • Bytes per Element: Depends on the dtype.
    • uint8, int8: 1 byte
    • int16, float16: 2 bytes
    • int32, float32: 4 bytes
    • int64, float64 (default for floats): 8 bytes
    • complex64: 8 bytes
    • complex128: 16 bytes
    • object: Size of pointers + object overhead (can be larger and less predictable).

Solutions for Linux Systems​

On Linux, memory allocation behavior is influenced by the vm.overcommit_memory kernel parameter.

Checking vm.overcommit_memory​

You can check the current setting:

cat /proc/sys/vm/overcommit_memory
  • 0 (Default): Heuristic overcommit handling. The kernel estimates if enough memory is available. Large requests might be denied if they seem too risky, even if technically there's enough swap. This is often the cause of the error for very large NumPy array requests on Linux.
  • 1: Always overcommit. The kernel allows virtually any memory allocation, assuming memory will be available when actually accessed. This can lead to the OOM (Out Of Memory) killer terminating processes if physical memory + swap is exhausted.
  • 2: Don't overcommit. The kernel limits allocations to a value related to total available swap space and a percentage of physical RAM (defined by vm.overcommit_ratio).

Adjusting vm.overcommit_memory (Temporary and Persistent)​

Setting vm.overcommit_memory to 1 can often resolve the "Unable to allocate" error for NumPy on Linux, as it makes the kernel more permissive.

  • Temporary Change (until next reboot):

    # You need superuser privileges
    sudo su
    echo 1 > /proc/sys/vm/overcommit_memory
    exit # Exit superuser shell

    After this, your Python script attempting to create the large NumPy array might succeed if the issue was purely the heuristic denial.

  • Persistent Change (method varies by Linux distribution): To make the change survive reboots, you typically edit /etc/sysctl.conf or a file in /etc/sysctl.d/.

    1. Open /etc/sysctl.conf (or create a new file like /etc/sysctl.d/99-numpy-overcommit.conf) with sudo: 
      sudo nano /etc/sysctl.conf
    2. Add or modify the line: 
      vm.overcommit_memory = 1
    3. Save the file.
    4. Apply the changes without rebooting: 
      sudo sysctl -p
    warning

    Caution: Setting vm.overcommit_memory = 1 globally can have system-wide implications. If your system truly runs out of memory, the OOM killer might terminate important processes. Use with understanding.

Solutions for Windows Systems​

On Windows, the equivalent concept to overcommit handling involves the virtual memory paging file size. If this is too small or managed in a restrictive way, large memory allocations can fail.

Adjusting Virtual Memory (Paging File Size)​

  1. Press Windows Key, type "SystemPropertiesAdvanced", and open it (run as administrator if prompted).
  2. In the "Advanced" tab, under the "Performance" section, click "Settings...".
  3. Go to the "Advanced" tab in the "Performance Options" window.
  4. Under "Virtual memory," click "Change...".
  5. Uncheck "Automatically manage paging file size for all drives."
  6. Select the drive where your paging file is located (usually C:).
  7. Select "Custom size:".
    • Initial size (MB): A common recommendation is 1.5 * Your Total RAM in MB.
    • Maximum size (MB): A common recommendation is 3 * Your Total RAM in MB (or 3 * Initial Size).
    • Example: If you have 16 GB RAM (16 * 1024 = 16384 MB):
      • Initial size: 16384 * 1.5 = 24576 MB
      • Maximum size: 16384 * 3 = 49152 MB
  8. Click "Set," then "OK" on all dialogs.
  9. Restart your computer for the changes to take effect.

This increases the disk space Windows can use as an extension of RAM, potentially allowing larger allocations. However, relying heavily on a slow disk for virtual memory will severely impact performance.

Platform-Agnostic Solutions and Best Practices​

These apply to any operating system:

Ensure a 64-bit Python Installation​

A 32-bit Python interpreter can only address a maximum of ~4 GB of memory, regardless of how much physical RAM your system has. For large data, a 64-bit Python installation is essential.

  • Check your Python architecture: 
    import sys
    is_64bit = sys.maxsize > 2**32
    print(f"Python is 64-bit: {is_64bit}")
  • If False, download and install a 64-bit version of Python (often labeled x86-64 or amd64) from python.org. Also, ensure your IDE (like PyCharm) is using the 64-bit interpreter.

Optimize Data Types (e.g., dtype='uint8')​

The error message often specifies the dtype. If your data can fit into a smaller data type, use it to reduce memory significantly.

  • np.zeros(shape, dtype='uint8'): Unsigned 8-bit integers (0-255). Uses 1 byte per element.
  • np.zeros(shape, dtype='float32'): Single-precision float. Uses 4 bytes, vs. 8 for default float64.
  • np.zeros(shape, dtype='int16'): Signed 16-bit integers. Uses 2 bytes.

Choose the smallest dtype that can accurately represent your data range.

import numpy as np
# Example: If values are only 0-255 and non-negative
# Original error was with this large shape and uint8
# arr = np.zeros((126816, 36, 55406), dtype='uint8') # Needs ~236 GiB

# If you can use a much smaller shape because your problem allows it:
smaller_shape = (10000, 1000, 10) # Needs 10000*1000*10*1 byte = ~95 MiB
try:
    arr_smaller = np.zeros(smaller_shape, dtype='uint8')
    print(f"Successfully allocated array of shape {arr_smaller.shape} with uint8.")
except np.core._exceptions._ArrayMemoryError as e:
    print(f"Still unable to allocate even smaller array: {e}")

Process Data in Chunks or Batches​

If the entire dataset doesn't need to be in memory at once, process it in smaller chunks:

  • When reading files (e.g., CSVs with Pandas, HDF5 files), many libraries offer ways to read and process data iteratively in chunks.
  • If generating data, generate and process parts of the array sequentially, saving intermediate results if needed, rather than creating one massive array.

Use Memory-Efficient Libraries or Techniques​

  • Sparse Arrays: If your array contains mostly zeros (or a constant value), libraries like scipy.sparse offer sparse matrix formats that store only non-zero elements, saving enormous amounts of memory.
  • Memory-Mapped Files (np.memmap): For arrays larger than RAM that need to be persisted to disk and accessed like in-memory arrays (though with performance overhead for disk I/O). np.memmap creates an array whose data resides on disk.

When the Data is Simply Too Large for RAM​

Sometimes, the requested array is genuinely larger than your available physical RAM and configured virtual memory/swap combined. In such cases:

  • Add More RAM: The most direct hardware solution.
  • Cloud Computing/HPC: Utilize machines with significantly more memory.
  • Distributed Computing: Frameworks like Dask or Spark can distribute array computations across multiple machines if the problem is parallelizable.
  • Algorithmic Changes: Re-think your algorithm to avoid creating such a massive intermediate array. Can the computation be done iteratively or with a different data structure?

Conclusion​

The NumPy Unable to allocate array error is a signal that your system's memory resources (RAM and virtual memory/swap) are insufficient for the requested array size, or OS-level memory management policies are preventing the allocation.

  1. Linux: Check and consider adjusting vm.overcommit_memory (temporarily or persistently).
  2. Windows: Check and consider increasing your virtual memory (paging file size).
  3. Always:
    • Ensure you are using a 64-bit Python installation.
    • Optimize data types (dtype) to use the smallest appropriate type.
    • If possible, process data in chunks rather than loading/creating everything into one giant array.
    • Explore memory-efficient alternatives like sparse arrays or memory-mapped files if applicable.

If these steps don't resolve the issue, the requested array size may fundamentally exceed your current hardware capabilities, requiring a re-evaluation of your approach or hardware resources.

Table of Contents