Skip to main content

How to Wait for Subprocesses in Python

When you launch a subprocess in Python (e.g., to run another program or command), you often need to wait for it to finish before proceeding.

This guide explains how to wait for subprocesses to complete, covering the recommended subprocess.run() method, the more flexible subprocess.Popen class, and using psutil for managing multiple processes.

The subprocess.run() function is the recommended way to run and wait for subprocesses in most situations. It's simpler and safer than using Popen directly.

import subprocess

# Run a command and wait for it to finish
result = subprocess.run(['echo', 'tutorialreference.com'], capture_output=True, text=True)
print(result.stdout) # Output: tutorialreference.com

result = subprocess.run(['echo', 'google.com'], capture_output=True, text=True)
print(result.stdout) # Output: google.com
  • subprocess.run(['echo', 'tutorialreference.com'], ...): Runs the command echo tutorialreference.com. The command and its arguments are passed as a list. This is safer than passing a single string, as it avoids shell injection vulnerabilities.
  • capture_output=True: Captures the standard output and standard error of the subprocess. Without this, the output would go directly to your console.
  • text=True: Decodes the output as text (using the system's default encoding, or you can specify encoding='utf-8' explicitly). If you omit this, you'll get bytes objects.
  • The result is a CompletedProcess object that holds all the important information.

Handling Timeouts​

You can specify a timeout (in seconds) to prevent your script from hanging indefinitely if the subprocess takes too long:

import subprocess

try:
    result = subprocess.run(['sleep', '5'], timeout=3, capture_output=True, text=True)
    # This will raise a TimeoutExpired exception
except subprocess.TimeoutExpired:
    print("Command timed out!")
  • The code will attempt to run the command and if there is a timeout error, it will be handled.

Handling Errors​

By default, subprocess.run() doesn't raise an exception if the subprocess returns a non-zero exit code (which usually indicates an error). To automatically raise an exception on error, use check=True:

import subprocess

try:
    result = subprocess.run(['ls', 'nonexistent_file'], capture_output=True, text=True, check=True)
except subprocess.CalledProcessError as e:
    print(f"Command failed with exit code {e.returncode}:")
    print(f"  stdout: {e.stdout}")
    print(f"  stderr: {e.stderr}")
  • The code checks if the execution of the command was successful.
  • If check=True and the exit code is non-zero then the CalledProcessError is raised.

Using subprocess.Popen (For More Control)​

subprocess.Popen gives you more fine-grained control over the subprocess, but it's also more complex to use correctly. Use it when you need to interact with the subprocess while it's running (e.g., send input to its standard input, read from its standard output/error in real-time).

import subprocess

p1 = subprocess.Popen(['echo', 'tutorialreference.com'], stdout=subprocess.PIPE, text=True)
p2 = subprocess.Popen(['echo', 'google.com'], stdout=subprocess.PIPE, text=True)
  • The example above creates two processes using the Popen class.

Waiting using wait()​

import subprocess
p1 = subprocess.Popen(['echo', 'tutorialreference.com'], stdout=subprocess.PIPE, text=True)
p2 = subprocess.Popen(['echo', 'google.com'], stdout=subprocess.PIPE, text=True)

exit_code1 = p1.wait()  # Wait for p1 to finish
print(exit_code1)       # Print exit code (usually 0 for success)

exit_code2 = p2.wait()  # Wait for p2 to finish
print(exit_code2)
  • p1.wait(): Blocks until the subprocess p1 completes. It returns the process's exit code.
  • You can also get the exit codes by using a list comprehension to call wait() on each object.
exit_codes = [p.wait() for p in (p1, p2)]
print(exit_codes) # Output: [0, 0]

Timeouts with wait()​

wait() can also accept a timeout argument.

import subprocess
p1 = subprocess.Popen(['echo', 'tutorialreference.com'], stdout=subprocess.PIPE, text=True)
def wait_child_process():
    exit_code1 = p1.wait(timeout=5) # Set timeout
    print(exit_code1)
try:
   wait_child_process()
except subprocess.TimeoutExpired:
   wait_child_process()
# Output: 0

Waiting for Multiple Processes with psutil​

The psutil library provides a cross-platform way to manage processes, including waiting for multiple processes to finish:

import subprocess
import psutil
import os # For os.getpid

# Start some long-running processes (replace with your actual commands)
p1 = subprocess.Popen(['sleep', '2']) # Runs for 2 seconds
p2 = subprocess.Popen(['sleep', '3']) # Runs for 3 seconds

processes = [psutil.Process(p1.pid), psutil.Process(p2.pid)]

def on_terminate(proc):
    print(f"process {proc} terminated with exit code {proc.returncode}")

gone, alive = psutil.wait_procs(processes, timeout=5, callback=on_terminate)

print("Processes that finished:", gone)
print("Processes still alive:", alive)
  • First install psutil using pip install psutil.
  • psutil.Process(pid): Creates a psutil.Process object representing the process with the given PID.
  • psutil.wait_procs(processes, timeout=3, callback=on_terminate): Waits for up to 3 seconds for the processes in the processes list to finish. The on_terminate function (if provided) is called when a process terminates.
  • gone: A list of psutil.Process objects that terminated.
  • alive: A list of psutil.Process objects that are still running (after the timeout).

Table of Contents