How to Resolve Python ValueError: Circular reference detected (in JSON Serialization)

When serializing Python objects into JSON format using json.dumps(), you might encounter the ValueError: Circular reference detected. This error indicates that the object structure you're trying to serialize contains a loop, i.e. an object refers back to itself, either directly or indirectly through nested objects. The JSON format standard doesn't support these circular references, hence the error.

This guide explains what circular references are, why they cause issues with json.dumps(), and provides methods to resolve the error.

Understanding the Error: Circular References and JSON

JSON (JavaScript Object Notation) is a data interchange format based on a subset of JavaScript syntax. It supports objects (key-value pairs), arrays (ordered lists), strings, numbers, booleans, and null. Crucially, it does not have a standard way to represent an object that contains a reference back to itself or an ancestor object in its hierarchy. Attempting to serialize such a structure would theoretically lead to infinite recursion. Python's json.dumps() detects this potential infinite loop and raises the ValueError to prevent it.

What is a Circular Reference?

A circular reference occurs when an object (like a dictionary or list) contains a reference (a pointer in memory) to itself, or when a chain of references leads back to an earlier object in the chain.

# Example: Direct circular reference in a dictionary
my_dict = {'id': 1}
print(f"Object ID before cycle: {id(my_dict)}")

# ⚠️ Creating a circular reference
my_dict['self_ref'] = my_dict # The 'self_ref' key points back to the dict itself

print(f"\nDictionary structure (repr might abbreviate): {my_dict}")
# Output might look like: {'id': 1, 'self_ref': {...}}

# Verify using the 'is' operator (checks if they are the SAME object in memory)
is_circular = my_dict['self_ref'] is my_dict
print(f"Is my_dict['self_ref'] the same object as my_dict? {is_circular}")
# Output: Is my_dict['self_ref'] the same object as my_dict? True

In this case, my_dict['self_ref'] doesn't just contain similar data; it points to the exact same dictionary object in memory.

Why json.dumps() Fails with Circular References

When json.dumps() traverses the object structure to convert it to a JSON string, it follows references. If it encounters a circular reference like my_dict['self_ref'], it would try to serialize my_dict again, which contains my_dict['self_ref'], leading back to my_dict, and so on infinitely. The ValueError prevents this infinite loop.

import json

my_dict = {'id': 1}
my_dict['self_ref'] = my_dict # Create the cycle

try:
    # ⛔️ ValueError: Circular reference detected
    json_string = json.dumps(my_dict, indent=2) # Using indent for readability if it worked
    print(json_string)
except ValueError as e:
    print(e)

Solution 1: Avoid Creating the Cycle (Use Copies)

If the intention wasn't to create a true circular reference but merely to include a snapshot or copy of the object's current state at that point, use a copying method when assigning the value.

Shallow Copy (.copy())

A shallow copy creates a new dictionary object, but the values inside it are still references to the original objects (if those values are mutable, like nested lists/dicts). For simple circular references where the top-level object refers to itself, a shallow copy is often sufficient to break the cycle for JSON serialization.

import json

my_dict = {'id': 1}
print(f"Original dict ID: {id(my_dict)}")

# Use a shallow copy instead of direct assignment
my_dict['nested_copy'] = my_dict.copy()
print(f"Copy dict ID: {id(my_dict['nested_copy'])}")

# Verify they are different objects now
is_still_circular = my_dict['nested_copy'] is my_dict
print(f"Is 'nested_copy' the same object? {is_still_circular}") # Output: False

# Serialization now works
json_string = json.dumps(my_dict, indent=2)
print(f"\nJSON output (with copy):\n{json_string}")

Output:

JSON output (with copy):
{
  "id": 1,
  "nested_copy": {
    "id": 1
  }
}

Using .copy() creates a distinct dictionary object for the nested_copy value, breaking the circular reference at the top level.

Deep Copy (copy.deepcopy())

If your dictionary contains nested mutable objects (like lists or other dictionaries) and you need to ensure that the copy is completely independent (even nested objects are copied, not just referenced), use copy.deepcopy().

import json
import copy # Required import for deepcopy

my_dict = {'id': 1, 'items': [10, 20]}

# Use a deep copy for complete independence
my_dict['nested_deep_copy'] = copy.deepcopy(my_dict)

# Verify nested lists are different objects
is_nested_list_same = my_dict['items'] is my_dict['nested_deep_copy']['items']
print(f"Are nested lists the same object? {is_nested_list_same}") # Output: False

# Serialization works
json_string = json.dumps(my_dict, indent=2)
print(f"\nJSON output (with deepcopy):\n{json_string}")

Output:

JSON output (with deepcopy):
{
  "id": 1,
  "items": [
    10,
    20
  ],
  "nested_deep_copy": {
    "id": 1,
    "items": [
      10,
      20
    ]
  }
}

While deepcopy also solves the circular reference for json.dumps, it's more resource-intensive than a shallow copy. Use it when true independence of nested mutable structures is required.

Solution 2: Remove the Cycle Before Serialization

If the circular reference exists but shouldn't be part of the JSON output, remove or modify the problematic key before calling json.dumps().

Using del

Permanently remove the key causing the cycle.

import json

my_dict = {'id': 1}
my_dict['self_ref'] = my_dict # Create the cycle

# Delete the key causing the circular reference
del my_dict['self_ref']

# Serialization now works
json_string = json.dumps(my_dict, indent=2)
print(f"JSON output (after del):\n{json_string}")

Output:

JSON output (after del):
{
  "id": 1
}

Setting to None

Replace the circular reference with None (which is JSON serializable as null).

import json

my_dict = {'id': 1, 'parent': None}
my_dict['parent'] = my_dict # Create the cycle

# Set the key causing the cycle to None (or another serializable value)
my_dict['parent'] = None

# Serialization now works
json_string = json.dumps(my_dict, indent=2)
print(f"JSON output (after setting to None):\n{json_string}")

Output:

JSON output (after setting to None):
{
  "id": 1,
  "parent": null
}

Solution 3: Custom Handling (Advanced)

For complex, deeply nested structures where manually finding and breaking cycles is difficult, you might need a custom function that traverses the object graph, keeps track of visited object IDs (using id(obj) and a set), and replaces detected cycles with a placeholder like None or a special string marker. Implementing this correctly is non-trivial. Libraries specifically designed for handling complex object graphs might offer solutions, or you could adapt algorithms found in resources like Stack Overflow (being careful to understand their limitations). This is generally beyond a simple fix.

Potential Issue: Incorrect default Function Usage

While less direct, the ValueError could theoretically occur if you provide a custom function to the default argument of json.dumps(), and that function itself somehow returns an object that leads back into a cycle or isn't serializable in a way that breaks the cycle detection.

import json

a_dict = {}
a_dict['self'] = a_dict # Circular

# Example of a potentially problematic default function (contrived)
def problematic_default(obj):
    print(f"Default called for type: {type(obj)}")
    # If this function somehow returned 'a_dict' again for an unhandled type,
    # it could re-introduce the cycle. Or if it raised an unexpected error.
    # A correct default should return a *serializable* representation or raise TypeError.
    raise TypeError(f"Default cannot handle {type(obj)}")

try:
    # Usually, the error happens before default is called for the main dict cycle.
    # This demonstrates where default *could* cause issues if misused.
    # json.dumps(a_dict, default=problematic_default) # This still errors early
    pass # Error happens before default is invoked for the cycle itself
except ValueError as e:
    # Error still originates from the cycle detection
    print(f"Original Error: {e}")
except TypeError as e:
     print(f"TypeError from default function: {e}")

Ensure your default function returns a valid, serializable JSON representation (string, number, list, dict, bool, None) or raises a TypeError for objects it can't handle, rather than returning something that perpetuates or causes a cycle.

Conclusion

The ValueError: Circular reference detected occurs when json.dumps() encounters an object structure that refers back to itself, which cannot be represented in standard JSON.

To resolve this:

1. Avoid creating the cycle: If you meant to include a snapshot, use .copy() (shallow) or copy.deepcopy() when assigning the problematic value.
2. Break the cycle before serialization: Remove the key (del my_dict['key']) or set it to a serializable placeholder (my_dict['key'] = None) before calling json.dumps().
3. For complex cases: Consider advanced traversal functions or libraries designed to handle object graphs if manual breaking isn't feasible.
4. Ensure any custom default function passed to json.dumps() returns serializable types or raises TypeError, not values that could cause cycles.

Modifying the data structure to remove or avoid the circular reference before serialization is the key to fixing this error.