Skip to main content

How to Resolve Python Error "UnicodeEncodeError: 'charmap' codec can't encode characters..."

The UnicodeEncodeError: 'charmap' codec can't encode characters in position X-Y: character maps to <undefined> is a Python error indicating that you are trying to convert a string containing certain Unicode characters into a sequence of bytes using an encoding (like cp1252, cp437, often referred to generically as 'charmap' in error messages, especially on Windows) that cannot represent those specific characters.

This guide explains why this encoding error occurs and provides solutions focused on using a more capable encoding like UTF-8.

Understanding the Error: Encoding Limitations​

  • Strings (str): Represent text using Unicode, capable of holding characters from virtually any language or symbol set.
  • Bytes (bytes): Represent raw sequences of byte values (0-255).
  • Encoding: The process of converting a str to bytes. Each encoding (codec) defines a mapping between characters and byte sequences.
  • Limited Encodings (charmap, cp1252, ascii): Older or simpler encodings like ASCII or various "charmap" encodings (such as cp1252, the default on many Western Windows systems) can only represent a small subset of Unicode characters (typically 128 or 256). They literally have no byte representation defined for characters outside their specific set.
  • UTF-8: A universal encoding capable of representing all valid Unicode characters using variable-length byte sequences.

The UnicodeEncodeError: 'charmap' codec can't encode... occurs when you attempt the encoding step (str -> bytes) using a limited codec like 'charmap' or 'cp1252', but your string contains Unicode characters (like '😊', '€', 'é', '加大', '𝘈') that are outside the range representable by that specific codec.

The Cause: Encoding Unicode Characters with Limited Codecs (like charmap/cp1252)​

The error happens when Python tries to map a character from your string to a byte sequence using the specified (or default) limited encoding, finds no valid mapping for that character, and raises the error. This often occurs implicitly when writing to files or terminals if the default system encoding is not UTF-8.

A common error scenario is when a string contains characters not present in typical charmap encodings like cp1252:

my_string = "Euro symbol: €"

try:
    # ⛔️ UnicodeEncodeError: 'charmap' codec can't encode character '\u20ac' in position 13: character maps to <undefined>
    # Attempting to encode using cp1252 (a common 'charmap') which might not have € in all versions/setups
    # Or Python might default to another limited encoding causing the error.
    encoded_bytes = my_string.encode('cp1252') # Or some other 'charmap' like cp437
    print(encoded_bytes)
except UnicodeEncodeError as e:
    print(e)

Another common scenario: Writing to console/file with default limited encoding:

my_string = "Euro symbol: €"

try:
    print(my_string) # Might raise error if console encoding is not UTF-8
except UnicodeEncodeError as e:
    print(f"Error printing: {e}")

Solution 1: Specify a Capable Encoding (Usually utf-8)​

The most robust and recommended solution is to explicitly use the UTF-8 encoding whenever encoding strings or writing text files that might contain any non-ASCII characters.

For str.encode()​

my_string = "Euro symbol: € / Cyrillic: ф"

# ✅ Encode using UTF-8
utf8_bytes = my_string.encode('utf-8')

print(f"Original: '{my_string}'")
print(f"UTF-8 Bytes: {utf8_bytes}")
# Output: UTF-8 Bytes: b'Euro symbol: \xe2\x82\xac / Cyrillic: \xd1\x84'

# Verify by decoding back
print(f"Decoded: '{utf8_bytes.decode('utf-8')}'") # Works

Output:

Original: 'Euro symbol: € / Cyrillic: ф'
UTF-8 Bytes: b'Euro symbol: \xe2\x82\xac / Cyrillic: \xd1\x84'
Decoded: 'Euro symbol: € / Cyrillic: ф'

For open() (Writing to Files)​

Always specify encoding='utf-8' when writing text files.

my_string = "Résumé with accents and symbols € α β"
filename = "output_file.txt"

try:
    # ✅ Specify UTF-8 encoding for writing
    with open(filename, 'w', encoding='utf-8') as f:
        f.write(my_string)
    print(f"Successfully wrote '{filename}' using UTF-8.")
except Exception as e:
    print(f"Error writing file: {e}")

Output:

Successfully wrote 'output_file.txt' using UTF-8.

For Libraries (e.g., csv, BeautifulSoup)​

Ensure UTF-8 encoding is used when interacting with libraries that handle text or files.

  • CSV: 
    import csv
    data = [["Name", "Comment"], ["Alice", "Great work 😊"]]
    with open('output.csv', 'w', newline='', encoding='utf-8') as csvfile: # ✅ Specify encoding
        writer = csv.writer(csvfile)
        writer.writerows(data)
  • BeautifulSoup: When converting soup back to bytes, specify UTF-8. 
    # Assuming 'soup' is a BeautifulSoup object
    # print(soup) # Printing might use default encoding
    # ✅ Encode explicitly for reliable byte output
    utf8_output_bytes = soup.encode("utf-8")
    # print(utf8_output_bytes)

Solution 2: Handle Encoding Errors (errors parameter - Use Cautiously)​

The .encode() method and open() function accept an errors parameter to handle characters that cannot be encoded. Using this often leads to data loss or alteration and should generally be avoided if possible.

Using errors='ignore' (Data Loss)​

Silently discards unencodable characters.

my_string = "Price: €10"
# ⚠️ Ignoring errors leads to data loss
encoded_bytes = my_string.encode('ascii', errors='ignore')
print(f"Encoded ('ascii', ignore): {encoded_bytes}") # Output: b'Price: 10' (€ lost)

Output:

Encoded ('ascii', ignore): b'Price: 10'

Using errors='replace' (Replacement Character)​

Replaces unencodable characters with a placeholder (usually ?).

my_string = "Price: €10"
# ⚠️ Replacing errors alters data
encoded_bytes = my_string.encode('ascii', errors='replace')
print(f"Encoded ('ascii', replace): {encoded_bytes}")

Output:

Encoded ('ascii', replace): b'Price: ?10'

Other Handlers (xmlcharrefreplace, backslashreplace)​

Replace with XML entities (&#...;) or Python backslash escapes (\u...).

Useful sometimes, but doesn't produce simple encoded text.

note

Prefer using UTF-8 (Solution 1) over error handlers.

Solution 3: Set System/Environment Default Encoding (Advanced)​

Modifying the default encoding Python uses for I/O can sometimes work around issues in complex environments or when dealing with libraries that don't allow specifying encoding easily. This is generally less preferred than explicit encoding specification in code.

PYTHONIOENCODING Environment Variable​

Setting this before launching Python forces standard input, output, and error streams to use the specified encoding.

# Linux/macOS
export PYTHONIOENCODING=utf-8

# Windows (Command Prompt)
set PYTHONIOENCODING=utf-8

# Windows (PowerShell)
$env:PYTHONIOENCODING = 'utf-8'

# Now run your python script
python your_script.py

sys.reconfigure (Use Sparingly)​

Within a script (at the very beginning), you can try to reconfigure standard streams. This is fragile and might not always work depending on the system state.

import sys
import io

# --- Attempt to force UTF-8 (Use with caution) ---
# This needs to run VERY early in your script execution
try:
    if sys.stdout.encoding != 'utf-8':
      sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
    if sys.stderr.encoding != 'utf-8':
      sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')
    # sys.stdin might also need reconfiguring if reading input
    print("Attempted to reconfigure stdio streams to UTF-8.")
except Exception as e:
    print(f"Could not reconfigure streams: {e}")

# --- Rest of your script ---
my_string = "Euro symbol: €"
print(my_string) # Now might work even if console default wasn't UTF-8

Output:

Attempted to reconfigure stdio streams to UTF-8.
Euro symbol: €

Debugging the Error​

  1. Identify Operation: Is the error from .encode(), open(), print(), or a library function?
  2. Check String Content: Does the string contain any non-ASCII characters? Use print(repr(my_string)) to reveal unusual characters.
  3. Verify Encoding: What encoding is being explicitly specified or implicitly used? If it's 'ascii' or a specific 'charmap'/'cp...', that's likely the issue if non-ASCII characters are present.
  4. Test with UTF-8: Try explicitly using encoding='utf-8' for the operation. Does the error disappear?

Conclusion​

The UnicodeEncodeError: 'charmap' codec can't encode characters... occurs when trying to convert a string containing Unicode characters beyond the limited repertoire of the specified encoding (like 'ascii', 'cp1252', or other locale-specific 'charmaps').

The most robust and recommended solution is:

  1. Use UTF-8 encoding consistently whenever encoding strings or writing text data that might contain non-ASCII characters:
    • my_string.encode('utf-8')
    • open(filename, 'w', encoding='utf-8')
    • Specify encoding='utf-8' in library functions (like csv.writer, pandas.read_csv if applicable during writing, although reading is more common for decode errors).

Using error handlers like errors='ignore' or 'replace' with limited encodings like 'ascii' typically leads to data loss and should be avoided. Setting global encodings is possible but less explicit than handling it directly in your code with utf-8.

Table of Contents