How to Resolve Python Error "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff..."

The UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte is a specific Python error indicating a problem during text decoding. It typically occurs when you try to interpret a sequence of bytes as text using the UTF-8 encoding standard, but the very beginning of the data (position 0) contains a byte (0xff or 0xfe) that signals the presence of a Byte Order Mark (BOM) for a different encoding, most commonly UTF-16 or sometimes UTF-32.

This guide explains BOMs, why UTF-8 fails on them, and provides solutions involving specifying the correct UTF-16/UTF-32 encoding or handling BOMs appropriately.

Understanding the Error: UTF-8 vs. Byte Order Marks (BOM)

UTF-8: A variable-width encoding standard. It does not typically use a BOM, although one (EF BB BF) exists but is discouraged. Crucially, the byte sequence 0xFF 0xFE (or 0xFE 0xFF) is invalid at the beginning of a UTF-8 stream according to the standard.
UTF-16 & UTF-32: These encodings use 2 bytes (UTF-16) or 4 bytes (UTF-32) per character. Because computer systems can store multi-byte sequences in different orders ("endianness"), these encodings often use a Byte Order Mark (BOM) at the very beginning of the byte stream to indicate the byte order.
- 0xFF 0xFE: BOM for UTF-16 Little Endian.
- 0xFE 0xFF: BOM for UTF-16 Big Endian.
- 0x00 0x00 0xFE 0xFF: BOM for UTF-32 Big Endian.
- 0xFF 0xFE 0x00 0x00: BOM for UTF-32 Little Endian.
The Error: The error ...can't decode byte 0xff in position 0... occurs because the UTF-8 decoder sees the first byte 0xFF. This byte is not a valid start for any UTF-8 character sequence, and it strongly suggests the data is actually encoded using UTF-16 Little Endian (or possibly UTF-32 LE).

The Cause: Decoding UTF-16/UTF-32 BOM as UTF-8

The error arises when your byte data starts with a UTF-16 or UTF-32 BOM, but you attempt to decode it using the 'utf-8' codec.

Scenario: Decoding Byte Objects

# String containing a character representable in UTF-16
original_string = "Hello ÿ" # ÿ might cause multi-byte in UTF-8, simple in UTF-16

# Encode using UTF-16 (often adds a BOM like b'\xff\xfe' at the start)
utf16_bytes = original_string.encode('utf-16')
print(f"Encoded bytes (UTF-16): {utf16_bytes}")
# Output: b'\xff\xfeH\x00e\x00l\x00l\x00o\x00 \x00\xff\x00' (Starts with 0xff 0xfe BOM)

try:
    # ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
    # Trying to decode UTF-16 bytes (with BOM) as UTF-8
    decoded_string = utf16_bytes.decode('utf-8')
    print(decoded_string)
except UnicodeDecodeError as e:
    print(e)

Scenario: Reading Files (`open()`, `pandas.read_csv()`)

This frequently happens when trying to read files (especially those originating from Windows applications like Excel saved as "Unicode Text") that are saved with UTF-16 encoding (including a BOM) using encoding='utf-8'.

# Assume 'my_utf16_file.txt' is saved with UTF-16 encoding (with BOM)

try:
    # ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0...
    with open('my_utf16_file.txt', 'r', encoding='utf-8') as f:
        content = f.read()
except UnicodeDecodeError as e:
    print(f"Error reading file with UTF-8: {e}")
except FileNotFoundError:
    print("Error: File not found.")

# Similarly with pandas
import pandas as pd
try:
    # ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0...
    df = pd.read_csv('my_utf16_file.csv', encoding='utf-8')
except UnicodeDecodeError as e:
    print(f"Error reading CSV with pandas/UTF-8: {e}")
except FileNotFoundError:
    print("Error: File not found.")

Solution 1: Specify the Correct Encoding (`utf-16`, `utf-32`)

The most reliable solution is to use the encoding that matches the data, which is likely 'utf-16' (or potentially 'utf-32') if you see the 0xff or 0xfe byte error at position 0.

For `bytes.decode()`

original_string = "Hello ÿ"
utf16_bytes = original_string.encode('utf-16') # b'\xff\xfe...'

# ✅ Decode using the original encoding ('utf-16')
try:
    decoded_string = utf16_bytes.decode('utf-16')
    print(f"Correctly decoded: '{decoded_string}'")
    # Output: Correctly decoded: 'Hello ÿ'
except UnicodeDecodeError as e:
    print(f"Unexpected error with utf-16: {e}")

For `open()`

filename = 'my_utf16_file.txt' # Assume UTF-16 encoded file

try:
    # ✅ Specify 'utf-16' encoding when opening
    with open(filename, 'r', encoding='utf-16') as f:
        content = f.read()
        print(f"File content read with utf-16:\n'{content}'")
except FileNotFoundError:
    print(f"Error: File '{filename}' not found.")
except Exception as e:
    print(f"Error reading file with utf-16: {e}")

For `pandas.read_csv()`

import pandas as pd
filename = 'my_utf16_file.csv' # Assume UTF-16 encoded CSV

try:
    # ✅ Specify 'utf-16' encoding for pandas
    df = pd.read_csv(filename, encoding='utf-16')
    print("Pandas DataFrame read successfully with utf-16.")
    # print(df.head())
except FileNotFoundError:
    print(f"Error: File '{filename}' not found.")
except Exception as e:
    print(f"Other error reading CSV with utf-16: {e}")

Solution 2: Use Specific BOM-Aware Encodings (`utf-8-sig`, `utf-16`, `utf-32`)

Python provides specific encoding names that automatically handle the BOM:

'utf-16': Automatically detects 0xFFFE or 0xFEFF BOM to determine endianness and consumes it.
'utf-32': Automatically detects UTF-32 BOMs (00 00 FE FF or FF FE 00 00) and consumes them.
'utf-8-sig': This encoding specifically looks for and consumes the UTF-8 BOM (EF BB BF) if present at the start. It then decodes the rest as standard UTF-8. While the error mentions 0xff (suggesting UTF-16), using utf-8-sig is relevant if you suspect a file might have the less common UTF-8 BOM.

Using 'utf-16' as shown in Solution 1 is often sufficient as it handles the BOM correctly. Using 'utf-8-sig' is only relevant if you specifically suspect a UTF-8 BOM (which doesn't start with 0xff).

Solution 3: Read File in Binary Mode (`'rb'`)

If you don't need to interpret the file as text (e.g., copying bytes, uploading), read it in binary mode. This avoids decoding altogether.

filename = 'my_utf16_file.txt'

try:
    # ✅ Read raw bytes
    with open(filename, 'rb') as f:
        byte_content = f.read()
        print(f"Read {len(byte_content)} bytes from {filename}.")
        # Note the BOM bytes at the start if present
        print(f"First 10 bytes: {byte_content[:10]}")
except FileNotFoundError:
    print(f"Error: File '{filename}' not found.")
except Exception as e:
    print(f"Error reading file in binary mode: {e}")

Solution 4: Handle Decoding Errors (`errors` parameter - Use Cautiously)

You can use errors='ignore' or errors='replace' with the 'utf-8' codec, but this is strongly discouraged for this specific error (0xff at position 0). It will likely corrupt the beginning of your data because the BOM bytes will be misinterpreted or discarded, potentially causing the rest of the decoding (if it were actually UTF-16) to fail in different ways. Stick to Solution 1 or 2.

Debugging: Identifying the Encoding

If you aren't sure of the file's encoding:

Using Linux/macOS file command:

file my_file.txt
# Example output might be: "my_file.txt: Little-endian UTF-16 Unicode text, with CRLF line terminators"

Using Windows Notepad's "Save As" dialog: Open the file in basic Notepad, go to "File" -> "Save As...", and look at the "Encoding" dropdown near the Save button. It often detects common encodings like UTF-16.

Using the chardet library: Install pip install chardet, then use it to guess the encoding.

import chardet

filename = 'my_unknown_encoding_file.txt'
try:
    with open(filename, 'rb') as f: # Read bytes
        raw_data = f.read()
        result = chardet.detect(raw_data)
        encoding_guess = result['encoding']
        confidence = result['confidence']
        print(f"Chardet guess: {encoding_guess} (Confidence: {confidence:.2f})")
        # Now try opening with the guessed encoding
        # with open(filename, 'r', encoding=encoding_guess) as f_reopen: ...
except FileNotFoundError:
    print(f"Error: File '{filename}' not found.")
except Exception as e:
    print(f"Error during chardet: {e}")

chardet provides a probability but isn't guaranteed to be perfect.

Conclusion

The UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0... specifically indicates that your byte data likely starts with a UTF-16 Little Endian Byte Order Mark (BOM), but you are attempting to decode it as UTF-8.

The recommended solutions are:

Specify the correct encoding: Use encoding='utf-16' when calling open(), bytes.decode(), or pd.read_csv(). Python's 'utf-16' codec automatically handles both Little Endian and Big Endian BOMs.
Read in Binary Mode ('rb'): If you only need the raw bytes and not the decoded text.

Avoid using errors='ignore' or errors='replace' with the 'utf-8' codec for this particular error, as it will likely lead to data corruption starting from the very beginning of the file due to the misinterpreted BOM. Always prioritize identifying and using the correct encoding (utf-16 in this case).

How to Resolve Python Error "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff..."

Understanding the Error: UTF-8 vs. Byte Order Marks (BOM)​

The Cause: Decoding UTF-16/UTF-32 BOM as UTF-8​

Scenario: Decoding Byte Objects​

Scenario: Reading Files (open(), pandas.read_csv())​

Solution 1: Specify the Correct Encoding (utf-16, utf-32)​

For bytes.decode()​

For open()​

For pandas.read_csv()​

Solution 2: Use Specific BOM-Aware Encodings (utf-8-sig, utf-16, utf-32)​

Solution 3: Read File in Binary Mode ('rb')​

Solution 4: Handle Decoding Errors (errors parameter - Use Cautiously)​

Debugging: Identifying the Encoding​

Conclusion​

Table of Contents