Encoding error using Python

Thanks for Martijn Pieters. We figured out that the email had two different encode. I had to split this parts and treat individually.

Specifying the source code encoding worked for me. It's the code at the top of my example code below. This should be defined at the top of your python file.

#!/usr/bin/python
# -*- coding: iso-8859-15 -*-

value = """...instalação de eletrônicos...""".decode("iso-8859-15")
print value
# prints: ...instalação de eletrônicos...

import unicodedata
value = unicodedata.normalize('NFKD', value).encode('ascii','ignore')
print value
# prints: ...instalacao de eletronicos...

And now you can do str(value) without an exception as well.

See: http://docs.python.org/2/library/unicodedata.html

This seems to keep all accents:

#!/usr/bin/python
# -*- coding: iso-8859-15 -*-
import unicodedata
value = """...instalação de eletrônicos...""".decode("iso-8859-15")
value = unicodedata.normalize('NFKC', value).encode('utf-8')
print value
print str(value)

# prints (without exceptions/errors):
# ...instalação de eletrônicos...
# ...instalação de eletrônicos...

EDIT:

Do note that with the last version even though the outcome looks the same it doesn't return equal is True. In example:

#!/usr/bin/python
# -*- coding: iso-8859-15 -*-
import unicodedata
inValue = """...instalação de eletrônicos...""".decode("iso-8859-15")
normalizedValue = unicodedata.normalize('NFKC', inValue).encode('utf-8')

try:
    print inValue == normalizedValue
except UnicodeWarning:
    pass
# False

EDIT2:

This returns the same:

normalizedValue = unicode("""...instalação de eletrônicos...""".decode("iso-8859-15")).encode('utf-8')
print normalizedValue 
print str(normalizedValue )

# prints (without exceptions/errors):
# ...instalação de eletrônicos...
# ...instalação de eletrônicos...

Though I'm not sure this will actually be valid for a utf-8 encoded database. Probably not?

The header says it is using "ISO-8859-1" charset. So you need to decode the string with that encoding.

Try this:

data[0][1].decode('iso-8859-1')