Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 5 down vote favorite

I'd like to distinguishing None and empty strings when going back and forth between python data structure and csv representation using python's csv module.

My issue is that when I run 

import csv, cStringIO

data = [['NULL/None value',None],
        ['empty string','']]

f = cStringIO.StringIO()
csv.writer(f).writerows(data)

f = cStringIO.StringIO(f.getvalue())
data2 = [e for e in csv.reader(f)]

print "input : ", data
print "ouput : ", data2

I get the following output :

input :  [['NULL/None value', None], ['empty string', '']]
ouput :  [['NULL/None value', ''], ['empty string', '']]

Of course, I could play with data and data2 to distinguish None and empty string with things like

data =[d if d!=None else 'None' for d in data]
data2 =[d if d!='None' else None for d in data2]

But that would partly defeat the interest of the csv module (quick deserialization/serialization implemented in c, specially when you are dealing with large lists).

Is there a csv.Dialect or parameters to csv.writer, csv.reader that would enable to distinguish between '' and None in this use-case ?

If not, would there be an interest in implementing a patch to csv writer to enable this kind of back and forth ? (Possibly a Dialect.None_translate_to parameter defaulting to '' to ensure backward compatibility)

share|improve this question

5 Answers

up vote 2 down vote accepted

The documentation suggests that what you want is not possible:

To make it as easy as possible to interface with modules which implement the DB API, the value None is written as the empty string.

This is in the documentation for the writer class, suggesting it is true for all dialects and is an intrinsic limitation of the csv module.

I for one would support changing this (along with various other limitations of the csv module), but it may be that people would want to offload this sort of work into a different library, and keep the CSV module simple (or at least as simple as it is).

If you need more powerful file-reading capabilities, you might want to look at the CSV reading functions in numpy, scipy, and pandas, which as I recall have more options.

share|improve this answer
Yep confirmed : looking at csv_writerow in Modules/_csv.c ( if (field == Py_None) ... ). There is no way to distinguish between a '' and a None. Really a shame, given the Dialect abstraction you would have hoped for a bit more flexibility. You mention other limitations of the csv module, would you mind elaborating (If there are other issues I really should start looking at other csv-reading writing) ? – user1509316 Jul 8 '12 at 0:48
One limitation I've found annoying sometimes is that delimiters must be a single character. So you can't parse a file where columns are separated by, say, two tabs. Like the None thing you came across, this is fairly easy to work around, but still annoying. – BrenBarn Jul 8 '12 at 2:21
Another is the hard coded ascii limitation within the module. – Spencer Rathbun Jan 18 at 14:02
up vote 3 down vote

You could at least partially side-step what the csv module does by creating your own version of None:

class NULL(object):
    def __repr__(self):    # what csv.writer class uses to write value
        return '<NULL>'

NULL = NULL()  # singleton

import csv, cStringIO
data = [['NULL/None value', NULL], ['empty string', '']]
f = cStringIO.StringIO()
csv.writer(f).writerows(data)
f = cStringIO.StringIO(f.getvalue())
data2 = [e for e in csv.reader(f)]
print "input : ", data
print "ouput : ", data2

Output:

input :  [['NULL/None value', <NULL>], ['empty string', '']]
ouput :  [['NULL/None value', '<NULL>'], ['empty string', '']]

The will preserve enough information for you to be able to differentiate between NULL/None and empty-string values.

Alternatively
You could take things a little further and create a pair of csv.reader and csv.writer proxy classes, which ought to remain reasonably fast since the majority of the processing is still done by the underlying built-in csv module classes:

import csv

_NONE = '<None>'  # unique string value to represent None

class csvWriter(object):
    def __init__(self, csvfile, *args, **kwrags):
        self.csv_writer = csv.writer(csvfile, *args, **kwrags)
    def writerow(self, row):
        self.csv_writer.writerow([_NONE if val is None else val for val in row])
    def writerows(self, rows):
        map(self.writerow, rows)

class csvReader(object):
    def __init__(self, csvfile, *args, **kwrags):
        self.csv_reader = csv.reader(csvfile, *args, **kwrags)
    def __iter__(self):
        return self
    def next(self):
        return [None if val == _NONE else val for val in self.csv_reader.next()]

if __name__ == '__main__':
    import cStringIO as StringIO
    data = [['Null/None value', None], ['empty string', '']]
    f = StringIO.StringIO()
    csvWriter(f).writerows(data)
    f = StringIO.StringIO(f.getvalue())
    data2 = [e for e in csvReader(f)]
    print "input : ", data
    print "ouput : ", data2

Output:

input :  [['Null/None value', None], ['empty string', '']]
ouput :  [['Null/None value', None], ['empty string', '']]
share|improve this answer
up vote 0 down vote

As others have pointed out you can't really do this via csv.Dialect or parameters to csv.writer and/or csv.reader. However as I said in one comment, you implement it by effectively subclassing the latter two (you apparently can't really do because they're built-in). What the "subclasses" do on writing is simply intercept None values and change them into a unique string and reverse the process when reading them back in. Here's a fully worked-out example:

import csv, cStringIO
NULL = '<NULL>'  # something unlikely to ever appear as a regular value in your csv files

class MyCsvWriter(object):
    def __init__(self, *args, **kwrds):
        self.csv_writer = csv.writer(*args, **kwrds)

    def __getattr__(self, name):
        return getattr(self.csv_writer, name)

    def writerow(self, row):
        self.csv_writer.writerow([item if item is not None else NULL
                                      for item in row])
    def writerows(self, rows):
        for row in rows:
            self.writerow(row)

class MyCsvReader(object):
    def __init__(self, *args, **kwrds):
        self.csv_reader = csv.reader(*args, **kwrds)

    def __getattr__(self, name):
        return getattr(self.csv_reader, name)

    def __iter__(self):
        rows = iter(self.csv_reader)
        for row in rows:
            yield [item if item != NULL else None for item in row]

data = [['NULL/None value', None],
        ['empty string', '']]

f = cStringIO.StringIO()
MyCsvWriter(f).writerows(data)  # instead of csv.writer(f).writerows(data)

f = cStringIO.StringIO(f.getvalue())
data2 = [e for e in MyCsvReader(f)]  # instead of [e for e in csv.reader(f)]

print "input : ", data
print "ouput : ", data2

Output:

input :  [['NULL/None value', None], ['empty string', '']]
ouput :  [['NULL/None value', None], ['empty string', '']]

It's a tad verbose and probably slows the reading & writing of csv file a bit (since they're written in C/C++) but that may make little difference since the process is likely low-level I/O bound anyway.

share|improve this answer
up vote 1 down vote

As you have control over both the consumer and the creator of the serialised data, consider using a format that does support that distinction.

Example:

>>> import json
>>> json.dumps(['foo', '', None, 666])
'["foo", "", null, 666]'
>>>
share|improve this answer
up vote 1 down vote

I don't think it would be possible to do what you want with a mere dialect, but you could write your own csv.reader/write subclass. On the other hand, I still think that is overkill for this use case. Even if you want to catch more than just None, you probably just want str():

>>> data = [['NULL/None value',None],['empty string','']]
>>> i = cStringIO.StringIO()
>>> csv.writer(i).writerows(map(str,row) for row in data)
>>> print i.getvalue()
NULL/None value,None
empty string,
share|improve this answer
Actually, you can't subclass csv.reader and csv.writer. – martineau Apr 5 at 3:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.