using numpy to read a file into python

Question

So this might be a bit of a noob question as I don't have a whole lot of python experience. I have a .dat file that I've converted into .csv that I'm trying to read into python. This should be very simple as numpy has a built in function for this. My code is:

import numpy as np
import csv

d = np.loadtxt('scl1.csv', delimiter="\t")

The error I get says that it could not convert string to float. How do I fix this? For reference the data file has 6 columns, where each entry is a number (ie 7.33390715197523163E-002) and they are delimited by tabs. I thought that the problem could be with the E in some of the numbers but I checked and numpy can read that as a float. Thanks very much for any help.

Edited to include the exact error message if its helpful:

Traceback (most recent call last):
  File "/Applications/Python 3.4/Markov Chain/Monte Carlo.py", line 10, in <module>
    d = np.loadtxt('scl2.dat', delimiter="\t")
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/numpy/lib/npyio.py", line 848, in loadtxt
    items = [conv(val) for (conv, val) in zip(converters, vals)]
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/numpy/lib/npyio.py", line 848, in <listcomp>
    items = [conv(val) for (conv, val) in zip(converters, vals)]
ValueError: could not convert string to float: b'   7.33390715197523163E-002  7.68126324487871659E-002  5.46056179421958582E-002  -22.791933511352461        8.7787362443778942       -3.6715272730208461     '

Answer 1

loadtxt() assumes the file consists only of floats, but your .csv file (likely) has column headings that are strings--hence the conversion error.

If your csv file contains only numbers and no quotes, ignore the column names by skipping the first row of the file.

np.loadtxt('scl1.csv', delimiter='\t', skiprows=1)

Or to grab the headers as well:

DELIM = '\t'
fh = open('scl1.csv')
headers = fh.readline().strip().split(DELIM)
data = np.loadtxt(fh, delimiter=DELIM)

Otherwise, you could use the csv module and cast the types yourself.

read_csv = csv.reader(open('scl1.csv'))
for fields in read_csv:
     pass # process fields list, which are all strings, and includes the col names

Also, random tip: don't name things with both an 'l' and a '1' if you can help it!