Any time that a row ID (oddly placed in column 8, i.e. row[7]) is repeated after the first instance, I want to write those rows into a second file. The code I'm using is extremely slow -- it's a 40-column CSV with about a million rows. This is what I have:
def in_out_gorbsplit(inf, outf1, outf2):
outf1 = csv.writer(open(outf1, 'wb'), delimiter=',', lineterminator='\n')
outf2 = csv.writer(open(outf2, 'wb'), delimiter=',', lineterminator='\n')
inf1 = csv.reader(open(inf, 'rbU'), delimiter=',')
inf1.next()
checklist = []
for row in inf1:
id_num = str(row[7])
if id_num not in checklist:
outf1.writerow(row)
checklist.append(id_num)
else:
outf2.writerow(row)
