current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.
up vote 5 down vote favorite

I'm writing a program to process log files. Currently, I have a script that will extract the desired information from each log and put it into a list - I have about nine different lists all containing different information.

Currently, I'm loading the file, extracting the desired infromation, and storing it in a list, finally I write the lists to a CSV.

The program roughly follows this outline.

#load data

#create lists
list1 = []
list2 = []
list3 = []
list4 = []
list5 = []

#extract information to populate lists.

#write CSV's
with open('output1.csv', 'wb') as csvfile:
        writer = csv.writer(out)
    for line in list1:
        writer.writerow(line)

with open('output2.csv', 'wb') as csvfile:
    writer = csv.writer(out)
    for line in list2:
        writer.writerow(line)

with open('output3.csv', 'wb') as csvfile:
    writer = csv.writer(out)
    for line in list3:
        writer.writerow(line)

with open('output4.csv', 'wb') as csvfile:
    writer = csv.writer(out)
    for line in list4:
        writer.writerow(line)

with open('output5.csv', 'wb') as csvfile:
    writer = csv.writer(out)
    for line in list5:
        writer.writerow(line)

with open('output6.csv', 'wb') as csvfile:
    writer = csv.writer(out)
    for line in list6:
        writer.writerow(line)

I'm wondering if there is a better/faster way to do this. Considering these lists can get pretty large. I feel like it might be quicker/faster to load the data and immediately write is to a file as it gets processed. Thoughts?

share|improve this question

1 Answer 1

up vote 5 down vote

Are you more worried about memory, speed or maintainability? if the lists are long enough that you worry about them filling up your available memory, write them out one at a time. If not, process them in bulk.

in either case, a list-of-lists is better than maintaining 6 copies of identical code. You could use a dictionary to store the different lists with their output filenames and avoid the duplication:

 lots_o_data = {
     'list1': [.....],
     'list2': [.....],
     'list3': [.....]
 }

 for eachfilename, eachlist in lots_o_data.items():
     with open(eachfilename, 'wb') as csvfile:
          writer = csv.writer(csvfile)
          for line in eachlist: 
              writer.writerow(line)

If the data is interleaved - stuff from different ouput lists comes up intermingled - you have to collect it all before writing. If it's sequential you can do it read A - write A - read B - write B style. Ordinarily the total time won't change much - reading and writing the data will be way slower than whatever you do in the processing.

One thing you don't want to do is open the same source file multiple times if you don't have to - that will add extra time. Only do that if you're worried about memory.

share|improve this answer
    
Yeah, I was just thinking from a speed point of view. I'll have enough memory to process in bulk, but was wondering if reading/processing/writing one log at a time offered any substantial speed benefit. I'll add the lists to a dictionary just to clean up the code. Thanks –  cloud36 Feb 11 at 23:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.