Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

--Hi guys, --

I have about 4000 (1-50MB) files to sort.

I was thinking to have Python call the Linux sort command. And since I'm thinking this might be somewhat I/O bound, I would use the threading library.

So here's what I have but I when I run it and watch the system monitor I don't see 25 sort tasks pop up. It seems to be running one at a time? What am I doing wrong?

...
print "starting sort"
def sort_unique(file_path):
    """Run linux sort -ug on a file"""
    out = commands.getoutput('sort -ug -o "%s" "%s"' % (file_path, file_path))
    assert not out

pool = ThreadPool(25)
for fn in os.listdir(target_dir):
    fp = os.path.join(target_dir,fn)
    pool.add_task(sort_unique, fp)

pool.wait_completion()

Here's where ThreadPool comes from, perhaps that is broken?

share|improve this question

3 Answers

up vote 0 down vote

Normally people do this by spawning multiple processes. The multiprocessing module makes this easy to do.

On the other hand, Python is pretty good at sorting, so why not just read the file into a list of strings file.readlines() and then sort it in Python. You would have to write a key function to use with list.sort() to do the -g option, and you would also have to remove duplicates, i.e. -u option. The easiest way (and a fast way) to remove duplicates is to do list(set(UNsortedfile)) before you do the sort.

share|improve this answer
up vote 0 down vote

Actually this does seem to be working. I spoke too soon. I'm not sure if you guys want to delete this or what? Sorry about that.

share|improve this answer
up vote 1 down vote

You're doing everything correct.

There is something which is called GIL in python;
Global Interpreter Lock - which eventually cause python to execute only one thread at time.

Choose subprocess instead :), python is not multithreaded.

share|improve this answer
I know about the GIL, but I think the GIL should be released when the Linux command is running, right? – Greg Nov 3 '11 at 17:08
I wouldn't say so, but I would like to hear confirmation from someone else as well. – bua Nov 3 '11 at 17:11
I agree with bua, use subprocess. – mkj Nov 3 '11 at 20:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.