current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other.

Join them; it only takes a minute:

Sign up
Join the Stack Overflow community to:
  1. Ask programming questions
  2. Answer and help your peers
  3. Get recognized for your expertise
up vote 1 down vote favorite
1

I have to import a compressed sql dump into a PostgreSQL database. The dump gets created by:

pg_dump -F c

So I get a compressed file, which I can't just parse line by line and then use psycopg2 to upload it into my database.

The compressed dumps I'm dealing with are quite big (up to 10GB). What would be an efficient way to do import them?

share|improve this question
    
Is there a reason you can't re-import using the command line? Doing it through Python is going to be inefficient. – syrion Feb 10 '11 at 21:48
    
If you have to "control" it via Python, you can always execute the necessary command line command from within Python. – payne Feb 10 '11 at 21:50
    
Why not just use postgres's own tools, such as pg_restore? – Keith Feb 10 '11 at 22:34
    
Do you know what compression algorithm it uses? – Keith Feb 10 '11 at 22:35
    
@Keith: Its own, as its says in the man. File is 'src/bin/pg_dump/pg_backup_archiver.c' in the source tree. It uses zlib but it has its own structure. Its not just zipped. Best bet would probably be interfacing with it in C and using the pg headers if you really have to do it this way. – nate c Feb 10 '11 at 23:26
up vote 1 down vote accepted

You basically can't do that unless you reimplement pg_restore in your Python project. Consider instead calling pg_restore from your Python program.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.