I'm having a bit of trouble. I want to run a shell command within Python in a specific directory. Based on the code I found on the internet I need the following inclusions:
import os
import subprocess
import shlex
And then the code itself is below
os.chdir('/etc/test/')
cmd = 'scrapy crawl test'
subprocess.call(shlex.split(cmd))
As it looks like, I am trying to run the command "scrapy crawl test" within the /etc/test/ directory. When I run this manually with terminal it seems to work fine however when I run it with this python code it gives me an error:
INFO Exception occured while scraping: [Errno 2] No such file or directory
Is anyone able to tell me if my code is incorrect, or if I am going about this the wrong way perhaps.
cmd = ['scrapy', 'crawl', 'test']thensubprocess.call(cmd)is simpler, and probably harder to get wrong; no need to useshlexhere. But that' won't affect the problem you're trying to solve. – abarnert Aug 19 '13 at 20:37scrapyis a Python library. Have you gone through the Getting Started and Tutorial stuff? Everything you want to do from within Python, you can do in Python. Everything you want to do at the shell or in a cronjob or whatever, you can do with the command line tool. If you're trying to run the command-line tool from within Python, you're probably making a mistake earlier on in the process… but it's hard to be sure what that is without more information on what you're trying to do. – abarnert Aug 19 '13 at 20:54scrapy, not from your code. That could mean there's a bug in your spider, or your directory layout isn't what you expect, or a million other things. Have you tried usingscrapy shellto debug it, as described in the tutorial? – abarnert Aug 19 '13 at 20:55