current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 3 down vote favorite
1

I am running scrapy on ubuntu server 13. And while it is crawling the network becomes very slow and it becomes difficult to even ssh into the system.

I think I can solve the problem by having one more network interface card connected to a different router that scrapy doesn't use. Is it possible? How?

share|improve this question

1 Answer 1

up vote 2 down vote

Generally, yes. A few notes that might be useful:

  1. If you have more than one interface, you can set up static routes that specify which interface should be used to get to a specific destination. CentOS networking docs have a decent guide. That lets you specify what interface the system uses to get to a specific outbound address.

  2. It sounds more like you want to be able to get in on an alternate interface. Most daemons, like ssh, let you configure which interface they listen on. For ssh, see the sshd_config ListenAddress configuration.

  3. Neither of those were you actual question, of course, which was if it was possible to configure an application to use a specific interface. That's also possible, if you use the iptables --uid-owner option to specify a route that's only used by apps from that user. There's a useful discussion of this scenario on linuxquestions.org .

  4. Finally, are you sure you're network bottlenecked? If you can't even get a prompt, I'd check to make sure scrapy isn't consuming too much CPU, and consider nice-ing it etc. if so.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.