current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

I'm stuck with a python problem, look.

I have to scrap a page that has JS functions, but that's not the real problem, the real problem is that the information I need is provided by the function.. So I need to run the function to make sure it'll build the HTML code that I need, and then work on it to get what I want..

Just to make sure i'm clear, the JS function build the HTML code, but when I scrap it, it doesn't get HTML buid, it just return the JS function...

I am using mechanize and beautifulSoup for the scraping process.. does anyone know what do I have to do to emulate the JS function to get the HTML code that I need?

Thanks in advance.

share|improve this question
    
I would figure out how the JS function builds the HTML. Mechanize doesn't execute JS, so it won't be too helpful. –  Blender Dec 12 '12 at 2:13

1 Answer 1

up vote 0 down vote

You need a scrapping framework that supports javascript. Selenium is one of them and I got good results using along with BeautifulSoup.

You may want to check PyVirtualDisplay if you are going to use Selenium with Mozilla Firefox.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.