Problems scraping pages with JavaScript function in python

Question

I'm stuck with a python problem, look.

I have to scrap a page that has JS functions, but that's not the real problem, the real problem is that the information I need is provided by the function.. So I need to run the function to make sure it'll build the HTML code that I need, and then work on it to get what I want..

Just to make sure i'm clear, the JS function build the HTML code, but when I scrap it, it doesn't get HTML buid, it just return the JS function...

I am using mechanize and beautifulSoup for the scraping process.. does anyone know what do I have to do to emulate the JS function to get the HTML code that I need?

Thanks in advance.

I would figure out how the JS function builds the HTML. Mechanize doesn't execute JS, so it won't be too helpful. — Blender
– Blender, Commented Dec 12, 2012 at 2:13

Paulo Scardine · Accepted Answer · 2012-12-12 02:15:56Z

0

You need a scrapping framework that supports javascript. Selenium is one of them and I got good results using along with BeautifulSoup.

You may want to check PyVirtualDisplay if you are going to use Selenium with Mozilla Firefox.

answered Dec 12, 2012 at 2:15

Paulo Scardine

78.1k12 gold badges133 silver badges153 bronze badges

Add a comment |

Collectives™ on Stack Overflow

Problems scraping pages with JavaScript function in python

1 Answer 1

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Related