How to parse javascript results with Python

Question

I'm having trouble with my Python Script. All i want to do is to parse a div element with an id value:value and to store all the changed values. The value of this element is generating by javascript. This means that the value of the element is depending on user's input. To be more specific the html element looks like that

<div id="value">...Here the frequently changed value generated by javascript...</div>

My python script is the following:

from bs4 import BeautifulSoup
import urllib
x=urllib.urlopen("http://example.com")
s = x.read()
soup = BeautifulSoup(s)

m = soup.find("div",{"id":"value"})
val = m.text
print val

The result is None but on the webpage the changes are obvious! Please help me to figure it out.

Answer 1

If the value is generated by javascript - the easiest solution would be to make use of a real browser to crawl the web page. This is where selenium would help. Here's a simple example:

from selenium import webdriver

browser = webdriver.Firefox()
browser.get('http://example.com')

element = browser.find_element_by_id('value')
print element.text