current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Sign up ×
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other. Join them; it only takes a minute:
up vote 0 down vote favorite

I am using Selenium and python and I have a website that has Javascript. I define a driver:

chromedriver = "/foldername/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)

I navigate to the page:

url = '...html'
driver.get(url)

Then I get the page source:

print(driver.page_source)

The issue is that the stuff I actually want to scrape is not present anywhere. It is also not present when I view the page source in Chrome. The only way I know how to view it is by using Firefox - Firebug.

I have also tried things like:

u = driver.find_elements_by_id("wrapper")
text = u[0].get_attribute("innerHTML")

But the stuff I want is also not in text.

Anything else I can try?

share|improve this question
    
It turns out that iframes are in play. These posts were helpful: stackoverflow.com/questions/21445966/… and stackoverflow.com/questions/7534622/… – bill999 Aug 18 at 12:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.