up vote 4 down vote favorite

I have a Python script that uses Selenium WebDriver (with PyVirtualDisplay as the display) to log into Flickr.

When I run it as myself on my Debian server, it works fine. (I’m a sudoer, but I don’t use sudo when running the script.)

When I run it as the user www-data (which is what it’ll be running as eventually, because I want to trigger it from a Django website), I get two problems, one small, one big:

  1. (Small): the webdriver.Firefox() call takes 30–45 seconds to return, compared to 2 seconds when run as myself
  2. (Big): the script fails to log into Flickr. In order to log in, I find the username and password fields on the Flickr signin page (http://www.flickr.com/signin/), and use element.send_keys() to enter the username and password. Although Selenium seems to find the elements (i.e. no NoSuchElementException is thrown), the values do not get entered in the fields when the script is run as www-data (according to the screenshots I take using browser.save_screenshot), unlike when the script is run as myself.

Why does send_keys() not work when the script is run as www-data? (And is it related to the browser taking much longer to start?)

share|improve this question
92% accept rate
feedback

This question has an open bounty worth +500 reputation from Paul D. Waite ending in 5 hours.

This question has not received enough attention.

4 Answers

up vote 2 down vote

Maybe you have something different in your environment.

Try copy by example your ~/.bashrc in /home/www-data

If it's not sufficient, run this command both as your current user & as www-data:

strace -tt -f -s 1000 -o /tmp/trace ./script.py

And paste it (filter out your logins/passwords) somewhere.

We will see what's happens.

share|improve this answer
Excellent, sounds like good troubleshooting steps. I’m not at my machine right now, but I’ll do that when I am. – Paul D. Waite Oct 7 at 17:39
Right. Copying my ~/.bashrc file into /home/www-data (which I created first, as it didn’t exist) didn’t seem to have any effect: the script behaved as before when run as me and as www-data. I’ve installed strace on the server and run the command you suggested, but for some reason /tmp/trace isn’t getting created when I run the script as either user. (I get lots of output on my screen, but I don’t get a file.) Under both users, strace doesn’t seem to return once the script’s finished though — I have to press <kbd>Ctrl</kbd><kbd>C</kbd> to get a shell prompt again. – Paul D. Waite Oct 7 at 22:02
1  
See my edited POST, there was an error in the strace command line. – sputnick Oct 7 at 22:11
Ah yes, I see it now. Just having a look at the stack traces, as when saved to text files they ended up around 48 MB and 25 MB respectively. Should be able to get them up onto the web in some form tomorrow. – Paul D. Waite Oct 7 at 23:06
1  
Maybe the first 100000 lines is sufficient for the moment. You can filter that out with head -n 100000 FILE > NEW_FILE – sputnick Oct 7 at 23:11
show 4 more comments
feedback
up vote 1 down vote

Sometimes, Firefox performs some nasty plugin compatibility check during startup. As each user can have a different set of browser plugins, this could be responsible for the difference in startup times. You could try to sync your Firefox profiles between users.

Then, are you sure that Firefox as user www-data has proper network/internet access? Can you confirm that the Flickr site loads properly via SeleniumHQ? "The script fails to log into Flickr" is too unprecise. Some more details about why it fails might reveal the problem instantaneously.

Edit: Sorry, I just understood that there shouldn't be a difference in profiles, because Selenium creates one. Nevertheless, my second point might be useful, so I won't delete this answer.

share|improve this answer
That’s a good point re confirming what Selenium’s doing — I’ll amend my test script to pop out some screen shots so that I can provide more detail. Hopefully plugins should’t be a problem — I installed Firefox on the Debian server using aptitude (via squeeze-backports) and otherwise left it alone, so if I understand correctly it shouldn’t have any plugins installed. – Paul D. Waite Oct 10 at 7:59
Alrighty. Having written out some screenshots whilst Selenium is controlling Firefox, it looks like the username and password are not entered into the fields on the web page when the script is run as www-data. I take one screenshot before I use field.send_keys to enter the username and password, and one after. When I run the script as me, the second screenshot has the values entered in the field. When I run the script as www-data, it does not. – Paul D. Waite 2 days ago
I wonder if it’s anything to do with this bug: code.google.com/p/selenium/issues/detail?id=2465 – Paul D. Waite 2 days ago
feedback
up vote 1 down vote

Some more things to ponder about:

  • Could you spawn firefox manually from www-data account once and make sure that Firefox is not updating itself before every execution of the script? I once faced this problem with Selenium RC on Windows and had to let the update finish before starting the script with the updated binary.

  • As a workaround, I guess you could you try running the script as www-data user but connecting remotely to a webdriver server running in your login (aka "grid" mode). Would that work for you?

share|improve this answer
“Could you spawn firefox manually from www-data account once and make sure that Firefox is not updating itself before every execution of the script?” — Possibly. Thing is, I don’t have the password of the www-data account, so I don’t think I can log into the GUI as www-data. – Paul D. Waite Oct 10 at 11:24
Hmm. Perhaps you could give an explicit location to the latest version of the Firefox binary instead of letting WebDriver use the system default. Look for instructions here: stackoverflow.com/a/10566625/1537330 – Siddharth Toshniwal Oct 10 at 11:42
1  
@PaulD.Waite: Use gksu -u www-data firefox (using your own password). – Mechanical snail Oct 10 at 20:00
1  
It‘s actually a bit tricky to launch Firefox from the www-data account, even using gksu. www-data’s home folder is /var/www, but it doesn’t have write permissions there. This means it can’t create a Firefox profile when Firefox launches, and thus Firefox doesn’t launch. However, I have launched Firefox from Selenium running as me, and when Selenium launches Firefox, it creates a Firefox profile in /tmp/ (which Firefox presumably uses during its Selenium session). – Paul D. Waite 2 days ago
1  
@PaulD.Waite: There's a command-line switch to specify a profile directory, so specify somewhere in /tmp/ or /run/shm/ to bypass that issue. – Mechanical snail yesterday
show 2 more comments
feedback
up vote 0 down vote

I would suggest getting the latest chrome from google and trying input.send_keys() in that browser instead.

Sometimes some features of webdriver get broken with new releases.. If you are bent on testing with firefox, you might have better luck with an older/newer version of selenium webdriver.

I remember having a similar issue regarding send_keys() on a mac.. My issue was that send_keys() did not work in certain modal windows after I updated selenium webdriver.. I fixed it by reverting to an older webdriver that I knew to work. However, I was using Ruby and not Python to drive webdriver.

sometimes, there might also be a problem with getting the correct ENV variables in your shell if you use it as a different user. I would suggest trying to troubleshoot and see if all the shell ENV variables are set properly under www-data.

share|improve this answer
Chrome: worth a try, I’ll give it a go. On the environment variables, any idea what particularly I might be looking for? I’ve compared my env variables with www-data’s, and there doesn’t seem to be anyting obvious or Selenium/Python-related missing for www-data. – Paul D. Waite yesterday
there shouldn't be any environment variables required in particular for firefox.. but chrome does require some additional configuration (including one environment variable).. hope this helps... – sambehera yesterday
feedback

Your Answer

 
or
required, but never shown
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.