Tell me more ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 1 down vote favorite

I'm trying to use the Python mechanize module to retrieve data through this form: http://archive.stsci.edu/kepler/data_search/search.php?form=fuf

The thing I'm having trouble with is "Output Columns" area in the lower left, which uses javascript to specify the output format. I'd like to be able to select a specific list of items, which would normally be selected with the surrounding javascript buttons.

I'm new to mechanize and haven't looked at javascript in ages. I've been looking through the responsible javascript but am unsure of what to do. Any tips?

More specifically: Is there a way to use mechanize to modify the elements in a list?

share|improve this question
add comment

1 Answer

up vote 2 down vote

I recommend making the POST call directly to that form, rather than interacting with the form somehow through mechanize. Here is how I would do this with mechanize:

import mechanize
from urllib import urlencode
opener = mechanize.build_opener()
data = {"resolver":"NED","radius":"0.02","equinox":"J2000","ktc_target_type[]":"LC","ktc_target_type[]":"SC","extra_column_name_1":"ktc_kepler_id","extra_column_value_1":"","extra_column_name_2":"ktc_kepler_id","extra_column_value_2":"","extra_column_name_3":"ktc_kepler_id","extra_column_value_3":"","extra_column_name_4":"ktc_kepler_id","extra_column_value_4":"","selectedColumnsCsv":"Mark,ktc_kepler_id,ktc_investigation_id,sci_data_set_name,ktc_target_type","selectedColumnsList[]":"ktc_target_type","availableColumns":"Mark","ordercolumn1":"ang_sep","ordercolumn2":"ktc_kepler_id","ordercolumn3":"","coordformat":"sex","outputformat":"HTML_Table","max_records":"1001","max_rpp":"100","action":"Search"}
d = urlencode(data)
url = 'http://archive.stsci.edu/kepler/data_search/search.php'
stuff = opener.open(url,d)
info_i_want = stuff.read()

Basically, put the contents of your search in a dictionary, make a POST call to the server, and read your response.

share|improve this answer
 
Thanks for the answer. I'd been thinking of doing it through POST, but hadn't had any knowledge of that so had been reluctant to try; your code works great, though. What about file upload, though? How do I do that through urlencode? –  StrangeQuirk Mar 18 '12 at 3:11
 
Also, I've figured out that with mechanize I can simply edit the contents of the hidden selectedColumnsCsv element to choose the output columns; I had assumed that it would be wiped clean by the javascript preprocess() function. In this case, is there any advantage to using POST over mechanize? –  StrangeQuirk Mar 18 '12 at 3:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.