current community

your communities

more stack exchange communities

Stack Exchange
tour help
careers 2.0
Take the 2-minute tour ×
Stack Overflow is a question and answer site for professional and enthusiast programmers. It's 100% free, no registration required.
up vote 0 down vote favorite

This works for me:


import xml.etree.ElementTree as ET
from urllib2 import urlopen

url = 'http://example.com'
# this url points to a `xml` page
tree = ET.parse(urlopen(url))

However, when I switch to requests, something was wrong:


import requests
import xml.etree.ElementTree as ET
url = 'http://example.com'
# this url points to a `xml` page
tree = ET.parse(requests.get(url))

The trackback error is showed below:


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in ()
----> 1 tree = ET.parse(requests.get(url, proxies={'http': '192.168.235.36:7788'}))

/usr/lib/python2.7/xml/etree/ElementTree.py in parse(source, parser)
   1180 def parse(source, parser=None):
   1181     tree = ElementTree()
-> 1182     tree.parse(source, parser)
   1183     return tree
   1184 

/usr/lib/python2.7/xml/etree/ElementTree.py in parse(self, source, parser)
    645         close_source = False
    646         if not hasattr(source, "read"):
--> 647             source = open(source, "rb")
    648             close_source = True
    649         try:

TypeError: coercing to Unicode: need string or buffer, Response found

So, my question is: wha is wrong with requests in my situation and how can I make it work ET with requests?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

You are passing the requests respones object to ElementTree; you want to pass in the raw file object instead:

r = requests.get(url, stream=True)
ET.parse(r.raw)

.raw returns the 'file-like' socket object, from which ElementTree.parse() will read, just like it'll read from the urllib2 response (which is itself a file-like object).

Concrete example:

>>> r = requests.get('http://www.enetpulse.com/wp-content/uploads/sample_xml_feed_enetpulse_soccer.xml', stream=True)
>>> tree = ET.parse(r.raw)
>>> tree
<xml.etree.ElementTree.ElementTree object at 0x109dadc50>
>>> tree.getroot().tag
'spocosy'

If you have a compressed URL, the raw socket (like urllib2) returns the compressed data undecoded; in that case you can use the ET.fromstring() method on the binary response content:

r = requests.get(url)
ET.fromstring(r.content)
share|improve this answer
    
I have tries this before, but not worked. I got this back: ParseError: no element found: line 1, column 0 –  holys Jun 5 '13 at 7:22
    
My apologies, the current API requires that you use stream=True for the raw reads to work properly, otherwise the data is downloaded early. Try again with the updated answer. –  Martijn Pieters Jun 5 '13 at 7:27
    
This doesn't work as is, as requests has already read data coming from the socket at the end of the first line. Passing stream=True as argument to the request is mandatory –  doukremt Jun 5 '13 at 7:28
    
@doukremt: that is what I am saying in my comments. :-) –  Martijn Pieters Jun 5 '13 at 7:30
    
@MartijnPieters It works! You have always been very helpful to me. Thank you so much:) –  holys Jun 5 '13 at 7:37
up vote 0 down vote

You're not feeding ElementTree the response text, but the requests Response object itself, which is why you get the type error: need string or buffer, Response found. Do this instead:

r = requests.get(url)
tree = ET.fromstring(r.text)
share|improve this answer
1  
This won't work for two reasons: r.text is the Unicode (decoded) result, you should always parse XML as un-decoded data, and ET.parse() wants a filename or file-like object. ET.parse() will see the r.text result as a filename and pass it to open(). –  Martijn Pieters Jun 5 '13 at 7:17
    
I know, that's why I edited my answer :-) –  doukremt Jun 5 '13 at 13:05
    
You are still using r.text instead of r.content though.. –  Martijn Pieters Jun 5 '13 at 13:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.