Retrying a web request in Python

Instead of using while to count a specific number of times and manually handle the number, you should use for _ in range(num_of_retries). It works the same except that you don't need to manually change num_of_retries. And this way num_of_retries wont be modified from the initial value, in case you need to access it again for any reason.

There is one small change to make this work. You need to use the for ... else syntax to raise if the loop is not broken. An else block after a for block will be executed if the for loop never reaches a break statement. This fits your case perfectly as break is called on success, but if there's never success then it will run through all the iterations and reach the else block where you can raise:

for _ in range(num_of_retries):
    try:        
        request_id = str(uuid.uuid4())
        params = [('requestId', request_id)]
        url = 'service.com' 
        url = url + urllib.urlencode(params)
        request = urllib2.Request(url, data=html)
        request.add_header('Accept', 'application/pdf')
        request.add_header('Content-Type', 'text/html')
        handle = urllib2.urlopen(request)
        pdf_blob = handle.read() 
        break
    except Exception:
        typ, val, tb = sys.exc_info()
        logger.error(traceback.format_exception(typ, val, tb))
        time.sleep(time_interval)
else:
    raise

Without any explicit argument, raise will just use the last exception that has been raised and handled. See the SO answer here about it. Basically when you try except an exception, its details are being retained by Python and are passed to raise in the absence of any explicit exception passed to raise.

You should try to reduce the amount of code protected by the try .. except construct. To me, the only line that should be protected as such is urllib2.urlopen(request), the rest being before the try or in an else clause.

You should also specialize a bit more your exception handling: trying again and again to reach an invalid url is a bad idea. But except Exception will permit that, since it catches ValueErrors raised by urllib2.urlopen when the url entered is invalid or blank. The only exceptions you should care about in your case are urllib2.URLError (and possibly urllib2.HTTPError its subclass). You will thus fail sooner if the cause of failure is not network-related.

The body of the loop would look like:

request_id = str(uuid.uuid4())
params = [('requestId', request_id)]
url = 'service.com' 
url = url + urllib.urlencode(params)
request = urllib2.Request(url, data=html)
request.add_header('Accept', 'application/pdf')
request.add_header('Content-Type', 'text/html')

try:
    handle = urllib2.urlopen(request)
except urllib2.URLError:
    typ, val, tb = sys.exc_info()
    logger.error(traceback.format_exception(typ, val, tb))
    num_of_retries -= 1
    # If there aren't any retries - propogate
    if not num_of_retries: 
        raise
    time.sleep(time_interval)
else:
    pdf_blob = handle.read()
    break

Note that I also moved the call to time.sleep(time_interval) to avoid waiting before re-raising the last exception.

The code would be clearer if you used a function decorator for the retry logic. One such decorator is retrying, but there are other similar libraries, and it wouldn't be hard to write one yourself.

@retry(stop_max_attempt_number=2, wait_fixed=2000)
def getblob(html):
    request_id = str(uuid.uuid4())
    …