A Python app can make an asynchronous request to the URL Fetch service to fetch a URL in the background, while the application code does other things.

This API is provided by the google.appengine.api.urlfetch

1. Making Asynchronous Requests
2. The create_rpc() Function
3. The make_fetch_call() Function
4. The RPC object methods:
   • wait
   • check_success
   • get_result

Making Asynchronous Requests

An application can call the URL Fetch service to fetch URLs in the background while the application does other things. To do so, the application calls the create_rpc() function to create an RPC object that represents the asynchronous call. It then calls the make_fetch_call() function, passing the RPC object as an argument, to initiate the fetch. The app calls methods on the RPC object to wait for the call to complete and retrieve the results.

The make_fetch_call() function initiates the URL fetch then returns immediately, allowing the app to do other things while the service fetches the URL. When the app is ready for the results, it calls the get_result() method of the RPC object. If the service has not completed the fetch when the app calls get_result(), the method waits until the request is complete (or has reached the deadline, or an error occurs). The method returns the result object, or raises an exception if an error occurred during the fetch.

from google.appengine.api import urlfetch

rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, "http://www.google.com/")

# ... do other things ...

try:
    result = rpc.get_result()
    if result.status_code == 200:
        text = result.content
        # ...
except urlfetch.DownloadError:
    # Request timed out or failed.
    # ...

You can set a maximum amount of time the service will wait for the remote host to respond when you create the RPC object. The timer starts when make_fetch_call() is called.

The RPC object can be created with an optional callback function. This function is called when the application calls a method on the RPC object that causes it to wait for the request to complete. The callback does not occur in the background; the application must call a method (wait(), check_success() or get_result()) to invoke the callback function. You can assign the callback function to the RPC object after the object has been created, so the callback can refer to the RPC and call check_success() or get_result():

def handle_result(rpc):
    result = rpc.get_result()
    # ... Do something with result...

# Use a helper function to define the scope of the callback.
def create_callback(rpc):
    return lambda: handle_result(rpc)

rpcs = []
for url in urls:
    rpc = urlfetch.create_rpc()
    rpc.callback = create_callback(rpc)
    urlfetch.make_fetch_call(rpc, url)
    rpcs.append(rpc)

# ...

# Finish all RPCs, and let callbacks process the results.
for rpc in rpcs:
    rpc.wait()

Due to how RPC waiting is implemented, if you have multiple simultaneous asynchronous requests with callbacks, calling the wait() method of one RPC object may cause the callbacks of other RPC objects to trigger if they complete while the app is waiting. Calling wait() on an RPC whose callback has already been called will not trigger the callback again.

The synchronous fetch() function is equivalent to starting an asynchronous call, then immediately calling get_result().

An RPC object can only represent one URL fetch call in its lifetime. To make multiple asynchronous calls, you must create an RPC object for each call.

The development server does not make asynchronous URL fetch calls in the background. Instead, it performs the URL fetch synchronously when the RPC object's wait() method is called. Asynchronous URL fetch calls will have different performance characteristics on App Engine than on the development server. Such calls may complete out of order on App Engine, but will always complete in order on the development server.

The create_rpc() Function

The create_rpc() function returns an RPC object for making an asynchronous request.

create_rpc(deadline=None, callback=None)

Creates and returns an RPC object for making an asynchronous request to the URL Fetch service. To initiate the URL fetch, the app calls the make_fetch_call() function and passes it this object.

Arguments

deadline

The most amount of time to wait for a response from the remote host, as a number of seconds. If the remote host does not respond in this amount of time, a DownloadError is raised when the check_success method or the get_result method of the RPC object is called.

If the app attempts to access results before the remote host has responded and before the deadline has elapsed, the call will wait until either of these have occurred, and either return results or raise the exception. If the app request timer expires while the app is waiting, the call is canceled.

The maximum deadline is 60 seconds for HTTP requests and 10 minutes for task queue and cron job requests. If deadline is None, the deadline is set to 5 seconds.

callback

A Python function to be called when the service returns results successfully. The function is called without arguments.

The callback function is called when the wait() method of the RPC object is called, either directly by the application or implicitly with a call to check_success() or get_result(). The function does not get called in a background process or thread; it only gets called when one of these methods is called by the application.

The callback function is called even if the request fails or if the RPC deadline elapses.

The make_fetch_call() Function

The make_fetch_call() function initiates an asynchronous URL fetch.

make_fetch_call(rpc, url, payload=None, method=GET, headers={}, allow_truncated=False, follow_redirects=True, validate_certificate=False)

Initiates an asynchronous URL fetch, and associates the given RPC object with the fetch call. The method returns immediately, and the URL fetch occurs in the background. To retrieve the results of the fetch, the app calls the get_result() method of the RPC object.

Arguments

rpc

The RPC object returned by the create_rpc() function.

url

An http or https URL. If the URL is invalid, a InvalidURLError is raised.

payload

Body content for a POST, PUT, or PATCH request.

method

The HTTP method to use for the request. Acceptable values include GET, POST, HEAD, PUT, DELETE, or PATCH. These values are constants provided by the package. The value can also be a string equivalent to the name of the method.

headers

The set of HTTP headers to include with the request, as a mapping of names and values. For security reasons, some HTTP headers cannot be modified by the application. See the URL Fetch Service overview.

allow_truncated

If False and the response data exceeds the maximum allowed response size, a ResponseTooLargeError exception is raised when check_success() or get_result() is called on the RPC object. If True, no exception is raised, and the response's content is truncated to the maximum size, and the response object's content_was_truncated attribute is set to True. For more information on limits, see the URL Fetch Service overview.

follow_redirects

If True, responses that are HTTP redirects are followed, up to 5 consecutive redirects. The response data is from the final location, as if the data were for the requested location. If False, redirects are not followed, and a redirect response is returned directly to the application, including the header information that describes the redirect.

Warning: Cookies are not handled upon redirection. If cookie handling is needed, set follow_redirects to False and handle both cookies and redirects manually.

validate_certificate

A value of True instructs the application to send a request to the server only if the certificate is valid and signed by a trusted certificate authority (CA), and also includes a hostname that matches the certificate. If the certificate validation fails, a SSLCertificateError exception is raised. A value of False instructs the application t perform no certificate validation.

Note: This parameter currently defaults to False, but will default to True in the near future. If you rely upon making requests to a site with an invalid or untrusted certificate, you should explicitly pass validate_certificate=False to avoid errors in future versions.

The RPC Object

The create_rpc() function returns an object that represents the asynchronous procedure call. The app uses the make_fetch_call() function with this object to initiate a URL fetch, then calls methods on this object to wait for and retrieve results.

The object has the following methods.

wait()

Waits for the URL fetch in progress to complete, invokes the RPC callback function if any, then returns.

If the request has already completed by the time the method is called, the method calls the callback function (if any) and returns immediately. The request is complete when the remote host successfully returns a response, the deadline elapses, or the request fails for another reason.

If wait() has already been called for this RPC object, a subsequent call does nothing. The callback function is not called again.

check_success()

Calls the wait() method, then checks whether the request got a response successfully. If the deadline elapsed, or if the request failed, this method raises an appropriate exception. Otherwise, the method returns without raising an exception.

get_result()

Calls check_success() (which in turn calls the wait() method), then returns an object containing the response data. See Response Objects.

The call to check_success() raises an exception if the deadline elapsed or if the request failed. get_result() propagates this exception to the caller.

The object has the following properties:

callback

The callback function. You can modify this property after the object has been created to use a callback function that can refer to the RPC object, such as to call get_result(). See the example in Making Asynchronous Requests, above.

deadline

The deadline for the fetch. This property is read-only.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies.