requests-futures

Small add-on for the python requests http library that makes use of the standard library's concurrent.futures. Requires Python 3.10 or newer.

The additional API and changes are minimal and strives to avoid surprises.

The following synchronous code:

from requests import Session

session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)

Can be translated to make use of futures, and thus be asynchronous by creating a FuturesSession and catching the returned Future in place of Response. The Response can be retrieved by calling the result method on the Future:

from requests_futures.sessions import FuturesSession

session = FuturesSession()
# first request is started in background
future_one = session.get('http://httpbin.org/get')
# second requests is started immediately
future_two = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)

By default a ThreadPoolExecutor is created with 8 workers. That default executor is a single requests.Session shared across all of its worker threads, and Session is not thread-safe: concurrently mutating shared state on it, such as the cookie jar or session.headers, from multiple in-flight requests is a real hazard. See the full thread-safety discussion in the docs for what to do instead.

That's it. The api of requests.Session is preserved without any modifications beyond returning a Future rather than Response. As with all futures exceptions are shifted (thrown) to the future.result() call so try/except blocks should be moved there.

The most common piece of information needed is the URL of the request. This can be accessed without any extra steps using the request property of the response object.

from concurrent.futures import as_completed
from pprint import pprint
from requests_futures.sessions import FuturesSession

session = FuturesSession()

futures=[session.get(f'http://httpbin.org/get?{i}') for i in range(3)]

for future in as_completed(futures):
    resp = future.result()
    pprint({
        'url': resp.request.url,
        'content': resp.json(),
    })

There are situations in which you may want to tie additional information to a request/response. There are a number of ways to go about this, the simplest is to attach additional information to the future object itself.

from concurrent.futures import as_completed
from pprint import pprint
from requests_futures.sessions import FuturesSession

session = FuturesSession()

futures=[]
for i in range(3):
    future = session.get('http://httpbin.org/get')
    future.i = i
    futures.append(future)

for future in as_completed(futures):
    resp = future.result()
    pprint({
        'i': future.i,
        'content': resp.json(),
    })

If you know that you won't be needing any additional responses from futures that haven't yet resolved, it's a good idea to cancel those requests. You can do this by using the session as a context manager:

from requests_futures.sessions import FuturesSession
with FuturesSession(max_workers=1) as session:
    future = session.get('https://httpbin.org/get')
    future2 = session.get('https://httpbin.org/delay/10')
    future3 = session.get('https://httpbin.org/delay/10')
    response = future.result()

Exiting the with block calls close(), which shuts down the session's own executor with cancel_futures=True: every queued request that hasn't started running yet is cancelled immediately, not merely skipped one at a time. Here, with max_workers=1, future is already done by the time the block exits, but neither future2 nor future3 has started, so both are cancelled together, saving the time and resources their requests would otherwise have used. A request that is already running when close() runs is left to finish.

A ProcessPoolExecutor can be supplied in place of a ThreadPoolExecutor to run requests in separate processes instead of threads, which is useful when per-request memory usage is high enough that cycling the interpreter is needed to release it back to the OS.

Everything submitted to a process pool must be picklable, which means any FuturesSession subclass used this way must be importable at module scope (not defined inline, e.g. in a function or __main__ script body). If something in the request isn't picklable, FuturesSession raises a RuntimeError pointing back to this documentation rather than letting the pickling failure surface on its own.

from concurrent.futures import ProcessPoolExecutor
from requests_futures.sessions import FuturesSession

session = FuturesSession(executor=ProcessPoolExecutor(max_workers=10))
# ... use as before

See the full ProcessPoolExecutor guide in the docs for a module-global callback example and more on the pickling requirements.

Full documentation, including everything below, lives at https://requests-futures.readthedocs.io/:

• as_completed with a timeout
• error handling across the future boundary (submit-time vs. result()-time)
• retries via a mounted HTTPAdapter and urllib3's Retry
• sizing max_workers against the connection pool, including with a supplied session=
• streaming responses (stream=True) and why it interacts badly with background work
• sharing one executor across several sessions, and what close() does in that configuration
• hooks, the recommended replacement for the deprecated background_callback
• context propagation and tracing, including OpenTelemetry
• the full ProcessPoolExecutor guide
• the full thread-safety discussion

pip install requests-futures