logging – Log all requests from the python-requests module

logging – Log all requests from the python-requests module

You need to enable debugging at httplib level (requests → urllib3 → httplib).

Heres some functions to both toggle (..._on() and ..._off()) or temporarily have it on:

import logging
import contextlib
try:
    from http.client import HTTPConnection # py3
except ImportError:
    from httplib import HTTPConnection # py2

def debug_requests_on():
    Switches on logging of the requests module.
    HTTPConnection.debuglevel = 1

    logging.basicConfig()
    logging.getLogger().setLevel(logging.DEBUG)
    requests_log = logging.getLogger(requests.packages.urllib3)
    requests_log.setLevel(logging.DEBUG)
    requests_log.propagate = True

def debug_requests_off():
    Switches off logging of the requests module, might be some side-effects
    HTTPConnection.debuglevel = 0

    root_logger = logging.getLogger()
    root_logger.setLevel(logging.WARNING)
    root_logger.handlers = []
    requests_log = logging.getLogger(requests.packages.urllib3)
    requests_log.setLevel(logging.WARNING)
    requests_log.propagate = False

@contextlib.contextmanager
def debug_requests():
    Use with with!
    debug_requests_on()
    yield
    debug_requests_off()

Demo use:

>>> requests.get(http://httpbin.org/)
<Response [200]>

>>> debug_requests_on()
>>> requests.get(http://httpbin.org/)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:GET / HTTP/1.1 200 12150
send: GET / HTTP/1.1rnHost: httpbin.orgrnConnection: keep-alivernAccept-
Encoding: gzip, deflaternAccept: */*rnUser-Agent: python-requests/2.11.1rnrn
reply: HTTP/1.1 200 OKrn
header: Server: nginx
...
<Response [200]>

>>> debug_requests_off()
>>> requests.get(http://httpbin.org/)
<Response [200]>

>>> with debug_requests():
...     requests.get(http://httpbin.org/)
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
...
<Response [200]>

You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA. The only thing missing will be the response.body which is not logged.

Source

The underlying urllib3 library logs all new connections and URLs with the logging module, but not POST bodies. For GET requests this should be enough:

import logging

logging.basicConfig(level=logging.DEBUG)

which gives you the most verbose logging option; see the logging HOWTO for more details on how to configure logging levels and destinations.

Short demo:

>>> import requests
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> r = requests.get(http://httpbin.org/get?foo=bar&baz=python)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80
DEBUG:urllib3.connectionpool:http://httpbin.org:80 GET /get?foo=bar&baz=python HTTP/1.1 200 366

Depending on the exact version of urllib3, the following messages are logged:

• INFO: Redirects
• WARN: Connection pool full (if this happens often increase the connection pool size)
• WARN: Failed to parse headers (response headers with invalid format)
• WARN: Retrying the connection
• WARN: Certificate did not match expected hostname
• WARN: Received response with both Content-Length and Transfer-Encoding, when processing a chunked response
• DEBUG: New connections (HTTP or HTTPS)
• DEBUG: Dropped connections
• DEBUG: Connection details: method, path, HTTP version, status code and response length
• DEBUG: Retry count increments

This doesnt include headers or bodies. urllib3 uses the http.client.HTTPConnection class to do the grunt-work, but that class doesnt support logging, it can normally only be configured to print to stdout. However, you can rig it to send all debug information to logging instead by introducing an alternative print name into that module:

import logging
import http.client

httpclient_logger = logging.getLogger(http.client)

def httpclient_logging_patch(level=logging.DEBUG):
    Enable HTTPConnection debug logging to the logging framework

    def httpclient_log(*args):
        httpclient_logger.log(level,  .join(args))

    # mask the print() built-in in the http.client module to use
    # logging instead
    http.client.print = httpclient_log
    # enable debugging
    http.client.HTTPConnection.debuglevel = 1

Calling httpclient_logging_patch() causes http.client connections to output all debug information to a standard logger, and so are picked up by logging.basicConfig():

>>> httpclient_logging_patch()
>>> r = requests.get(http://httpbin.org/get?foo=bar&baz=python)
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80
DEBUG:http.client:send: bGET /get?foo=bar&baz=python HTTP/1.1rnHost: httpbin.orgrnUser-Agent: python-requests/2.22.0rnAccept-Encoding: gzip, deflaternAccept: */*rnConnection: keep-alivernrn
DEBUG:http.client:reply: HTTP/1.1 200 OKrn
DEBUG:http.client:header: Date: Tue, 04 Feb 2020 13:36:53 GMT
DEBUG:http.client:header: Content-Type: application/json
DEBUG:http.client:header: Content-Length: 366
DEBUG:http.client:header: Connection: keep-alive
DEBUG:http.client:header: Server: gunicorn/19.9.0
DEBUG:http.client:header: Access-Control-Allow-Origin: *
DEBUG:http.client:header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:http://httpbin.org:80 GET /get?foo=bar&baz=python HTTP/1.1 200 366

logging – Log all requests from the python-requests module

For those using python 3+

import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger(requests.packages.urllib3)
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True