Get html using Python requests?

Get html using Python requests?

The server in question is giving you a gzipped response. The server is also very broken; it sends the following headers:

$ curl -D - -o /dev/null -s -H Accept-Encoding: gzip, deflate
HTTP/1.1 200 OK
Date: Tue, 06 Jan 2015 17:46:49 GMT
Server: Apache
<!DOCTYPE HTML PUBLIC -//W3C//DTD XHTML 1.0 Transitional//EN DTD/xhtml1-transitional.dtd><html xmlns=http: // lang=en-US>
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 3659
Content-Type: text/html

The <!DOCTYPE..> line there is not a valid HTTP header. As such, the remaining headers past Server are ignored. Why the server interjects that is unclear; in all likely hood is a CGI script that doesnt output headers but does include a double newline after the doctype line, duping the Apache server into inserting additional headers there.

As such, requests also doesnt detect that the data is gzip-encoded. The data is all there, you just have to decode it. Or you could if it wasnt rather incomplete.

The work-around is to tell the server not to bother with compression:

headers = {Accept-Encoding: identity}
r = requests.get(url, headers=headers)

and an uncompressed response is returned.

Incidentally, on Python 2 the HTTP header parser is not so strict and manages to declare the doctype a header:

>>> pprint(dict(r.headers))
{<!doctype html public -//w3c//dtd xhtml 1.0 transitional//en dtd/xhtml1-transitional.dtd><html xmlns=http: // lang=en-US>,
 connection: Keep-Alive,
 content-encoding: gzip,
 content-length: 3659,
 content-type: text/html,
 date: Tue, 06 Jan 2015 17:42:06 GMT,
 keep-alive: timeout=5, max=100,
 server: Apache,
 vary: Accept-Encoding}

and the content-encoding information survives, so there requests decodes the content for you, as expected.

The HTTP headers for this URL have now been fixed.

>>> import requests
>>> print requests.__version__
>>> r = requests.get(
>>> r.text[:100]
un<!DOCTYPE html>n<HTML>n<HEAD><TITLE>Monthly Average of Precipitation, Station id: 028815</TITLE></H
>>> r.headers
{content-length: 3672, content-encoding: gzip, vary: Accept-Encoding, keep-alive: timeout=5, max=100, server: Apache, connection: Keep-Alive, date: Thu, 12 Feb 2015 18:59:37 GMT, content-type: text/html; charset=utf-8}

Get html using Python requests?

Id solve that problem in a more simple way. Just import html library to decode HTML special characters:

import html

r = requests.get(


Leave a Reply

Your email address will not be published. Required fields are marked *