Python equivalent of a given wget command

There is also a nice Python module named wget that is pretty easy to use. Keep in mind that the package has not been updated since 2015 and has not implemented a number of important features, so it may be better to use other methods. It depends entirely on your use case. For simple downloading, this module is the ticket. If you need to do more, there are other solutions out there.

>>> import wget
>>> url = http://www.futurecrew.com/skaven/song_files/mp3/razorback.mp3
>>> filename = wget.download(url)
100% [................................................] 3841532 / 3841532>
>> filename
razorback.mp3

Enjoy.

However, if wget doesnt work (Ive had trouble with certain PDF files), try this solution.

Edit: You can also use the out parameter to use a custom output directory instead of current working directory.

>>> output_directory = <directory_name>
>>> filename = wget.download(url, out=output_directory)
>>> filename
razorback.mp3

urllib.request should work.
Just set it up in a while(not done) loop, check if a localfile already exists, if it does send a GET with a RANGE header, specifying how far you got in downloading the localfile.
Be sure to use read() to append to the localfile until an error occurs.

This is also potentially a duplicate of Python urllib2 resume download doesnt work when network reconnects

Python equivalent of a given wget command

I had to do something like this on a version of linux that didnt have the right options compiled into wget. This example is for downloading the memory analysis tool guppy. Im not sure if its important or not, but I kept the target files name the same as the url target name…

Heres what I came up with:

python -c import requests; r = requests.get(https://pypi.python.org/packages/source/g/guppy/guppy-0.1.10.tar.gz) ; open(guppy-0.1.10.tar.gz , wb).write(r.content)

Thats the one-liner, heres it a little more readable:

import requests
fname = guppy-0.1.10.tar.gz
url = https://pypi.python.org/packages/source/g/guppy/ + fname
r = requests.get(url)
open(fname , wb).write(r.content)

This worked for downloading a tarball. I was able to extract the package and download it after downloading.

EDIT:

To address a question, here is an implementation with a progress bar printed to STDOUT. There is probably a more portable way to do this without the clint package, but this was tested on my machine and works fine:

#!/usr/bin/env python

from clint.textui import progress
import requests

fname = guppy-0.1.10.tar.gz
url = https://pypi.python.org/packages/source/g/guppy/ + fname

r = requests.get(url, stream=True)
with open(fname, wb) as f:
    total_length = int(r.headers.get(content-length))
    for chunk in progress.bar(r.iter_content(chunk_size=1024), expected_size=(total_length/1024) + 1): 
        if chunk:
            f.write(chunk)
            f.flush()

Python equivalent of a given wget command

Python equivalent of a given wget command

Leave a Reply Cancel reply