mnist – Extract images from .idx3-ubyte file or GZIP via Python

Download the training/test images and labels:

train-images-idx3-ubyte.gz: training set images
train-labels-idx1-ubyte.gz: training set labels
t10k-images-idx3-ubyte.gz: test set images
t10k-labels-idx1-ubyte.gz: test set labels

And uncompress them in a workdir, say samples/.

Get the python-mnist package from PyPi:

pip install python-mnist

Import the mnist package and read the training/test images:

from mnist import MNIST

mndata = MNIST(samples)

images, labels = mndata.load_training()
# or
images, labels = mndata.load_testing()

To display an image to the console:

index = random.randrange(0, len(images))  # choose an index ;-)
print(mndata.display(images[index]))

Youll get something like this:

............................
............................
............................
............................
............................
.................@@.........
..............@@@@@.........
............@@@@............
..........@@................
..........@.................
...........@................
...........@................
...........@...@............
...........@@@@@.@..........
...........@@@...@@.........
...........@@.....@.........
..................@.........
..................@@........
..................@@........
..................@.........
.................@@.........
...........@.....@..........
...........@....@@..........
............@@@@............
.............@..............
............................
............................
............................

Explanation:

Each image of the images list is a Python list of unsigned bytes.
The labels is an Python array of unsigned bytes.

(Using only matplotlib, gzip and numpy)

Extract image data:

import gzip
f = gzip.open(train-images-idx3-ubyte.gz,r)

image_size = 28
num_images = 5

import numpy as np
f.read(16)
buf = f.read(image_size * image_size * num_images)
data = np.frombuffer(buf, dtype=np.uint8).astype(np.float32)
data = data.reshape(num_images, image_size, image_size, 1)

Print images:

import matplotlib.pyplot as plt
image = np.asarray(data[2]).squeeze()
plt.imshow(image)
plt.show()

Print first 50 labels:

f = gzip.open(train-labels-idx1-ubyte.gz,r)
f.read(8)
for i in range(0,50):   
    buf = f.read(1)
    labels = np.frombuffer(buf, dtype=np.uint8).astype(np.int64)
    print(labels)

mnist – Extract images from .idx3-ubyte file or GZIP via Python

You could actually use the idx2numpy package available at PyPI. Its extremely simple to use and directly converts the data to numpy arrays.
Heres what you have to do:

Downloading the data

Download the MNIST dataset from the official website.
If youre using Linux then you can use wget to get it from command line itself. Just run:

wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz

Decompressing the data

Unzip or decompress the data. On Linux, you could use gzip

Ultimately, you should have the following files:

data/train-images-idx3-ubyte
data/train-labels-idx1-ubyte
data/t10k-images-idx3-ubyte
data/t10k-labels-idx1-ubyte

The prefix data/ is just because Ive extracted them into a folder named data. Your question looks like youre well done till here, so keep reading.

Using idx2numpy

Heres a simple python code to read everything from the decompressed files as numpy arrays.

import idx2numpy
import numpy as np
file = data/train-images-idx3-ubyte
arr = idx2numpy.convert_from_file(file)
# arr is now a np.ndarray type of object of shape 60000, 28, 28

You can now use it with OpenCV juts the same way how you display any other image, using something like

cv.imshow(Image, arr[4])

To install idx2numpy, you can use PyPI (pip package manager). Simply run the command:

pip install idx2numpy