Differences between numpy.random.rand vs numpy.random.randn in Python

Differences between numpy.random.rand vs numpy.random.randn in Python

First, as you see from the documentation numpy.random.randn generates samples from the normal distribution, while numpy.random.rand from a uniform distribution (in the range [0,1)).

Second, why did the uniform distribution not work? The main reason is the activation function, especially in your case where you use the sigmoid function. The plot of the sigmoid looks like the following:

enter

So you can see that if your input is away from 0, the slope of the function decreases quite fast and as a result you get a tiny gradient and tiny weight update. And if you have many layers – those gradients get multiplied many times in the back pass, so even proper gradients after multiplications become small and stop making any influence. So if you have a lot of weights which bring your input to those regions you network is hardly trainable. Thats why it is a usual practice to initialize network variables around zero value. This is done to ensure that you get reasonable gradients (close to 1) to train your net.

However, uniform distribution is not something completely undesirable, you just need to make the range smaller and closer to zero. As one of good practices is using Xavier initialization. In this approach you can initialize your weights with:

  1. Normal distribution. Where mean is 0 and var = sqrt(2. / (in + out)), where in – is the number of inputs to the neurons and out – number of outputs.
  2. Uniform distribution in range [-sqrt(6. / (in + out)), +sqrt(6. / (in + out))]
  • np.random.rand is for Uniform distribution (in the half-open interval [0.0, 1.0))
  • np.random.randn is for Standard Normal (aka. Gaussian) distribution (mean 0 and variance 1)

You can visually explore the differences between these two very easily:

import numpy as np
import matplotlib.pyplot as plt

sample_size = 100000
uniform = np.random.rand(sample_size)
normal = np.random.randn(sample_size)

pdf, bins, patches = plt.hist(uniform, bins=20, range=(0, 1), density=True)
plt.title(rand: uniform)
plt.show()

pdf, bins, patches = plt.hist(normal, bins=20, range=(-4, 4), density=True)
plt.title(randn: normal)
plt.show()

Which produce:

enter

and

enter

Differences between numpy.random.rand vs numpy.random.randn in Python

1) numpy.random.rand from uniform (in range [0,1))

2) numpy.random.randn generates samples from the normal distribution

Related posts:

  • python – How to input a 1-D array of dimensions into numpy.random.randn?
  • python – get integet value from a tf.experimental.numpy.random.randint
  • python – Mean of normal distribution generated using numpy.random.randn() is not 0
  • Draw multivariate Gaussian distribution samples using Python numpy.random.randn
  • python – When to use numpy.random.randn(…) and when numpy.random.rand(…)?
  • python – TypeError: an integer is required when initializing random weight matrix in numpy with numpy.random.randn()
  • arrays – Freezing output after running numpy.random.randint() in Python
  • python – Same random numbers in C++ as computed by Python3 numpy.random.rand
  • Differences between numpy.random.rand vs numpy.random.randn in Python
  • python – What does -1 mean in numpy reshape?

Leave a Reply

Your email address will not be published.