Random Notes

From Noah.org
Jump to: navigation, search


Random Notes

This is not a collection of random notes. This is a collection of notes about random numbers.

/dev/random and /dev/urandom

The file /dev/random is the direct interface to the Linux kernel entropy pool and cause reads to block when the entropy pool is empty. The file /dev/urandom is an unblocked interface to the entropy pool. It will reuse the entropy if it runs out. It does blend this through a pseudorandom number generator, so you shouldn't actually see repeated patterns in the numbers; however, these numbers are not considered as random the numbers from /dev/random. This random source may not be as secure for software that depends on random numbers for security.

/dev/random is a limited resource

The device /dev/random is better, but it is "slow" because it depends on a limited resource. It reads directly from the Linux kernel entropy pool to give you truly random numbers. The entropy pool generates truly random numbers by sampling data from the environment such as clock drift and timing of input from device drivers. This sampled data is blended and tested to ensure that it is truly random and without bias. This is an expensive operation, so the kernel constantly maintains a pool (cache) of random numbers which can be read from /dev/random. The problem is that once you have read all the random numbers from the pool further read operations must block until the kernel writes more random numbers into the pool. You see just how slow this is by running the following command:

dd if=/dev/random bs=1 | od -x

You should see a bunch of random hex digits dumped after which `dd` will block and then slowly print new random numbers as the pool gets filled. You may notice that if you move your mouse a lot that new random numbers will be printed more quickly. This shows that the kernel gets some of its environmental input from the mouse device driver.

You can see how many random bytes are in the pool by looking at the file /proc/sys/kernel/random/entropy_avail. If you watch this file (`watch -n 1 cat /proc/sys/kernel/random/entropy_avail`) you may see that the number goes up and down. This is because various services running in the background and even kernel device drivers also read from the entropy pool. In particular, services that use SSL will deplete the entropy pool. If you "waste" the entropy pool by running `dd if=/dev/random bs=1 | od -x` then you may slow down these services because they much block until more entropy is available. Also, if you have many processes running which all use SSL then they may slow each other down. The CPU load will be load, but load average will go up as more processes are forced to block waiting for more entropy.

One way to increase the speed of the supply to the entropy pool is to use a dedicated hardware random number generator. These devices provide a good entropy source much more quickly than what the kernel can obtain from the environment. The Ubuntu package rng-tools can be used to setup and control sources of entropy.

Another way to increase the speed of the supply to the entropy pool is to hack the kernel entropy pool to make it less random (less good). In general, you should not notice the difference, but be aware that you may compromise secure applications such as web services and anything that accepts SSL connections. This doesn't create a huge back door into these applications, but if you are dealing with valuable sensitive data then you shouldn't be playing around with mathematics that you do not understand. In other words, if lives, money, or your job is at stake then get a real hardware random number generator... To hack the kernel entropy pool you must install rng-tools. Then edit /etc/default/rng-tools and edit or insert a line for HRNGDEVICE to the following:

HRNGDEVICE=/dev/urandom

Finally, restart the `rngd` daemon:

sudo service rng-tools restart
 # or
sudo /etc/init.d/rng-tools restart

Doing this basically bypasses /dev/random with data from /dev/urandom. You can now read as much data from /dev/random as you want without blocking. Remember, this is a horrible thing to do. It will make security and statistics nerds go into a rage. I only show this because it is a good learning experience.

testing random numbers for randomness

There are a few Ubuntu packages which provide tools for testing the quality of random number generators. rng-tools, ent, and dieharder. The openssl command is also useful for testing.

sudo aptitude install ent rng-tools dieharder

generate test data

Both dieharder and openssl can be used to generate pseudorandom numbers. I use opensll to generate pseudorandom numbers. The following generates a 1 GB file named "entropy.bin". This uses the password 1234567890 as the seed.

dd if=/dev/zero bs=1000000 count=1000 | openssl aes-256-cbc -e -pass pass:1234567890 -out entropy.bin
cat entropy.bin | rngtest

The output can also be fed directly to a random number tester:

dd if=/dev/zero bs=1000000 count=1000 | openssl aes-256-cbc -e -pass pass:1234567890 | rngtest

Note that 1 GB is insufficient for dieharder. It needs over 100 GB of data for testing, so it's better to feed the output of openssl directly to dieharder.

cat /dev/zero | openssl aes-256-cbc -e -pass pass:1234567890 | dieharder -a -g 200

You should find that aes-256-cbc is a very good random number generator. It should be since AES is the state of the art for block ciphers. The down-side of using it as a pseudorandom number generator is that it is slow compared to a dedicated PRNG such as Mersenne Twister MT19937. You can compare the timing as follows:

time cat /dev/zero | openssl aes-256-cbc -e -pass pass:1234567890 | dieharder -a -g 200
time dieharder -a -g 13

Both give excellent results, but Mersenne Twister MT19937 (-g 13) is about twice as fast as OpenSSL aes-256-cbc (MT19937: 1049 s, OpenSSL aes-256-cbc: 2127 s).

ent

The ent testing tool is easy to use. Interpreting these values is beyond the scope of this introduction.

dd if=/dev/urandom bs=10 count=100000 | ent

rngtest in rng-tools

The rngtest utility will test a given stream against the FIPS 140-2 randomness tests. It is very easy to use and the results are fairly easy to read. It operates on blocks of 20000 bits at a time. After testing a report is printed showing the number of blocks that successfully passed or failed a given test. Note that if you test a large number of blocks that you should expect a few failures.

cat entropy.bin | rngtest

If you have a limitless random number source then you might want to display statistics after a given number of bits have been tested. The rngtest tool tests blocks of 20000 bits at a time. It throws away the first 32 bits of any input stream. The following will test /dev/urandom forever and print statistics every 10000 blocks (200000032 bits, or about 25 MB).

cat /dev/urandom | rngtest -b 10000

dieharder

Dieharder is both a random number generator and a RNG tester. It contains a large number of tests and RNG algorithms to test against. Dieharder requires lots of sample data for testing. The tests take a long time to run. For testing a file of random bytes I use the following command:

cat entropy.bin | dieharder -a -g 200

The -a option selects all tests. The -g NNN option selects a random number source. In the first example below -g 200 selects raw binary from stdin. Many other random number sources may be selected. Most sources choose internal Dieharder generators. For example, the following two examples are equivalent.

cat /dev/urandom | dieharder -a -g 200
dieharder -a -g 501

Dieharder can be used to test other random number testers by generating random numbers. The following will generate 1 MB of random numbers using algorithm 13 (Mersenne Twister MT19937) with starting seed 0. The Mersenne Twister MT19937 algorithm is a 32-bit generator, so I select 250000 iterations to produce 1 MB.

dieharder -g 13 -S 0 -t 250000 -o -B | rngtest

You can see that selecting a different seed can cause some tests to fail. This is normal. Over a large number of blocks no RNG will always pass every FIPS 140-2 test.

dieharder -g 13 -S 1 -t 250000 -o -B | rngtest

online testing

CAcert has an online testing tool. It requires a minimum of 12 MB of binary data for testing. See http://www.cacert.at/random/

play random noise

aplay -c 2 -f S16_LE -r 44100 /dev/urandom

Also

cat /dev/urandom | padsp tee /dev/audio > /dev/null
# or
cat /dev/urandom | pacat

Loop the microphone to the speaker.

padsp cat /dev/audio | padsp tee /dev/audio > /dev/null

The padsp command looks for /dev/audio in the given command and then transparently substitutes the PulseAudio driver calls using LD_PRELOAD to substitute calls.

More reading

An old analysis of the Linux Random Number Generator (LRNG) written by Gutterman, Pinkas, and Reinman can be found here: http://eprint.iacr.org/2006/086.pdf

This is a good directory of information about random numbers http://www.cs.berkeley.edu/~daw/rnd/

'Mouse Trap Bubble Light Entropy Engine' -- A truly random number generator

If you need a simple source of entropy, you might consider building my Mouse Trap Bubble Light Entropy Engine.