Web server offers true random numbers via fluctuations of the quantum vacuum(jwz.org)
jwz.org
Web server offers true random numbers via fluctuations of the quantum vacuum
http://www.jwz.org/blog/2012/04/random-numbers-via-fluctuations-of-the-quantum-vacuum/
12 comments
On Linux, the need for hardware sources became more visible because of this:
"/dev/random vs /dev/urandom" http://www.onkarjoshi.com/blog/191/device-dev-random-vs-uran...
/dev/random is hardware entropy dependent, and if application users use it too much (and we all know that many application developers don't do what's minimal stress for hardware or what's optimal "I just make it fast, it works for me"), every such application can block too often. Especially servers are problematic because you can't accumulate the entropy by measuring keyboard or whatever. That's why Intel made RdRand. And their competitors made some similar solutions even before that.
Good detailed discussion also on: http://en.wikipedia.org/wiki//dev/random
In short, hardware RNG in the CPU is a very good thing.
Of course, whoever doesn't need cryptographically secure randomness should anyway just use plain old pure software pseudorandom number generator:
http://en.wikipedia.org/wiki/Pseudorandom_number_generator
"/dev/random vs /dev/urandom" http://www.onkarjoshi.com/blog/191/device-dev-random-vs-uran...
/dev/random is hardware entropy dependent, and if application users use it too much (and we all know that many application developers don't do what's minimal stress for hardware or what's optimal "I just make it fast, it works for me"), every such application can block too often. Especially servers are problematic because you can't accumulate the entropy by measuring keyboard or whatever. That's why Intel made RdRand. And their competitors made some similar solutions even before that.
Good detailed discussion also on: http://en.wikipedia.org/wiki//dev/random
In short, hardware RNG in the CPU is a very good thing.
Of course, whoever doesn't need cryptographically secure randomness should anyway just use plain old pure software pseudorandom number generator:
http://en.wikipedia.org/wiki/Pseudorandom_number_generator
ln -sf /dev/urandom /dev/random
OpenSSL, OpenBSD, and FreeBSD agree with me.
OpenSSL, OpenBSD, and FreeBSD agree with me.
One wouldn't want to give the impression that CSPRNG design is a solved problem, but I think you're right: exotic "true random" entropy sources definitely don't target the real challenges.
Are you aware of any instances of the OS /dev/Xrandom failing in an exploitable way, assuming proper initialization? I'm sure that people still mess up, but there are a lot of problems that are solved in that sense.
There's a well-known and very good Black Hat talk from I think 2007 that attacked cold-start entropy on embedded devices, but you could rephrase that as an entropy estimation problem.
I believe it is largely a solved problem. Or at least the unsolved parts are reduced to a single problem: entropy input estimation.
If you have a modernish machine running Linux, find out if it has a TPM (Trusted Platform Module) built in. If it does, you can use an application called "Trousers" to take advantage of its built in hardware random number generator. It will directly feed random data into /dev/random
If you don't have a TPM, you can get one of these usb sticks: http://www.entropykey.co.uk/ and it will also feed /dev/random with large amounts of real random data.
If you don't have a TPM, you can get one of these usb sticks: http://www.entropykey.co.uk/ and it will also feed /dev/random with large amounts of real random data.
Can anyone explain how this works? The entropy key site says "The Entropy Key contains two high-quality noise generators, and an ARM Cortex CPU that actively measures, checks and confirms all generated random numbers, before encrypting them and sending them to the server"... but why can't we just write high-quality noise generators in software? How is the hardware better?
In theory, as others that replied have noted, hardware is better ... in practice there are issues other than "software is deterministic".
Many naive hardware based random number generators suffer from being not as random as one might think - thanks to quantising levels in A/D converters, to supposedly random physical processes having "spectra" where measurable signal occurs more in some bands than others, and to a multitude of other odd little effects just making a hardware based RNG is as simple as it seems.
The entropy key cited above has multiple noise sources and PRNG processes that mix them up and running checks to see that things are working as expected. That level of paranoid checking is more or less the minimum level required in a RNG to be confident about it.
Many naive hardware based random number generators suffer from being not as random as one might think - thanks to quantising levels in A/D converters, to supposedly random physical processes having "spectra" where measurable signal occurs more in some bands than others, and to a multitude of other odd little effects just making a hardware based RNG is as simple as it seems.
The entropy key cited above has multiple noise sources and PRNG processes that mix them up and running checks to see that things are working as expected. That level of paranoid checking is more or less the minimum level required in a RNG to be confident about it.
Everyone has mentioned how software is deterministic.
Hardware RNGs sometimes rely on thermal noise (which is really random) but there are sometimes flaws with how that noise is sampled and de-skewed. Also, they need to be monitored to cope with hardware failure. Be aware, especially if you're using them for cryptography, that they might be a poor fit for your purpose.
Testing Hardware RNGs
(http://www.robertnz.net/true_rng.html)
EDIT:
Descriptions of various forms of noise:
(http://www.eie.polyu.edu.hk/~ensurya/lect_notes/commun_cir/C...)
And surely everyone on HN knows that part of the "snow" noise displayed on an untuned TV is cosmic background radiation, ie "afterglow" of the big bang. I still find that amazing.
Hardware RNGs sometimes rely on thermal noise (which is really random) but there are sometimes flaws with how that noise is sampled and de-skewed. Also, they need to be monitored to cope with hardware failure. Be aware, especially if you're using them for cryptography, that they might be a poor fit for your purpose.
Testing Hardware RNGs
(http://www.robertnz.net/true_rng.html)
EDIT:
Descriptions of various forms of noise:
(http://www.eie.polyu.edu.hk/~ensurya/lect_notes/commun_cir/C...)
And surely everyone on HN knows that part of the "snow" noise displayed on an untuned TV is cosmic background radiation, ie "afterglow" of the big bang. I still find that amazing.
Actually, only about 5% is. The rest has more local origin.
Pure software-based random generators are deterministic and thus not random.
When something is deterministic, it is reproducible and therefore not good for creating crypto keys.
Most /dev/random implementations use data from the ethernet driver, the keyboard, mouse etc to get some input which isn't easy to reproduce.
When something is deterministic, it is reproducible and therefore not good for creating crypto keys.
Most /dev/random implementations use data from the ethernet driver, the keyboard, mouse etc to get some input which isn't easy to reproduce.
[deleted]
Because you can't get really random number from deterministic hardware (and computers are deterministic, more or less :) ). No matter what you do, running the same algorithm on the same input gives you the same data on regular computer. And the randomness (measured in Kolmogorov complexity [1] of the resulting string) is by definition smaller than the length of the generator program.
And truly random string is such string, that there's no possible program producing this string as output, that is shorter than the string.
On deterministic computers you can only produce pseudorandom numbers, with varying quality of randomness, depending on generator program you use, but it's sometimes not enough (esp. in cryptography).
[1] http://en.wikipedia.org/wiki/Kolmogorov_complexity
And truly random string is such string, that there's no possible program producing this string as output, that is shorter than the string.
On deterministic computers you can only produce pseudorandom numbers, with varying quality of randomness, depending on generator program you use, but it's sometimes not enough (esp. in cryptography).
[1] http://en.wikipedia.org/wiki/Kolmogorov_complexity
Software is deterministic, so it is not really possible to generate noise. There are non-deterministic parts of software (for example, thread interaction in concurrent systems), but this is just observable behavior of hardware effects.
The typical thing people do to get large amounts of randomness is to first generate some "real" random noise from some source (hardware thingamajigs, network timings, user interactions) and extrapolate these into longer sequences of random-looking numbers. The extrapolation can be fast and simple (in which case the randomness is not so good), or it can be slow and complex (to get decent looking randomness).
The typical thing people do to get large amounts of randomness is to first generate some "real" random noise from some source (hardware thingamajigs, network timings, user interactions) and extrapolate these into longer sequences of random-looking numbers. The extrapolation can be fast and simple (in which case the randomness is not so good), or it can be slow and complex (to get decent looking randomness).
Others have explained why it's not possible to create a good software random number generator. Hardware RNGs don't just provide a much better quality stream of random data, they also provide a much higher quantity of it. I just generate a GnuPG 2048 bit keypair by running "gpg --gen-key" on my laptop. It took over 11 minutes to gather enough entropy to build the key. I then plugged my Entropy Key into the laptop and repeated the process. This time it took less than 2 seconds.
What kind of laptop is this? We generate 2048 bit keys all the time with GPG. It's never taken 11 minutes. I just generated one in under 5 seconds.
A Lenovo Thinkpad T420 with an Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 8GB of RAM and an SSD. It's running Ubuntu 10.04, GnuPG 2.0.14, and a 3.2.0 Linux kernel.
The laptop was relatively idle when the test was being run.
It's worth noting that when you do a --gen-key it does output the message:
"We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilise the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy."
I can't imagine it would state that if "5 seconds" was anything like normal. Perhaps you have some sort of additional source of entropy which you don't know about.
The laptop was relatively idle when the test was being run.
It's worth noting that when you do a --gen-key it does output the message:
"We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilise the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy."
I can't imagine it would state that if "5 seconds" was anything like normal. Perhaps you have some sort of additional source of entropy which you don't know about.
> If you have a modernish machine running Linux, find out if it has a TPM (Trusted Platform Module) built in.
How do you determine that on Linux?
How do you determine that on Linux?
How can a TPM provide as "real" random values as a quantum vacuum? (or cosmic rays, or anything else very much random in nature)
According to http://www.outflux.net/blog/archives/2009/10/22/tpm-as-rng/#..., TPM may even be using PRNGs. So you're right that the quality of output need not be better than what a modern OS already provides. (I am not aware of any case where a properly-used /dev/random failed to produce sufficient entropy. Note that doing something sensible at boot is part of "properly-used" here.)
It is possible to provide true randomness from a chip, but it's probably cheaper not to.
It is possible to provide true randomness from a chip, but it's probably cheaper not to.
The Intel TRNG design used for RdRand in Ivy Bridge has an impressively low transistor count so I don't think die-cost alone is the issue.
http://spectrum.ieee.org/semiconductors/processors/behind-in...
http://en.wikipedia.orghwiki/RdRand
I suspect they just didn't consider RNG quality as a big factor in the usefulness of the design. TPM pre-dates the recent advances in on-die TRNG design. Plus the chip tends to live on a low-speed bus on the system which makes it inherently a bit crappy as a place to keep a high-bandwidth application like an RNG.
http://spectrum.ieee.org/semiconductors/processors/behind-in...
http://en.wikipedia.orghwiki/RdRand
I suspect they just didn't consider RNG quality as a big factor in the usefulness of the design. TPM pre-dates the recent advances in on-die TRNG design. Plus the chip tends to live on a low-speed bus on the system which makes it inherently a bit crappy as a place to keep a high-bandwidth application like an RNG.
When did I say that it does? The randomness of the data is entirely down to the implementation.
Personally I'd trust the random data supplied by a third party web service less than the pseudo-random data that the software on my local machine produces. Especially if you mix in data from a TPM, especially if you mix in data from an Entropy Key as well.
Personally I'd trust the random data supplied by a third party web service less than the pseudo-random data that the software on my local machine produces. Especially if you mix in data from a TPM, especially if you mix in data from an Entropy Key as well.
Yes, this is a real issue. There's no way I'd get that past a financial regulator.
People, if you need random numbers for some application of yours, you shouldn't get them from their server, period. See alternatives in the posts here. The most modern example, apparently the following hardware instruction will be available in all next Intel processors: http://en.wikipedia.org/wiki/RdRand
The main article is just to appreciate the pure awesomeness of us being able to actually see something that's a result of quantum fluctuations! In vacuum!
The main article is just to appreciate the pure awesomeness of us being able to actually see something that's a result of quantum fluctuations! In vacuum!
FermiLab has been offering HotBits for a very long time:
http://www.fourmilab.ch/hotbits/
The site contains some interesting discussion around randomness and how their service works.
http://www.fourmilab.ch/hotbits/
The site contains some interesting discussion around randomness and how their service works.
Yes, but, uh ... Fourmilab is not Fermilab :-)
Ouch, that's a bit embarrassing...
Here's the original source:
http://150.203.48.55/index.php
And here's a pretty stream of random colors: http://150.203.48.55/RainCol.php
I wanted to post this too. Makes me feel special knowing that these are truly random.
What's wrong with the one in jwz's post? http://photonics.anu.edu.au/qoptics/Research/qrng.php
I've always wondered how many probabilistic algorithms would have been discovered which outperform deterministic ones if a high-bandwidth hardware RNG was available on every machine.
I understood that probabilistic algos only need something that "looks random enough" and are fine using PRNGs. True randomness doesn't buy you any improvement, as the algorithm is only counting on some of the properties of a random source.
The game is about speed -- probabilistic algorithm use PRNGs so heavily that their internal CPU demand may easily become the limiting factor.
so is the paper that describes this available for free anywhere? the trail of links from that page ends at http://apl.aip.org/resource/1/applab/v98/i23/p231103_s1?isAu...
update - similar article (free) from same people http://www.opticsinfobase.org/view_article.cfm?gotourl=http%... (source http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-19-21-2... )
[it seems like this is actually a big deal - they are getting huge throughput]
update - similar article (free) from same people http://www.opticsinfobase.org/view_article.cfm?gotourl=http%... (source http://www.opticsinfobase.org/oe/abstract.cfm?uri=oe-19-21-2... )
[it seems like this is actually a big deal - they are getting huge throughput]
oh noes. Not this poo again.
Save money, and do the following.
- Get a high res webcam - Open it up and remove the filters. - Hook up the device. - From time to time you'll see random white dots appear.
There you have your ultra cheap random number generator. Works best in high radiation environments.
- Get a high res webcam - Open it up and remove the filters. - Hook up the device. - From time to time you'll see random white dots appear.
There you have your ultra cheap random number generator. Works best in high radiation environments.
If you don't want to build your own generator, random.org provides a service like the ANU Quantum Random Number Server, i.e. provision of unique sets of true random numbers by http request, but generated from atmospheric noise. There's a convenience R package "random" I often use for this purpose. The articles on the website are quite edifying regarding the art and science of random numbers generally. Here's a link to the interface documentation: http://www.random.org/clients/http/
Do you know of any guides to do this? Are you aware of any free software to convert the image data into a usable stream of random data?
"Do you know of any guides to do this? Are you aware of any free software to convert the image data into a usable stream of random data?"
Look into LavaRnd [1][2][3][4].
[1] What is LavaRnd? http://www.lavarnd.org/what/index.html
[2] LavaRnd Process in Detail: http://www.lavarnd.org/what/process.html
[3] Construction of a LavaCan: http://www.lavarnd.org/what/can-details.html
[4] Source code (open source): http://www.lavarnd.org/download/index.html
Look into LavaRnd [1][2][3][4].
[1] What is LavaRnd? http://www.lavarnd.org/what/index.html
[2] LavaRnd Process in Detail: http://www.lavarnd.org/what/process.html
[3] Construction of a LavaCan: http://www.lavarnd.org/what/can-details.html
[4] Source code (open source): http://www.lavarnd.org/download/index.html
Looks like a good resource. Thanks.
"oh noes. Not this poo again. Save money, and do the following.
- Get a high res webcam - Open it up and remove the filters. - Hook up the device. - From time to time you'll see random white dots appear.
There you have your ultra cheap random number generator."
It would seem that this method would be muchmore expensive and time consuming than a simple web query. How precisely would this save me money?
It would seem that this method would be muchmore expensive and time consuming than a simple web query. How precisely would this save me money?
> How precisely would this save me money?
It's under your control. There's no chance of an attacker serving you skewed numbers, or of MITM, or etc etc.
That may be important. (Cryptography, for example.)
Some people don't care, they just need some random numbers. People doing modelling like a lot of random numbers that are random enough, but they don't care if that same set of numbers is available elsewhere, or if it's easily repeatable.
It's under your control. There's no chance of an attacker serving you skewed numbers, or of MITM, or etc etc.
That may be important. (Cryptography, for example.)
Some people don't care, they just need some random numbers. People doing modelling like a lot of random numbers that are random enough, but they don't care if that same set of numbers is available elsewhere, or if it's easily repeatable.
Pretty simple. You want random numbers for security.
If you want security, you should not trust a third party. Only yourself.
If you have to generate random numbers yourself; I just provided you a cheap solution.
That's random, but you still need to analyze the output for a while to normalize it.
Hasn't Via done something similar?
Since we have millions of transistors in our systems couldn't they (their collectors) be used to generate "truly" random numbers?
Since we have millions of transistors in our systems couldn't they (their collectors) be used to generate "truly" random numbers?
Perhaps they are only truly random until they are observed. After which they become deterministic. :)
That might sort of be funny if it weren't simply true, and not in any sort of funny-true way, either. Random numbers generated in the past are deterministic. Of course they are. What would non-deterministic-yet-already-generated numbers look like?
I dunno, it could be entangled...
You do not need exotic hardware to collect entropy to generate crypto keys. You need a correctly designed cryptographically secure PRNG which has had time to accumulate 100 or so bits of entropy (i.e., unknowability to the attacker). It's nice if it can be freshened every now and then with more entropy, but that really only should make a difference if you lose a backup tape and by some miracle the private key wasn't on there already. After it's warmed up, it should never need to block due to 'depleted entropy'. No one has ever broken such a CSPRNG. This is how the OpenSSL, OpenBSD, and FreeBSD /dev/randoms work.
On the other hand, what does tend to get broken regularly are overcomplicated and overengineered RNGs like this beast.
The one on the new Intel chips is handy, but it's largely overkill. We only need 100 bits now and then, not a steady rate at Gb/s (which can shut off abruptly when an attacker sharing your cloud hardware node decides to starve you of them). It's also not something that we can review for backdoors, so we should be reluctant to use it as the sole source of entropy.
A quantum vacuum random server is great like the way an internet-connected coffee pot is great. Fun, but not useful in production.