How odd is a cluster of plane crashes?(bbc.com)
bbc.com
How odd is a cluster of plane crashes?
http://www.bbc.com/news/magazine-28481060
3 comments
One of the first computer programs I ever wrote just put random dots on a screen and I remember looking at how 'lumpy' the results were. Intuitively most people assume that random distributions will be even distributions, but in fact they are usually full of clusters. Some mathematicians can reliably distinguish 'random' sequences created by people from truly random ones because people produce distributions that are too smoothed out. 'I won't put a dot here because it's too close to that other dot', but a truly random distribution doesn't care where previous dots were and won't avoid the conjunction.
The iPod shuffle algo was an example of this. It just didn't feel "random" enough for people to have a truly random generator.
I would think about this all the time when I got a shuffle in high school. After listening to it for a while, I could really predict the next track that would come on even in shuffle mode.
Isn't that also why it is shuffle and not random? When shuffling, you don't risk getting the same song twice in a row.
People are seldom precise when they talk about randomness.
For example, you could mean: picking tracks with `putting back'. You could shuffle tracks like you shuffle a deck of cards, ie picking without putting back. It's still random, but with a different distribution.
Even picking the first track with 90% probability, and the rest with some other smaller probability is random. It's just not a uniform distribution.
For example, you could mean: picking tracks with `putting back'. You could shuffle tracks like you shuffle a deck of cards, ie picking without putting back. It's still random, but with a different distribution.
Even picking the first track with 90% probability, and the rest with some other smaller probability is random. It's just not a uniform distribution.
I do wonder if that is down to sample size. For example, I wrote a program to simulate the roll of a dice. There is a one in 6 chance of getting any one number. But roll it 6 million times, you'll be very close to have each number 1 million times.
So I would anticipate the clumping to even out as the sample size increases. Or maybe the evening out is a large collection of clumps. Anyone know if this is the correct way to think about it?
So I would anticipate the clumping to even out as the sample size increases. Or maybe the evening out is a large collection of clumps. Anyone know if this is the correct way to think about it?
> So I would anticipate the clumping to even out as the sample size increases.
Clumping won't even out. A useful way of thinking about it is in terms of the distance between samples (e.g. a radial distribution function). If you don't have clumping, that distribution of distances is decidedly non-random and has peaks. Additionally, there's no specific length-scale for the clumping to happen on, it's essentially fractal in nature.
(Happy to be corrected if I'm wrong on this.)
Clumping won't even out. A useful way of thinking about it is in terms of the distance between samples (e.g. a radial distribution function). If you don't have clumping, that distribution of distances is decidedly non-random and has peaks. Additionally, there's no specific length-scale for the clumping to happen on, it's essentially fractal in nature.
(Happy to be corrected if I'm wrong on this.)
I think in order to pass any judgement on the correctness, you'd have to make your remarks more precise.
Are you talking about a random distribution of points on a line / square, or about rolling dice?
Are you talking about a random distribution of points on a line / square, or about rolling dice?
> Are you talking about a random distribution of points on a line / square, or about rolling dice?
Right, exactly. I made my comment in reply to the grandparent problem talking about dots on a screen. For a discrete distribution like a dice roll in the parent post it'd be a different matter. (I was going to edit my comment to clarify, but the edit window had expired.) Which serves to illustrate why it's important to be precise when talking about what distribution you're sampling.
Also, trying to run this experiment in practice is also problem even for dots on a screen, because eventually the finite display size of your dots will become an issue, as will floating point arithmetic (since you only really have a finite number of floats in a given range).
Right, exactly. I made my comment in reply to the grandparent problem talking about dots on a screen. For a discrete distribution like a dice roll in the parent post it'd be a different matter. (I was going to edit my comment to clarify, but the edit window had expired.) Which serves to illustrate why it's important to be precise when talking about what distribution you're sampling.
Also, trying to run this experiment in practice is also problem even for dots on a screen, because eventually the finite display size of your dots will become an issue, as will floating point arithmetic (since you only really have a finite number of floats in a given range).
Thanks for clarifying.
> Also, trying to run this experiment in practice is also problem even for dots on a screen, because eventually the finite display size of your dots will become an issue, as will floating point arithmetic (since you only really have a finite number of floats in a given range).
Yes, but that's not a problem for the theoretical analysis of fractal dimension.
Uniformly placed dots do have a characteristic density and a relatively simple distribution of nearest neighbour distances. Even the clumpiness doesn't lead to scale invariant lumps.
The lumps get absolutely bigger on bigger scales, but relatively smaller and smoother.
It would be useful to do the analysis properly, though.
> Also, trying to run this experiment in practice is also problem even for dots on a screen, because eventually the finite display size of your dots will become an issue, as will floating point arithmetic (since you only really have a finite number of floats in a given range).
Yes, but that's not a problem for the theoretical analysis of fractal dimension.
Uniformly placed dots do have a characteristic density and a relatively simple distribution of nearest neighbour distances. Even the clumpiness doesn't lead to scale invariant lumps.
The lumps get absolutely bigger on bigger scales, but relatively smaller and smoother.
It would be useful to do the analysis properly, though.
What you're describing for the dice is called the law of large numbers : https://en.m.wikipedia.org/wiki/Law_of_large_numbers
It would probably true for that too but as another comment says, we don't want to have an increased sample size here!
It would probably true for that too but as another comment says, we don't want to have an increased sample size here!
> So I would anticipate the clumping to even out as the sample size increases.
Are you saying we need to crash more planes? That seems a little harsh just to get crashes coming at a consistent rate. ;-)
Are you saying we need to crash more planes? That seems a little harsh just to get crashes coming at a consistent rate. ;-)
I'd prefer to simulate this with dots on a screen rather than with planes ;-)
Well depends of what you mean by very close.
It's the relative numbers that approaches the true distribution, not the absolute numbers.
Do the experiment you decribe and look at the numbers.
Do the experiment you decribe and look at the numbers.
This is what I mean:
So when you look at the clumping on random dots on a screen, given a large enough sample size, I would anticipate that the probability of keeping a clumped visual distribution would become quite improbable.
var results = [0,0,0,0,0,0];
for (var i = 0; i < 6000000; i += 1) {
results[Math.floor((Math.random() * 6))] += 1;
}
console.log(results);
[1002140, 999355, 1000009, 1000401, 1000014, 998081]
So when you look at the clumping on random dots on a screen, given a large enough sample size, I would anticipate that the probability of keeping a clumped visual distribution would become quite improbable.
I think the clumping is more about how many consecutive same numbers there are in the array.
For example, a naive me would generate: [3,4,5,1,2,4,2,1,2,6] whereas a true random distribution might generate: [3,4,4,4,4,1,2,2,2]. When you generate 6 million samples you would indeed have about 1 million per number. However you would still have subsequences of the same number.
For example, a naive me would generate: [3,4,5,1,2,4,2,1,2,6] whereas a true random distribution might generate: [3,4,4,4,4,1,2,2,2]. When you generate 6 million samples you would indeed have about 1 million per number. However you would still have subsequences of the same number.
The problem is not when you have much more samples than slots, in that case you should get "almost" the same number of hits in each slot as you measured.
The problem is when you get less samples than slots. For example if you roll 3 dices, you should get "1/2" hits for each result. So the question is: How probable is that one result is repeated? (i.e. you get two equal dices out of the three)
[spoiler alert]
All equal: 3%
Two equal and one different: 42%
Three different: 55%
"Intuitively", it's expected that with a uniform distribution you get something like 1% + 9% + 90% instead (this numbers are completely made up). Most people don't expect that the probability of getting a repeated dice is so close to the probability of getting all of them different.
The problem is when you get less samples than slots. For example if you roll 3 dices, you should get "1/2" hits for each result. So the question is: How probable is that one result is repeated? (i.e. you get two equal dices out of the three)
[spoiler alert]
All equal: 3%
Two equal and one different: 42%
Three different: 55%
"Intuitively", it's expected that with a uniform distribution you get something like 1% + 9% + 90% instead (this numbers are completely made up). Most people don't expect that the probability of getting a repeated dice is so close to the probability of getting all of them different.
[deleted]
I don't disagree at all and I remember seeing studies or articles writing about that phenomenon.
Just wanted to add that a few years ago (when you wrote your first computer program) a randomly selected random number generator was more likely to be less random than a randomly selected random number generator today.
Just wanted to add that a few years ago (when you wrote your first computer program) a randomly selected random number generator was more likely to be less random than a randomly selected random number generator today.
The method described in the article on how the statistician approached the problem is very similar to the approach used in the birthday problem[1].
[1] - https://en.wikipedia.org/wiki/Birthday_problem
[1] - https://en.wikipedia.org/wiki/Birthday_problem
Early random number generation was terrible. You wouldn't get clumps / no clumps, you'd get fuzzy straight lines.
Approximately 35 years ago :)
An other great example is how random is a terrible algorithm for dithering.
https://en.wikipedia.org/wiki/Dither#Algorithms
https://en.wikipedia.org/wiki/Dither#Algorithms
Perhaps just an artifact of a weak PRNG or not truly uniform distribution?
No. A truly uniform random distribution indeed has "clumps".
And a poor PRNG or poor implementation of a uniform distribution (both extremely common, for example: rand() % max) has more clumps, my question is still valid.
Edit: I wrote a program[1] to fill random pixels based on rand/modulo and mersenne twister/uniform distribution and I could not tell the difference in the generates images.
[1] https://gist.github.com/anonymous/9c726ea636f06900ba3a
Edit: I wrote a program[1] to fill random pixels based on rand/modulo and mersenne twister/uniform distribution and I could not tell the difference in the generates images.
[1] https://gist.github.com/anonymous/9c726ea636f06900ba3a
Great point. Here's a visualization of what you're talking about:
http://bl.ocks.org/mbostock/fe3f75700e70416e37cd
>Uniform random is pretty terrible. There is both severe under- and oversampling: many samples are densely-packed, even overlapping, leading to large empty areas. (Uniform random sampling also represents the lower bound of quality for the best-candidate algorithm, as when the number of candidates per sample is set to one.)
Taken from: http://bost.ocks.org/mike/algorithms/
http://bl.ocks.org/mbostock/fe3f75700e70416e37cd
>Uniform random is pretty terrible. There is both severe under- and oversampling: many samples are densely-packed, even overlapping, leading to large empty areas. (Uniform random sampling also represents the lower bound of quality for the best-candidate algorithm, as when the number of candidates per sample is set to one.)
Taken from: http://bost.ocks.org/mike/algorithms/
"If there is a crash on 1 August, the chance that the next crash occurs one day later on 2 August is 1/365. But the chance the next crash is on 3 August is (364/365) x (1/365), because the next crash occurs on 3 August only if there is no crash on 2 August."
Is this not the gambler's fallacy? Shouldn't the odds be unaffected by a previous crash?
Is this not the gambler's fallacy? Shouldn't the odds be unaffected by a previous crash?
You're right that the probability that there is a crash on 3 August is unaffected by a potential crash on the 2 of August, but the chance that the crash on the 3 of August is the next crash (after the 1st of August) is affected.
Also, pedantically, a plane that crashes on the 2nd August isn't going to crash again on the 3rd.
We can reduce the rate of crashes by taking planes out of the sky, and one way to do that is to crash them.
We can reduce the rate of crashes by taking planes out of the sky, and one way to do that is to crash them.
Unless there are reserve planes that come immediately on-line to fill the gap (or the remaining planes do more fly time).
In probability you have take attention to the very details. The statements "the next crash occurs on 3 August" and "there is a crash on 3 August" are not the same. Only the latter is independent from a crash on 2 August.
In this case I would agree with you. I don't understand the reasoning for the last sentence at all.
The focus is on being the next immediate crash. If you are asking whether there is a crash on 3 August, it is the unaffected odds of 1/365. But if you are asking whether the (first) next crash after 1 August is on 3 August, both the following must happen:
a) there is no crash on 2 August
b) there is a crash on 3 August
The gambler fallacy would be "I have seen 9 days without crash, on the 10th day there is higher odds of crash". But you are not asking this, you are asking: "What is the chance that in the next 10 days, exactly first 9 days would be no-crash and on the 10th day there is a crash".
EDIT: Also, another view:
If you have seen 9 blacks in a row, there is an even chance 1/2 (roulette without zero) that the next will be black or red. Because in all sequences of length 10 starting with 9 blacks (this is given, you observed this), there is equal number of those with 9 consecutive blacks and a red and of those with 9 consecutive blacks and a black. This is gambler fallacy, believing that after repeated colour, you have higher chance of the other colour.
Here, you are asking: how many of all sequences of length 10 have exactly 9 blacks and 1 red. And there are all kinds of sequences with 1 red in first 9 places, 2 reds, 3 reds, etc., so the odds of observing the 9 blacks in a row in the first place before asking about the 10th spin is lower, but it does not change the odds of the 10th spin itself.
The gambler fallacy would be "I have seen 9 days without crash, on the 10th day there is higher odds of crash". But you are not asking this, you are asking: "What is the chance that in the next 10 days, exactly first 9 days would be no-crash and on the 10th day there is a crash".
EDIT: Also, another view:
If you have seen 9 blacks in a row, there is an even chance 1/2 (roulette without zero) that the next will be black or red. Because in all sequences of length 10 starting with 9 blacks (this is given, you observed this), there is equal number of those with 9 consecutive blacks and a red and of those with 9 consecutive blacks and a black. This is gambler fallacy, believing that after repeated colour, you have higher chance of the other colour.
Here, you are asking: how many of all sequences of length 10 have exactly 9 blacks and 1 red. And there are all kinds of sequences with 1 red in first 9 places, 2 reds, 3 reds, etc., so the odds of observing the 9 blacks in a row in the first place before asking about the 10th spin is lower, but it does not change the odds of the 10th spin itself.
Super helpful! Thanks!
Pleases mark as 2014 ... I also believe this is a repeat but can't seem to find the original.
Here's the previous discussion: https://news.ycombinator.com/item?id=8085254
I had tried the Algolia search box from my tablet and got no results. Using Google (constrained to HN) brought up today's discussion and the one from last year.
site:news.ycombinator.com plane crash cluster
I had tried the Algolia search box from my tablet and got no results. Using Google (constrained to HN) brought up today's discussion and the one from last year.
site:news.ycombinator.com plane crash cluster