Air traffic system failure caused by computer memory shortage(reuters.com)
reuters.com
Air traffic system failure caused by computer memory shortage
http://www.reuters.com/article/2014/05/12/us-airtraffic-bug-exclusive-idUSBREA4B02320140512
9 comments
What's the point of flying the most stealthy plane on earth if you have to register a flight plan ?
I thought such a plane would be 100% autonomous and be able to "see" and avoid other planes.
I thought such a plane would be 100% autonomous and be able to "see" and avoid other planes.
> flying the most stealthy plane
A U2 is not at all stealthy. You must be thinking of some other plane.
> if you have to register a flight plan
It's not a military operation in a foreign country, it's a routine flight in the home country.
> I thought such a plane would be 100% autonomous
The U2 was made long before such computers existed. They do have drones now.
> and be able to "see" and avoid other planes.
It wasn't the plane with the problem, it was ground control.
This must be a world record: A faulty assumption in every single sentence!
A U2 is not at all stealthy. You must be thinking of some other plane.
> if you have to register a flight plan
It's not a military operation in a foreign country, it's a routine flight in the home country.
> I thought such a plane would be 100% autonomous
The U2 was made long before such computers existed. They do have drones now.
> and be able to "see" and avoid other planes.
It wasn't the plane with the problem, it was ground control.
This must be a world record: A faulty assumption in every single sentence!
"It's not a military operation in a foreign country, it's a routine flight in the home country"
Is it worth asking, at all, why we're flying a military reconnaissance aircraft back-and-forth over Los Angeles?
Is it worth asking, at all, why we're flying a military reconnaissance aircraft back-and-forth over Los Angeles?
Atmospheric studies? Training in commercial environments? To see what happens? Because they can? Sekrit stuff?
If there's an actual military mission for any military plane that requires it to fly over or through a commercial area, wouldn't you prefer that pilots and ground operators already know how to do that?
If there's an actual military mission for any military plane that requires it to fly over or through a commercial area, wouldn't you prefer that pilots and ground operators already know how to do that?
I effectively mistook the U2 and B2.
For 100% autonomous flight, I wasn't thinking no pilot, but about being able to navigate without any assistance from other radars
For 100% autonomous flight, I wasn't thinking no pilot, but about being able to navigate without any assistance from other radars
A good idea when you consider that it could be mistaken for some bad guys plane, shot down and probably cause WWIII (not that long shot - the US and USSR were at a few times very close to annihilation due to glitches in their own software)
If you don't register military flights on civil ATC systems, you run the risk of this -http://en.wikipedia.org/wiki/Hughes_Airwest_Flight_706
Sounds more like a bug rather than a lack of memory to me ...
The flight plan did not contain an altitude for the flight, one of the sources said. While a controller entered the usual altitude for a U-2 plane - about 60,000 feet - the system began to consider all altitudes between ground level and infinity.
The conflict generated error messages and caused the system to begin cycling through restarts.
The flight plan did not contain an altitude for the flight, one of the sources said. While a controller entered the usual altitude for a U-2 plane - about 60,000 feet - the system began to consider all altitudes between ground level and infinity.
The conflict generated error messages and caused the system to begin cycling through restarts.
They didn't have enough memory to cope with an infinite number of inputs.
Obviously a hardware problem...
Obviously a hardware problem...
is there a plane that flies at infinite altitude?
Is there an aircraft of concern to the FAA that flies above FL1000? One or two historic flights have been that high, so raise the ceiling to 150,000'
still not infinite.
Is there an aircraft of concern to the FAA that flies above FL1000? One or two historic flights have been that high, so raise the ceiling to 150,000'
still not infinite.
Yeah, let's hardcode a maximum altitude somewhere deep in the code and be completely surprised when, a couple of years in the future, newer, higher-flying planes/shuttles disappear from the system.
Well, considering that altitudes above FL1000 have been, for the last 60-odd years, considered to be outside the FAA's jurisdiction... what is the problem exactly?
One presumes they still need to know the location of those flights to deal with them descending into the range of altitudes used by commercial aircraft.
No commercial aircraft operate at such altitudes. No, seriously: you run out of air for air-breathing engines above FL600 (the SR-71 flew at FL800 very fast indeed partly because it needed to go like a bat out of hell to keep enough airflow through its engines: and because it was able to go that fast due to there not being much air up there to produce drag), and the turbofans used on commercial airliners rapidly run out of wheeze above FL400.
The only civil aircraft that ever flew above FL500 were Concorde and the Tu-144, neither of which are in service. But they didn't fly at supersonic speed and high altitude in crowded airspace -- simply because getting an SST up to altitude and speed takes several minutes, a couple of hundred miles, and a prodigious amount of fuel. To say nothing of the sonic boom issue, which basically limited supercruise to over-ocean flight.
While there's been a lot of hot air about supersonic bizjets over the past two decades, so far nothing's escaped from the CAD package much less gone into flight testing.
So what you're asking for is for an FAA system to be retrofitted to handle exotic military requirements, future space shuttles, and a class of commercial aircraft that would not operate in that flight regime within a couple of hundred miles of any airport.
The only civil aircraft that ever flew above FL500 were Concorde and the Tu-144, neither of which are in service. But they didn't fly at supersonic speed and high altitude in crowded airspace -- simply because getting an SST up to altitude and speed takes several minutes, a couple of hundred miles, and a prodigious amount of fuel. To say nothing of the sonic boom issue, which basically limited supercruise to over-ocean flight.
While there's been a lot of hot air about supersonic bizjets over the past two decades, so far nothing's escaped from the CAD package much less gone into flight testing.
So what you're asking for is for an FAA system to be retrofitted to handle exotic military requirements, future space shuttles, and a class of commercial aircraft that would not operate in that flight regime within a couple of hundred miles of any airport.
Funnily enough, I understood UK ATC service was provided to FIR FL245, with basic service UIR FL660.
They worry over them after takeoff until they ascend beyond range and once they descend into range.
I'm assuming they don't worry over satellites in orbit but they might have some concern if one decides to come back home.
I'm assuming they don't worry over satellites in orbit but they might have some concern if one decides to come back home.
No one said anything about burying it, it can be at the top of the main configuration file if you insist. The basic problem is that one quantity was limited (memory), without dealing with another (altitude). It makes perfect sense to limit both, while you're at it. But of course most important of all, test and study the effects of going over those limits ...
I think hudibras was been sarcastic
Whereas, of course, he should have been hudibrastic [0]:
A spyplane quietly in flight
At angels great, not infinite
Of course results in filling chips
And vex'd controllers passing slips
Also how dare they not to have cpu that can run infinite cycle in under 2 seconds.
unsigned short int altitude;The flight was at 60,000' - You would have to look far and wide for a platform in which "short int" wasn't two bytes.
For example, Here is what the compiler for the 8-bit ATMEL Atmega 328 in an arduino does (on a platform with 32KB for program text and 2KB for RAM.)
For example, Here is what the compiler for the 8-bit ATMEL Atmega 328 in an arduino does (on a platform with 32KB for program text and 2KB for RAM.)
sizeof(bool) == 1
sizeof(short) == 2
sizeof(int) == 2
sizeof(long) == 4
sizeof(long long) == 8
sizeof(void *) == 2The flight was at about 60,000 feet , so it could easily have been at slightly more than 216 feet.
Maybe it attempted to allocate a huge structure for each foot of altitude.
This scenario wasn't unit/functional tested then.
It sounds like the altitude existence assumption caused cascading errors which in turn exhausted the system memory.
That's not what the article said. They said that since it didn't know the true altitude the computer assumed the plane could be at every altitude, and checked every single possible one.
This was made worse by a flight plan that had the plane going in and out of the area, so there were a tremendous number of permutations of possible flight paths.
The computer attempted to allocate memory to calculate every one of them and ran out of memory.
This was made worse by a flight plan that had the plane going in and out of the area, so there were a tremendous number of permutations of possible flight paths.
The computer attempted to allocate memory to calculate every one of them and ran out of memory.
said Dan Kaminsky, co-founder of the White Ops security firm and an expert in attacks based on over-filling areas of computer memory.
That sounds more like buffer overrun than OOM. One is a Heartbleed-class vulnerability, the other is a pretty straightforward DoS situation. But of course why would we expect Reuters to be able to distinguish between the two...
That sounds more like buffer overrun than OOM. One is a Heartbleed-class vulnerability, the other is a pretty straightforward DoS situation. But of course why would we expect Reuters to be able to distinguish between the two...
That's irrelevant to the rest of the article, it's only how his expertise is characterized.
Having worked on ATC, this really does sound like a software bug. Fixing such a bug on a delivered system would require an immense amount of costly testing. Adding memory is the cheap option.
The other thing to consider is that some of these systems are rather old and could easily have less than a gig of ram.
The other thing to consider is that some of these systems are rather old and could easily have less than a gig of ram.
That sounds very worrying. The fact that this issue occurs shows that they didn't prepare for OOM conditions, and another issue could easily cause a different OOM condition they didn't prepare for. It's not inconceivable that there could be an issue which gobbles up an amount of memory you throw at it. It is a tragic irony that the fact that testing is costly may be the very reason this condition was not tested for.
I can no longer edit. "an amount" should be "any amount".
> Adding memory is the cheap option
Only in the sense of having a broken system, putting in place some procedural constraints and then praying that everyone follows the rules.
As you say, it's a software bug. Until it's been 100% characterised and tested this leaves a huge question mark hanging over the system. Adding memory is a band aid solution.
Only in the sense of having a broken system, putting in place some procedural constraints and then praying that everyone follows the rules.
As you say, it's a software bug. Until it's been 100% characterised and tested this leaves a huge question mark hanging over the system. Adding memory is a band aid solution.
I would guess that it was some form of an numeric overflow, signed/unsigned issue or out of bound array access, because the only abnormal thing here is U2's altitude.
Have you read the article? The cause is explained quite explicitly, and it was the very absence of the altitude not any of the things you mention.
> very absence of the altitude
And? Consider what this means exactly and how come the system saw the plane but didn't record the altitude.
And? Consider what this means exactly and how come the system saw the plane but didn't record the altitude.
This is part of a system the FAA has been trying to replace (unsuccessfully) since at least 1990: http://www.ainonline.com/aviation-news/aviation-internationa...
(The AAS often appears in software engineering classes as an example of a classic "train wreck" project)
(The AAS often appears in software engineering classes as an example of a classic "train wreck" project)
Lack of constraint leads to infinite search space. Sounds like constraint programming.
So much talk about Snowden, and then a newspaper reveals how to trigger a software bug than can shut down the entire air traffic control system for a large segment of the US. This is a genuinely damaging revelation, even if some critical details are lacking. Curious that the government is not up in arms about it.
Such systems should always preallocate fixed-size heapsfor each subsystem, each of which can then dynamically parcel out pieces of their own heaps if necessary. This helps limits the damage of any memory leaks in the system.