Ask HN: How do SpaceX/NASA/other maintain high code quality?
What is the state of the art test frameworks and/or patterns that mission critical companies follow nowadays? It seems that SpaceX uses C++ for most everything, but I haven't been able to find anything about how they or other modern mission-critical software companies maintain, what I assume is, well-tested, near bug-free code?
9 comments
Not directly on topic, but close, the simulation and environment is touched on:
http://space.stackexchange.com/questions/9243/what-computer-...
The above link references the AMA done by SpaceX engineers too: https://www.reddit.com/comments/1853ap
http://space.stackexchange.com/questions/9243/what-computer-...
The above link references the AMA done by SpaceX engineers too: https://www.reddit.com/comments/1853ap
Required reading: http://www.fastcompany.com/28121/they-write-right-stuff ("the on-board shuttle group produces grown-up software, and the way they do it is by being grown-ups.")
I was reading about the new horizons project, and for the spacecraft it seems they tested using simulation. The mission was thought about for years before the software was made. Then the spacecraft flew for a few years which gave ample time to simulate. Other mission critical projects simply have a room full of people scrutinizing ever line of code.
Here's the JPL Java Coding Standard.
http://lars-lab.jpl.nasa.gov/jpl-java-standard.pdf
"The primary purpose of the standard is to help Java programmers reduce the probability of run-time errors in their programs."
"The primary purpose of the standard is to help Java programmers reduce the probability of run-time errors in their programs."
I've heard that it's a common strategy for mission critical software to have a "bad" dev team of similar or equal size that works against the "good" team to break what ever they're building.
I don't know whether it is common to have one, but the original was IBM's black team: http://www.t3.org/tangledwebs/07/tw0706.html
As a former software engineer at SpaceX I'm rather interested in why you think the solution is a "state of the art test framework and/or pattern".
I am lead to believe that the hiring process at SpaceX is tough and that therefore the caliber of recruits is high. Couple that with SpaceX accomplishing major space breakthroughs in, what when compared to NASA, is a fraction of the time, would lead me to assume their processes must have some state-of-the-art procedures.
Just because a company has a tough hiring process and some accomplishments does not mean they have state of the art processes or bug free code.
It's the process, not the frameworks.
I can't speak about SpaceX, but with a number of prior/current NASA contractors it is pretty mundane. The key isn't some awesome framework or "new" technology. In fact, you'd probably be dumbfounded with the technology restrictions and how little they are using current technology. Overall their success comes from process, methodology and time. In addition, most of the time you are engineering for failure, so when failure happens recovery is possible. You can read about the flight computers aboard the shuttle (and Apollo) and how they handled things with them, there are some really interesting articles about it.
In more specific, I'd say 80% of the success comes from fairly rigid processes, including the coding standards and testing methodology. Like with C/C++ standards, you'll find a lot of restrictions on what you can use in the language, how you can use it etc. Here is the JPL's C standards, around flight and embedded type projects (they happen to publish most of their standards) http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
And traditionally there has been significant separation of teams, e.g. System Engineering doing requirements analysis and development, Software Engineers building the software and Testing Engineers writing software and procedures to make sure software hits all the requirements and it doesn't fail.
At least from what I know and have experienced you also will not see flight or "critical" software changed and flown (or actively utilized) within any short period of time. As time is a major factor in the testing/simulation regime to make sure things work properly.
My guess is that even at SpaceX, critical software components in flight today have been completed and tested for months and months before it hits a real flight. Doesn't mean they may not fly it in a passive mode to see what decisions it would make, but it wouldn't be in control. I am sure non-critical components and ground based software has a slightly less rigid process, but probably still fairly tough.
In more specific, I'd say 80% of the success comes from fairly rigid processes, including the coding standards and testing methodology. Like with C/C++ standards, you'll find a lot of restrictions on what you can use in the language, how you can use it etc. Here is the JPL's C standards, around flight and embedded type projects (they happen to publish most of their standards) http://lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
And traditionally there has been significant separation of teams, e.g. System Engineering doing requirements analysis and development, Software Engineers building the software and Testing Engineers writing software and procedures to make sure software hits all the requirements and it doesn't fail.
At least from what I know and have experienced you also will not see flight or "critical" software changed and flown (or actively utilized) within any short period of time. As time is a major factor in the testing/simulation regime to make sure things work properly.
My guess is that even at SpaceX, critical software components in flight today have been completed and tested for months and months before it hits a real flight. Doesn't mean they may not fly it in a passive mode to see what decisions it would make, but it wouldn't be in control. I am sure non-critical components and ground based software has a slightly less rigid process, but probably still fairly tough.