Lockheed Reassigns Workers to Fix F-35 Software(military.com)
military.com
Lockheed Reassigns Workers to Fix F-35 Software
http://www.military.com/daily-news/2013/06/19/lockheed-reassigns-workers-to-fix-f35-software.html
10 comments
For anyone who's unfamiliar with the term "Mythical Man Month," it's the title of a book which argues, amongst other things, that assigning more engineers to a late software project will make it even later.
The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.
Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.
I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.
I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.
The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.
Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.
I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.
I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.
Frequently aerospace companies will have a limited number of licenses for various pieces of software. C++ as a language obviously doesn't have this limitation, but a limited number of licenses for various custom test suites and third party software (e.g. Matlab) is frequently a bottleneck. If this is a bottleneck, it makes sense to have these licenses in use 24/7.
Purchasing more licenses is always an option, but usually not a quick one.
Purchasing more licenses is always an option, but usually not a quick one.
Maybe it's really obvious, I mean it should be, but Lockheed clearly isn't much of a software company.
Office space and third party software licenses mandating a 24 hour work cycle for the most expensive defense project of all time?! Maybe it should not be surprising. Their peers seem to just appropriate rights and constitutional interpretation to meet their needs, can't these guys just appropriate more matlab or whatever compiler it is licenses? Just seems like it is being treated like a manufacturing problem.
This same class of folks who are making these decisions are all contracting for NSA and have access to your data to keep us all safe...
Office space and third party software licenses mandating a 24 hour work cycle for the most expensive defense project of all time?! Maybe it should not be surprising. Their peers seem to just appropriate rights and constitutional interpretation to meet their needs, can't these guys just appropriate more matlab or whatever compiler it is licenses? Just seems like it is being treated like a manufacturing problem.
This same class of folks who are making these decisions are all contracting for NSA and have access to your data to keep us all safe...
Good point. It does make me wonder though: Does the time and money cost of purchasing more licenses outweigh the effect on employee morale resulting from night shifts?
Fascinating, thanks for the insight.
> I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway?
Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).
Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).
So they're running low on office space? That's a bottleneck for the F-35?
Why not? It's not like they can just go down to the local HackerNinjaRockStar DojoSpace and rent some Ikea standing desks. At Apple, for instance, there are sealed buildings and security requirements that impose a cost on expansion above espresso machines and foosball tables -- and they're not required to meet DoD legal standards.
This is probably true. Still, it seems like a red flag to me that on a project like the F-35, which presumably has adequate funding for office space and other such things, their only solution for providing offices for their own employees is night shifts.
There's a very good reason not to have engineers work nights. While some are undoubtedly night owls, many (most?) will have lives and families which will suffer from night shifts. That accelerates burnout and increases resentment--both of which are usually running high in a troubled project anyway.
There's a very good reason not to have engineers work nights. While some are undoubtedly night owls, many (most?) will have lives and families which will suffer from night shifts. That accelerates burnout and increases resentment--both of which are usually running high in a troubled project anyway.
Oh, the project seems certain to be a total clusterfuck; it's just not 100% surprising that they could be gated on physical space.
The F-35 has multiple variants, one of which is a STOVL (Short Takeoff and Vertical Landing) aircraft with a vertical flight system. The amount of code necessary to deal with that is likely a large part of the difference in LOC, not to mention the more complicated systems for weapons, radar, communications, etc...
In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.
In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.
>expanded their pool of potential programmers
Is that true? I'd think that the main requirements would be experience, intelligence, security clearance, probably familiarity with avionics.
Any good programmer worth working on such a project should be able to switch languages.
I'd hope it's not like some corporate web site that just needs the programming equivalent of warm bodies.
Is that true? I'd think that the main requirements would be experience, intelligence, security clearance, probably familiarity with avionics.
Any good programmer worth working on such a project should be able to switch languages.
I'd hope it's not like some corporate web site that just needs the programming equivalent of warm bodies.
"Should" does not imply "wants to."
I could switch to COBOL in probably a couple weeks, but you'd have to pay me multiples of my current salary to do so for any length of time.
I could switch to COBOL in probably a couple weeks, but you'd have to pay me multiples of my current salary to do so for any length of time.
Considering the number of job postings I've seen where they want N+ years of experience, I'd say it's easier to find that in C++ than Ada if they do that.
> system that is designed to never crash
Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."
Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.
Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."
Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.
sometimes do so automatically without physically crashing the jet
I hope by "sometimes" you mean "almost every time!"
I hope by "sometimes" you mean "almost every time!"
Sorry, bad grammar. I meant that sometimes they reboot automatically. The systems are designed to always reboot without physically crashing the jet, but sometimes the pilot has to command the reboot themselves.
> Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.
If that's the case, why not use something like Erlang?
If that's the case, why not use something like Erlang?
Because you don't want to be in a high performance jet when the VM decides to do garbage collection during a critical maneuver.
[deleted]
I think the difference in LOC count is likely due to the much more complicated avionics in the F-35 than the technology stack, as compared with older aircraft.
> a system that is designed to never crash
I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).
> choice of using C++
Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?
I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).
> choice of using C++
Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?
[deleted]
This website has some information about serious bugs. There are a variety of languages being used, by different industries, with either expensive war machines or expensive civil aeroplanes full of passengers or impossible to service space-craft.
(http://www5.in.tum.de/~huckle/bugse.html)
(http://www5.in.tum.de/~huckle/bugse.html)
Any one who's worked at Martin Marietta in the past (now part of Lockheed) will recognize this. Martin had a long, long history of doing this sort of thing for projects in trouble.
When they get past this (and they will, they have the DoD over a barrel, they just have to not gloat too publicly) the fallout to the F-35 will last as long as F-35s are in operation. In the early 90s, Martin people hand-assembled Titan rocket guidance and control software because some time in the distant past, the assembler had some bugs. The F-35 software group will live with whatever horrific "best practices" the current group of reassigned workers come up with.
When they get past this (and they will, they have the DoD over a barrel, they just have to not gloat too publicly) the fallout to the F-35 will last as long as F-35s are in operation. In the early 90s, Martin people hand-assembled Titan rocket guidance and control software because some time in the distant past, the assembler had some bugs. The F-35 software group will live with whatever horrific "best practices" the current group of reassigned workers come up with.
> In the early 90s, Martin people hand-assembled Titan rocket guidance and control software
What the fuck?
What the fuck?
To be completely truthful, they compared binary dumps of hand-assembled, and machine-assembled software, to ensure that everything was exactly the same. But yes, they hand-assembled some weird instruction set. I can't recall asking if they assembled to ASCII hexadecimal word (24-bit) representations, or to something else.
I believe this was for "Commerical Titan", a short-lived, Titan 34D variant. Four flights, one of which left Intelsat VI in the wrong orbit. That wasn't software, but plugging the wrong cable in.
I believe this was for "Commerical Titan", a short-lived, Titan 34D variant. Four flights, one of which left Intelsat VI in the wrong orbit. That wasn't software, but plugging the wrong cable in.
I'm hoping this will help justify limiting purchases of the F-35 and replacing it sooner with cheaper, more expendable, but roughly equally effective UCAVs and better missiles.
Common sense couldn't kill the utterly superfluous F-22; the F-35 has way more stakeholders, so don't count on it going away any time soon.
Adding manpower to a late software project makes it later.
- Frederick P. Brooks, Jr. "The Mythical Man-Month".
- Frederick P. Brooks, Jr. "The Mythical Man-Month".
>Nine women can't make a baby in one month.
Not only have they added 200 engineers from other projects which will now suffer delays and quality issues, both also "They're running 24-7, what we call lights out." which will ensure massive burn out. Brilliant...
If you time it right, you don't have to hit burnout. In my view, "timing it right" basically means completing the project before burnout. A successful completion followed by a little R&R goes an incredibly long way to combating burnout.
And I suppose they will use their awesome estimation powers that have led them into this mess, to time it right...
Exactly ;)
Why is it called "lights out" if the offices are occupied with coders non-stop?
Coder caves; no lights, especially not natural; time of day or passing is not in your peripheral.
Sounds lovely. Is that really a thing? I guess if you paid me enough money, I'd put up with that, but mostly, this makes me glad I declined their job offer several years ago.
"There’s nothing wrong with flying before we buy. In fact, most of us test drive cars before we [buy]"
Except that if we custom-order a car, the manufacturer isn't legally prohibited from selling it to someone else if we end up refusing to buy it.
Except that if we custom-order a car, the manufacturer isn't legally prohibited from selling it to someone else if we end up refusing to buy it.
There analogy is not good, but I would also assume that lockheed won a competitive bidding process to get the contract as well. They promised to build it at a certain price and in a certain time.
Yup, Lockheed's X-35 beat out a (horrendously ugly) Boeing airplane to win the contract, but my understanding was that the initial orders were cost-plus. That's pretty typical for large procurements of advanced aircraft. It's hard to know long it will take and how much it will cost to build something that bleeding-edge.
The mythical man-month in action!
"Accountability". A conspicuously heroic effort (likely counterproductive long-run), because their +$6 billion drip is threatened by lawmakers who don't think they're taking this whole thing seriously.
For the sake of whatever competent programmers were already working on the system, I hope this is mostly PR noise.
For the sake of whatever competent programmers were already working on the system, I hope this is mostly PR noise.
That plane is not a replacement for the A-10, unless you kid yourself. The loiter time is atrocious.
Because loitering and enduring battle damage make so much sense in an era of countermeasure-immune SAMs and huge directed-shrapnel warheads. Spitting DU rounds onto a city block also isn't a great idea in today's conflicts. The A-10 is obsolete, the tankbuster role is obsolete, the F-35 isn't perfect but is far more appropriate for today's conflicts.
Drones are far more appropriate for today's conflicts.
The program should be cancelled.
It's way over budget, not as stealthy as originally claimed and completely unnecessary. The DOD should be putting this money and effort into UAVs.
I also have to wonder how quickly the Chinese will get the software updates.
It's way over budget, not as stealthy as originally claimed and completely unnecessary. The DOD should be putting this money and effort into UAVs.
I also have to wonder how quickly the Chinese will get the software updates.
The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.
The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.