Reversing Industrial Firmware for Fun and Backdoors I(reversemode.com)
reversemode.com
Reversing Industrial Firmware for Fun and Backdoors I
http://reversemode.com/index.php?option=com_content&task=view&id=80&Itemid=1
2 comments
What makes you think that the engineering teams in companies like Siemens, Rockwell Automation, Mitsubishi, GE etc. are "a long way behind in terms of 'software engineering'"? It is extremely misguided to think that all the thousands of engineers working in the Industrial Automation field are somehow behind others. I guess most of the HN crowd has very little exposure to the technologies in this field and most people probably consider this field uncool. But that doesn't mean the engineers working on these technologies are dumb and lagging behind others.
> What makes you think that the engineering teams in companies like Siemens, Rockwell Automation, Mitsubishi, GE etc. are "a long way behind in terms of 'software engineering'"?
data.
http://embeddedgurus.com/barr-code/2010/09/the-sad-state-of-...
I was in a class taught by this guy and I seem to recall him saying that a large (20-40%) of embedded developers he surveyed reported using no version control software.
and look at some of the responses in that thread!
"I think that gcc -Wall -Wextra can be subtituted by lint tool (if you use gcc)."
data.
http://embeddedgurus.com/barr-code/2010/09/the-sad-state-of-...
I was in a class taught by this guy and I seem to recall him saying that a large (20-40%) of embedded developers he surveyed reported using no version control software.
and look at some of the responses in that thread!
"I think that gcc -Wall -Wextra can be subtituted by lint tool (if you use gcc)."
"I was in a class taught by this guy and I seem to recall him saying that a large (20-40%) of embedded developers he surveyed reported using no version control software."
Yeah, but that's a vendor issue; it's not that the Engineers don't want to use version control.
We'd like to, for example, but we usually can't. All of the file formats for these PLC tools are binary blobs, and since the language is usually a graphical language (ladder logic) the conventional idea of diff is pretty meaningless unless someone writes a proprietary version control to handle the binary blobs + representing ladder logic through diffs.
We still try to use it, but we lose most of the benefits. I suppose you still get a changelog, but you can't ever merge branches since merging binary blobs with binary blobs will not go well for you.
Yeah, but that's a vendor issue; it's not that the Engineers don't want to use version control.
We'd like to, for example, but we usually can't. All of the file formats for these PLC tools are binary blobs, and since the language is usually a graphical language (ladder logic) the conventional idea of diff is pretty meaningless unless someone writes a proprietary version control to handle the binary blobs + representing ladder logic through diffs.
We still try to use it, but we lose most of the benefits. I suppose you still get a changelog, but you can't ever merge branches since merging binary blobs with binary blobs will not go well for you.
You get a change log, a central repository, and the ability to roll back to previous versions.
That's pretty valuable, even without the ability to merge.
That's pretty valuable, even without the ability to merge.
It's not really as useful as you make it out to be, since PLC code doesn't exist in isolation; there is __usually__ corresponding hardware. Unless you plan on magically "rolling back the hardware" changes as well, rolling back the software isn't all that useful in practice. Even if you can, changes are not made by any one vendor. Plant modifications happen all the time, so it's more often the case that you just end up uploading directly from the controller itself to see what the latest code is. I don't really have a good solution to that.
The exception of course is some very large companies that do everything in house.
The exception of course is some very large companies that do everything in house.
> It's not really as useful as you make it out to be, since PLC code doesn't exist in isolation; there is __usually__ corresponding hardware.
You never change the SW without changing the hardware?
Wowsers. No new features and no bugs.
You never change the SW without changing the hardware?
Wowsers. No new features and no bugs.
There's a science to this, it's called "change management". How do you think Boeing manages a plane with several 1000s of engineers, hundreds of systems and everything needs to work together...
> a large (20-40%) of embedded developers he surveyed reported using no version control software.
that's obviously bad, but is it higher than developers across disciplines? i'm sure you guys all use version control, but if you surveyed all web devs, how many of them use version control? i'd bet its a similar number
that's obviously bad, but is it higher than developers across disciplines? i'm sure you guys all use version control, but if you surveyed all web devs, how many of them use version control? i'd bet its a similar number
I'm an embedded consultant at a small firm of young guys. We use subversion, git and mercurial internally. We have project management tools, bug tracking and issue management, scheduling and time tracking and version control. We use VMware and VirtualBox to keep versioned installs of operating systems and toolchains. We're up to speed on the latest languages and frameworks, and we use whatever is right for the task at hand.
Our customers do not, in general, have any of those things. Many of them have no central version control (independent teams sometimes have subversion setup for themselves). Those who have it are usually using CVS, and usually don't use it very well.
Embedded is a field of "old fogeys" still. It isn't taught in school, it isn't advertised very well, and the companies who do it don't offer the cool perks that web companies do. A lot of these companies are running into problems where a large portion of their engineering force is retiring, and nobody is available to replace them.
This is more likely to be the case with our big customers than the smaller ones. The small ones usually have at least have some people pushing for modern project management, though it's often not implemented across the whole company.
Many of them got into the field when microcontrollers were extremely limited and C compilers were unreliable -- most of them learned assembly and used it for most of their career. There is still a tendency for the older ones to not believe us when we say their project can be written in C. Ten years ago, they would have been right.
Obviously, you can't generalize an entire industry as being "way behind", but many embedded firms just haven't progressed the same way those in other industries have.
Also, it's critical to realize that it has nothing to do with the engineers not being smart. They tend to be quite brilliant, in fact. But many of them are electrical engineers with a background in analog electronics, physics, and math, instead of modern software development.
Our customers do not, in general, have any of those things. Many of them have no central version control (independent teams sometimes have subversion setup for themselves). Those who have it are usually using CVS, and usually don't use it very well.
Embedded is a field of "old fogeys" still. It isn't taught in school, it isn't advertised very well, and the companies who do it don't offer the cool perks that web companies do. A lot of these companies are running into problems where a large portion of their engineering force is retiring, and nobody is available to replace them.
This is more likely to be the case with our big customers than the smaller ones. The small ones usually have at least have some people pushing for modern project management, though it's often not implemented across the whole company.
Many of them got into the field when microcontrollers were extremely limited and C compilers were unreliable -- most of them learned assembly and used it for most of their career. There is still a tendency for the older ones to not believe us when we say their project can be written in C. Ten years ago, they would have been right.
Obviously, you can't generalize an entire industry as being "way behind", but many embedded firms just haven't progressed the same way those in other industries have.
Also, it's critical to realize that it has nothing to do with the engineers not being smart. They tend to be quite brilliant, in fact. But many of them are electrical engineers with a background in analog electronics, physics, and math, instead of modern software development.
What makes you think that the engineering teams in companies like Siemens, Rockwell Automation, Mitsubishi, GE etc. are "a long way behind in terms of 'software engineering'"?
A bit of personal experience, stuff like Stuxnet as well as seeing bits of security research that confirm that notion.
Besides, not all control engineering is done by large corporations. There are tons of customizations done by smaller companies, whose competence varies.
A bit of personal experience, stuff like Stuxnet as well as seeing bits of security research that confirm that notion.
Besides, not all control engineering is done by large corporations. There are tons of customizations done by smaller companies, whose competence varies.
You're right, I should have quantified that statement a little better. The PLC manufacturers I'm sure have good software engineering practices. I'm just not sure the rest of the ecosystem is as developed as for instance the "web development" ecosystem, atleast from my experiences.
Mind you, I do live in the Netherlands, and it seems lots of PLC systems are designed by electrical engineers who started out in that or something else and don't know about much on the Joel Test. But I can be proven wrong.
A contributing factor is probably that each vendor has his/her toolkit which in some sense is proprietary, and so the market for general tools is smaller, and it's an enterprise market making it harder to sell to.
Mind you, I do live in the Netherlands, and it seems lots of PLC systems are designed by electrical engineers who started out in that or something else and don't know about much on the Joel Test. But I can be proven wrong.
A contributing factor is probably that each vendor has his/her toolkit which in some sense is proprietary, and so the market for general tools is smaller, and it's an enterprise market making it harder to sell to.
"PLC systems are designed by electrical engineers who started out in that or something else and don't know about much on the Joel Test. But I can be proven wrong."
That's probably true, given that PLCs were originally for electrical engineers as means of having "software relays". The concept of computer engineers came much later, and for computer science, they typically have zero electrical background which means that even if they program really well, it can be the case that they don't understand the physical implications of the code that they've just written.
In computer science, writing True might mean it's magically on, but in real life, a valve actually has a small delay before it physically is open. So a time delay has to be placed on the output. That's a simple example, but there are other such nuances of electrical/physical devices. Also, much of PLC programming (especially in process control) is control theory (PID controllers, essentially differential equations so that devices actuate "smoothly" instead of in a "jerky" manner). This also seems to not be covered in a CS curriculum.
That's probably true, given that PLCs were originally for electrical engineers as means of having "software relays". The concept of computer engineers came much later, and for computer science, they typically have zero electrical background which means that even if they program really well, it can be the case that they don't understand the physical implications of the code that they've just written.
In computer science, writing True might mean it's magically on, but in real life, a valve actually has a small delay before it physically is open. So a time delay has to be placed on the output. That's a simple example, but there are other such nuances of electrical/physical devices. Also, much of PLC programming (especially in process control) is control theory (PID controllers, essentially differential equations so that devices actuate "smoothly" instead of in a "jerky" manner). This also seems to not be covered in a CS curriculum.
It's not really that big a deal to grasp the physical implications of code and timing. Any network code will have the same issue, i.e., any web coding.
The industrial controls industry has a very 1980s/1990s attitude for cybersecurity. I don't envy the public-facing engineers working for the vendors.
The industrial controls industry has a very 1980s/1990s attitude for cybersecurity. I don't envy the public-facing engineers working for the vendors.
> I guess most of the HN crowd has very little exposure to the technologies in this field and most people probably consider this field uncool.
I'm always reminded of a conversation with a fresh Google hire wherein she said "How can you stand working at a utility, don't you want your work to affect PEOPLE?", as if utilities don't have hundreds of thousands or millions of customers that we provide an essential service to.
I'm always reminded of a conversation with a fresh Google hire wherein she said "How can you stand working at a utility, don't you want your work to affect PEOPLE?", as if utilities don't have hundreds of thousands or millions of customers that we provide an essential service to.
Try to work with Siemens Step7 for just one day.
Siemens has quite a peculiar bit of software; it is the only product that I have seen that requires you reboot BEFORE installation of every single Siemens product.
I think installing all of the Siemens software requires something like 15 reboots, at last count. And I have no idea why.
I think installing all of the Siemens software requires something like 15 reboots, at last count. And I have no idea why.
> Despite I'm releasing this information when there is still no patch available, It has been my decission. I reported it to the ICS-CERT months ago, I would like to thank the ICS-CERT and the Schneider security team, they have taken these issues very seriously and are working on a patch. During the process they have been keeping me updated on every decission/progress. However, time ago I decided to change my disclosure policy.
That's a lousy disclosure policy.
That's a lousy disclosure policy.
Playing devil's advocate: As the title says, the vendor included backdoors in the application. I'm not sure I can trust them and their clearly irresponsible development practices to patiently wait for them to handle this on their own.
Almost all PLCs have back doors. Just because it uses Ethernet doesn't mean it does anything fancy. Almost all the PLCs I am aware of simply wrap existing RS 232 protocols over IP from the 80s and even earlier. Most of the plants actually seem to want these back doors, because downtime at a production facility is incredibly expensive.
PLCs are not designed to externally accessible, ever. The back doors are completely irrelevant anyway given that the PLC will accept any packet from anywhere, and perform the operation. The RS-232 commands are functions such as "enable bit", "disable bit", "set value", "read value", for actuating inputs and outputs.
PLCs are not designed to externally accessible, ever. The back doors are completely irrelevant anyway given that the PLC will accept any packet from anywhere, and perform the operation. The RS-232 commands are functions such as "enable bit", "disable bit", "set value", "read value", for actuating inputs and outputs.
Someone forgot to tell the water pump engineers that in Illinois
> Playing devil's advocate: (...)
No you aren't. The parent post is playing devel's advocate; you are a down-to-earth, honest, practical user of software.
No you aren't. The parent post is playing devel's advocate; you are a down-to-earth, honest, practical user of software.
Nope, the only thing lousy is not saying it sooner. I am glad he decided that he will not be complicit in such reprehensible behavior from a vendor. Knowingly exposing your customers to harm without disclosure should have stiff penalties.
PLC and other industrial software is a long way behind in terms of "software engineering". A college classmate did a graduation project/internship figuring out how to do version control and modularize PLC programs with device drivers.