Why has software supply chain security exploded?(opensourcesecurity.io)
opensourcesecurity.io
Why has software supply chain security exploded?
https://opensourcesecurity.io/2022/09/06/why-has-software-supply-chain-security-exploded/
24 コメント
I'm not sure "new tooling" (if you're contrasting language-specific with linux distro tooling) does give more visibility though. How well does "new tooling" track root certificates being up-to-date and available for example (ca-certificates is explicitly mentioned in the vlc dependencies in the link as an example)? Or across language/ecosystem boundaries? It would be interesting to compare vlc dependencies on Windows, MacOS and one of the BSDs, as unlike linux distros, there is a explicit and somewhat complete core (which would likely remove or at least reduce the number of "system" dependencies on the vlc list).
I do think better tooling can exist (and is being developed), but it needs to track the whole system, not just a single language.
I do think better tooling can exist (and is being developed), but it needs to track the whole system, not just a single language.
Yes, I mean something like Cargo vs apt.
If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained.
But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about.
OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too.
Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.
If you depend on libcurl, you count that as 1 (one) dependency. Nice, and restrained.
But curl has everything from gopher, FTP to SMTP and IMAP! It has a half a dozen TLS libraries, SSH, auth methods, multiple compression algorithms, socks proxies, etc, It's a lot of code, and a lot of transitive dependencies that you don't even think about.
OTOH when you add `reqwest` in Cargo, Cargo will visibly fetch everything needed to build it from scratch, and it will look like 140 dependencies, not 1. But in total it's still only a subset of what libcurl contains. There's no Samba or POP3. It's just a very detailed list showing you it contains mundane things like a CLI arg parser and base64 encoder, and a wrapper for ca-certificates too.
Please don't interpret it as a criticism of curl. It's just to contrast that you may see libcurl as 1 dependency, because it's one precompiled package, but you may see a smaller amount of code as extravagantly bloated shocking 140 dependencies, because it's just been split into tiny pieces and rubbed in your face.
Curl, even compiled against all of its optional dependencies to support every possible kind of auth, encryption scheme, compression, etc. is nowhere 140 dependencies, even transitively. All of Linux from Scratch is only 75 packages.
In contrast, when I tried to add a token-replacement library to mdbook, Cargo installed 187 crates.
There is a real and legitimate difference in the size and functionality scope of older C and C++ libraries with the type of single-function packages you see from NPM and Cargo that has made full supply-chain audits far more difficult. To get functionality equivalent to something like Boost, you might need to vet a thousand separate vendors, the majority of which are pseudonymous Github accounts with no physical address or backing organization.
I'm not just speculating about this, either. I used to be a platform lead on fully-disconnected, classified information systems where we needed to bring in and compile every dependency from source via DVDs hand-carried to a classified workstation. Somewhere around 2015 or so, this stopped being even a remotely feasible job and created multi-year backlogs of teams that had something fully working in their unclassified development environment with Internet access suddenly finding themselves staring at a minimum 6-month, more likely indefinite wait to get to production when they discovered security teams needed to individually approve every package and every developer they wanted to bring onto a classified system.
In contrast, when I tried to add a token-replacement library to mdbook, Cargo installed 187 crates.
There is a real and legitimate difference in the size and functionality scope of older C and C++ libraries with the type of single-function packages you see from NPM and Cargo that has made full supply-chain audits far more difficult. To get functionality equivalent to something like Boost, you might need to vet a thousand separate vendors, the majority of which are pseudonymous Github accounts with no physical address or backing organization.
I'm not just speculating about this, either. I used to be a platform lead on fully-disconnected, classified information systems where we needed to bring in and compile every dependency from source via DVDs hand-carried to a classified workstation. Somewhere around 2015 or so, this stopped being even a remotely feasible job and created multi-year backlogs of teams that had something fully working in their unclassified development environment with Internet access suddenly finding themselves staring at a minimum 6-month, more likely indefinite wait to get to production when they discovered security teams needed to individually approve every package and every developer they wanted to bring onto a classified system.
But does not cargo bottom out at libraries like https://crates.io/crates/openssl-sys/0.9.75, where you do need addition (non-managed) dependencies? I would be surprised that the more "conservative" linux distros didn't have a complete, deduplicated and unvendored tree (only encountering issues around binary firmware and similar), whereas from my minimal rust knowledge, cargo's reqwest metadata is incomplete. It's the incompleteness I'm suggesting is a problem (I'm more familiar with Python and wheels, where there are substantial non-Python dependencies people forget about), not the number of dependencies (though that's not great either).
The "system dependency" wrapper crates are unfortunately dependent on the environment outside of Cargo's view. The Rewrite-In-Rust Strike Force is working on eliminating this problem ;)
Cargo/crates.io has a publicly available index of every crate with every dependency. `cargo tree` lists everything in your project. `Cargo.lock` contains inventory of every package used in the build, including their checksums.
See https://lib.rs/stats
Cargo/crates.io has a publicly available index of every crate with every dependency. `cargo tree` lists everything in your project. `Cargo.lock` contains inventory of every package used in the build, including their checksums.
See https://lib.rs/stats
I agree with this.
The example of 1000s of packages having a transitive dep on a package which left-pads strings looks bad, however the two obvious alternatives to this are:
1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.
2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.
The example of 1000s of packages having a transitive dep on a package which left-pads strings looks bad, however the two obvious alternatives to this are:
1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.
2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.
> 1. everyone writes their own function to left-pad. As well as being wasteful, most of these implementations are incorrect or perform badly or have their own security issues.
You sure? You can write it, with no security issues, with acceptable performance for most cases, in the same time than it takes to find the dependency. In fact, you'll save time since you don't have to vet the dependency.
Uh, you do like, vet your dependencies right, whenever you add them, right? As well as every time you update to a new version, right?
If you're not doing any of that and just adding dependencies willy nilly then I could see how it could seem easier, but this is an illusion. You need a dependency to do more than just left padding to make it worth adding, unless you don't give a crap about stability or security.
> 2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.
Yeah this is how languages have traditionally done it, and when done well it is very nice. As you say, it is hard to change the contract because it gets depended on, and you know what? This is a feature, not a bug. Things that are really simple and settled like leftpad don't need to constantly change. This is also why emerging languages try to stay "alpha" without guarantees about breaking changes as long as possible early on, so by the time they have to really freeze the standard library, a lot of experience has helped polish it.
A similar model is something like Java, which has done a fantastic job of evolving its standard library over the years while staying very (but, mostly, not excessively... mostly...) backwards compatible. Mostly through the trick of letting users create their own popular libraries and then eventually integrating it with the JEP process once one gets really big. Kind of like a lot of the stuff from Guava, or Jodatime.
You sure? You can write it, with no security issues, with acceptable performance for most cases, in the same time than it takes to find the dependency. In fact, you'll save time since you don't have to vet the dependency.
Uh, you do like, vet your dependencies right, whenever you add them, right? As well as every time you update to a new version, right?
If you're not doing any of that and just adding dependencies willy nilly then I could see how it could seem easier, but this is an illusion. You need a dependency to do more than just left padding to make it worth adding, unless you don't give a crap about stability or security.
> 2. the function to left-pad is part of a huge package which does thousands of things, eg. the programming language itself, or some standard library. In order to get a new version of left-pad, you have to upgrade the whole thing, and are potentially left with irreconcilable incompatibilities.
Yeah this is how languages have traditionally done it, and when done well it is very nice. As you say, it is hard to change the contract because it gets depended on, and you know what? This is a feature, not a bug. Things that are really simple and settled like leftpad don't need to constantly change. This is also why emerging languages try to stay "alpha" without guarantees about breaking changes as long as possible early on, so by the time they have to really freeze the standard library, a lot of experience has helped polish it.
A similar model is something like Java, which has done a fantastic job of evolving its standard library over the years while staying very (but, mostly, not excessively... mostly...) backwards compatible. Mostly through the trick of letting users create their own popular libraries and then eventually integrating it with the JEP process once one gets really big. Kind of like a lot of the stuff from Guava, or Jodatime.
3. Have a package of common utility functions maintained by organization rather than an individual, like Apache Commons Lang.
The answer is "too many dependencies".
Package manages made it easy to have dependencies. Every added dependency adds risk. That doesn't mean you should never add dependencies - there are good reasons to do so - but you should weight their pros and cons. Adding dozends, hundreds or thousands of dependencies (as it is common in the npm ecosystem) adds lots of risk. None of that is particularly surprising.
Package manages made it easy to have dependencies. Every added dependency adds risk. That doesn't mean you should never add dependencies - there are good reasons to do so - but you should weight their pros and cons. Adding dozends, hundreds or thousands of dependencies (as it is common in the npm ecosystem) adds lots of risk. None of that is particularly surprising.
This is the #1 reason we use .NET for our banking apps. Everything we need is baked into the Microsoft-owned dependencies. Certainly, you could argue of transitive attacks still being possible but from a liability perspective we get to point our finger at Microsoft if something goes wrong (while we hastily rebuild with their patch)
Even for my side project - building a custom 3D rendering engine from scratch - I use libraries like System.Numerics to calculate the camera perspective transform and perform all the linear algebra for me. I was wasting a lot of time trying to reinvent this, which is a typical but lethal risk to any project.
Even for my side project - building a custom 3D rendering engine from scratch - I use libraries like System.Numerics to calculate the camera perspective transform and perform all the linear algebra for me. I was wasting a lot of time trying to reinvent this, which is a typical but lethal risk to any project.
As head of eng in a ftse100, this is so far from the truth. .Net is a heavily targeted ecosphere, possibly moreso than npm, precisely because of its relation with finance.
We have to contend with not only nuget and the sheer abundance of packages available (with their own transitive dependencies similar to the problems with npm - yes, even Microsoft packages have this risk) posing problems, but also the tooling itself.
VS/Code is a fucking minefield of potential threats. Their extensions market place(s) are automatic RCE risks, and a developer/admin/whomever box getting pwned is a monumental risk.
Zero-Trust applies to everyone and everything. Microsoft don't get a pass just for being Microsoft.
We have to contend with not only nuget and the sheer abundance of packages available (with their own transitive dependencies similar to the problems with npm - yes, even Microsoft packages have this risk) posing problems, but also the tooling itself.
VS/Code is a fucking minefield of potential threats. Their extensions market place(s) are automatic RCE risks, and a developer/admin/whomever box getting pwned is a monumental risk.
Zero-Trust applies to everyone and everything. Microsoft don't get a pass just for being Microsoft.
Your reply is very uncharitable. Newer versions of .NET rely on next to zero non-Microsoft dependencies. The most notable exception to this used to be Newtonsoft.Json, which has since been replaced with Microsoft-owned System.Text.Json. You can easily implement very complex business applications with app stacks like ASP.NET or WPF while relying on exactly zero non-Microsoft dependencies.
The GP never said Microsoft "gets a pass." They were pointing out that there are stacks like .NET where reliance on third-party unvetted libraries is greatly reduced if not eliminated outright. Additionally, nearly all businesses run Windows anyway, so using Visual Studio and Visual Studio Code are not the risks you make them out to be. Centralized Active Directory policies already help guard against rogue extensions and such, assuming IT even cares (they nearly never do in my experience).
Yes, the surface area threat of .NET and NuGet is far, far lower than that of NPM for nearly all apps.
The GP never said Microsoft "gets a pass." They were pointing out that there are stacks like .NET where reliance on third-party unvetted libraries is greatly reduced if not eliminated outright. Additionally, nearly all businesses run Windows anyway, so using Visual Studio and Visual Studio Code are not the risks you make them out to be. Centralized Active Directory policies already help guard against rogue extensions and such, assuming IT even cares (they nearly never do in my experience).
Yes, the surface area threat of .NET and NuGet is far, far lower than that of NPM for nearly all apps.
Are you claiming that Node/NPM is somehow a less vulnerable ecosystem than .NET/NuGet?
That's not what they said. They said .NET/NuGet shares some of the problems NPM has, and that it's targeted more because it's a juicer target.
That doesn't mean it's more vulnerable, but it means it has to be less vulnerable to offer the same security. To offer a bad analogy: if a bank vault was as vulnerable as my fridge, people would constantly break into the bank vault but keep ignoring my fridge.
That doesn't mean it's more vulnerable, but it means it has to be less vulnerable to offer the same security. To offer a bad analogy: if a bank vault was as vulnerable as my fridge, people would constantly break into the bank vault but keep ignoring my fridge.
Nope. Next strawman, please.
This is especially a problem in languages which have a chronic lack of decent standard library and everything relies on a third-party package.
For-profit package managers like NPM touted download and package counts as if they were KPI. Deep dependency trees is the natural result.
Installing react-scripts is ~1400 dependencies and around 2 million "edges" (where one package requires another). This blew my mind recently when "yarn audit" was failing with an OOM error!
The complete graph on 1400 nodes only has a million edges.
That's ret-conning a correlation. Npm aren't _generating_ packages themselves. The community at large is generating them.
I would wager that the PHP standard lib isn't all that comprehensive, but it doesn't suffer anywhere near as much from nesting dependencies. I think the culture is that a library takes care of itself for the most part.
I built zellij (rust tmux clone) recently. 193 dependencies that cargo downloaded. When did rust turn into javascript? Are there supply chain attacks in Rust?
I really wonder why supply chain security wasn't such a big issue before, and why there weren't really many supply-chain attacks on software before.
Too few attackers understood it well enough? Or were other attack vectors easier?
Too few attackers understood it well enough? Or were other attack vectors easier?
Because C does not have package managers. We were copy pasting libraries as .h files in our projects and never updating them.
I still find it mind boggling that businesses have build processes for their core properties that rely on pulling hugs swathes off third party code in from the internet. How can that ever possibly be secure and robust?
Had they written all the dependencies themselves, would they have been as secure and robust?
The tradeoff isn't simple, and will be different for each company. I think everyone can point at a couple of components that are complex and massively benefit from the work poured into them and the large install base finding all the edge cases. Equally everyone can name a couple of dependencies that anyone could rewrite in a couple hours, and maybe even improve on the way.
The tradeoff isn't simple, and will be different for each company. I think everyone can point at a couple of components that are complex and massively benefit from the work poured into them and the large install base finding all the edge cases. Equally everyone can name a couple of dependencies that anyone could rewrite in a couple hours, and maybe even improve on the way.
Maybe, maybe not, but they certainly wouldn’t have included as many widespread, well known vulnerabilities with exploits already in the wild. And even if the imported code was perfect at the time it was imported, pulling in changes automatically is still an attack vector that you could drive a fleet of buses through.
Note that I’m not saying “never use libraries”, which your points seem mainly aimed at. Code reuse is great! I’m just not sold on automatic, unmonitored updates to libraries.
Note that I’m not saying “never use libraries”, which your points seem mainly aimed at. Code reuse is great! I’m just not sold on automatic, unmonitored updates to libraries.
It’s even scarier when you actually sit down and review some of the code you’ve been importing for years…
It has resulted in a major thinking shift for me. Mostly initiated by a malicious nodejs package being quietly pulled in.
It has resulted in a major thinking shift for me. Mostly initiated by a malicious nodejs package being quietly pulled in.
Same with C++. Dependencies were boost, qt, gtk, and a few others and that was pretty much it. Since vcpkg, cmake, conan, and a few others, the C++ "ecosystem" has started growing far more in the last 10 years, than in the previous 20.
Is the modern tooling to blame for? It didn't use to be so simple that everything could be automatically updated or pulled. Had to actually go out and see what you were brining in and maybe even take a look in documentation or dates of last update.
I don't think there's one real reason but a combination, but I can list the features that C# and Java ecosystems had since the early 00s, and javascript/Rust have since they came around, but C++ hasn't had until recently.
1. one or more major package hubs
2. one package manager, or at least, a package standard
3. easy to consume/build (source code to binary)
4. easy to publish
5. easy to download (one liner in your build script, e.g. package.json, vcproj, cargo.toml, build.gradle etc.)
At the moment C++ only really has 5. - easy to download: cmake, conan, vcpkg, build2, can all download easily.
1-4 are a work in progress for C++, but are far better than 10 years ago.
1. one or more major package hubs
2. one package manager, or at least, a package standard
3. easy to consume/build (source code to binary)
4. easy to publish
5. easy to download (one liner in your build script, e.g. package.json, vcproj, cargo.toml, build.gradle etc.)
At the moment C++ only really has 5. - easy to download: cmake, conan, vcpkg, build2, can all download easily.
1-4 are a work in progress for C++, but are far better than 10 years ago.
Looking at https://wiki.alopex.li/LetsBeRealAboutDependencies
Perhaps the reason is package managers for C are called distro maintainers (for Linux/BSD)
Perhaps the reason is package managers for C are called distro maintainers (for Linux/BSD)
There's a copycat effect at play too. Once something becomes well-known enough, it enters as an idea into more people's minds that it's a possible thing, and some will attempt it. Before it was smashing the stack, now it's supply chains etc. As for what you can do with an attack vector to motivate beyond a simple satisfaction in breaking something, an obvious popular choice now is ransomware. Ransomware attacks would still continue even if the world somehow got rid of cryptocurrencies, they would just use less convenient but more traditional forms of payment. Because ransomware became popular.
Attackers do what makes money. A lot of that is market driven - you buy what exploits are available, what tools are available, etc. You monetize in a fairly standard way. Attackers aren't all just random people thinking "oh I could hack that" - that's a tiny minority of them.
Eventually, due to a lot of factors, threat landscapes start shifting and attackers start moving their targets and tooling. One of the bigger shifts would be the huge, rapid improvement to both OS and browser security that occurred within a few years, radically increasing the cost of attacking desktop users via malicious websites. Another would be crypto, where 'account takeover' attacks that could lead to wallet access are now easier to monetize + crypto itself as a tool for transfers.
With regards to supply chain, enough of these changes occurred that some attackers took the leap and have started looking at this area. There are probably a lot of reasons why - prevalence of dependencies, increased interest in tech companies, etc.
If it continues to prove viable (it's obviously viable from an attack perspective, unclear if it's something attackers will rally around to monetize) we'll see it escalate and get better tooling around the attacks.
Eventually, due to a lot of factors, threat landscapes start shifting and attackers start moving their targets and tooling. One of the bigger shifts would be the huge, rapid improvement to both OS and browser security that occurred within a few years, radically increasing the cost of attacking desktop users via malicious websites. Another would be crypto, where 'account takeover' attacks that could lead to wallet access are now easier to monetize + crypto itself as a tool for transfers.
With regards to supply chain, enough of these changes occurred that some attackers took the leap and have started looking at this area. There are probably a lot of reasons why - prevalence of dependencies, increased interest in tech companies, etc.
If it continues to prove viable (it's obviously viable from an attack perspective, unclear if it's something attackers will rally around to monetize) we'll see it escalate and get better tooling around the attacks.
Programming style has changed over time to include more dependencies from 3rd party. Look at style of programming from 70s, they would do everything from OS , filesystem to application. Today there are many dependencies.
EDIT: It's much like building a tower. Upper floors depends on lower floors. Taller the construction more unstable it gets.
EDIT: It's much like building a tower. Upper floors depends on lower floors. Taller the construction more unstable it gets.
Probably because the CIA was doing tons of it and did not want to tell people :-D
i have a personal take on this, which is that people have yet to realize that pulling new shiny dependencies off GitHub (or entire unaudited container images off Docker Hub) and putting them directly into production is not necessarily a good idea…
I mean, it's a speed v. security thing. Every engineer (and I don't mean software engineers here) will tell you this is insane. You can't trust your bricks and yet, you're building a cathedral.
I know that automating stuff to check for CVEs and vulnerabilities of sorts is getting a lot of traction recently, but I don't think this is really sufficient. Who do you trust? Are we going to see some companies creating a "Security Seal of Approval" for packages?
I'm not sure what's the answer here, but certainly it isn't what we have been doing for the last couple years.
I know that automating stuff to check for CVEs and vulnerabilities of sorts is getting a lot of traction recently, but I don't think this is really sufficient. Who do you trust? Are we going to see some companies creating a "Security Seal of Approval" for packages?
I'm not sure what's the answer here, but certainly it isn't what we have been doing for the last couple years.
There are a few companies in this space that are trying to do the "Security Seal of Approval" thing to various degrees.
Tidelift is one company that has a bunch of "catalogs"[0] of packages. I'm not sure how their package metadata is generated though -- maybe semi-manually?
There is also Bytesafe[1] which is supposed to help give you a way to "firewall" yourself from unapproved dependencies. I don't think they sell data though. Just tools.
A company based on the Open Source project, packj[2], is Ossilate[3] which is trying to use automated analysis for analyzing 3rd party packages.
And another that's in this space is Socket.dev[4] which tries to warn you in your PRs about bad packages.
I know about this space because I work on a project[5] that's also related to supply chain security. It's a bit different from all of the above since we're focused on patching known vulns, but the idea of "vetted packages" has crossed my mind before.
Are there any other services in this space that I missed?
0: https://tidelift.com/solutions/catalogs
1: https://bytesafe.dev/
2: https://github.com/ossillate-inc/packj
3: https://ossillate.com/
4: https://socket.dev/
5: https://github.com/lunasec-io/lunasec/
Tidelift is one company that has a bunch of "catalogs"[0] of packages. I'm not sure how their package metadata is generated though -- maybe semi-manually?
There is also Bytesafe[1] which is supposed to help give you a way to "firewall" yourself from unapproved dependencies. I don't think they sell data though. Just tools.
A company based on the Open Source project, packj[2], is Ossilate[3] which is trying to use automated analysis for analyzing 3rd party packages.
And another that's in this space is Socket.dev[4] which tries to warn you in your PRs about bad packages.
I know about this space because I work on a project[5] that's also related to supply chain security. It's a bit different from all of the above since we're focused on patching known vulns, but the idea of "vetted packages" has crossed my mind before.
Are there any other services in this space that I missed?
0: https://tidelift.com/solutions/catalogs
1: https://bytesafe.dev/
2: https://github.com/ossillate-inc/packj
3: https://ossillate.com/
4: https://socket.dev/
5: https://github.com/lunasec-io/lunasec/
You can trust those who depend upon a lifelihood from you.
You can trust who you pay properly.
Who entrust the survival of theire family to your continued existence.
Whos bread i take, whos song i sing.
Thats the supply chain.
Turns out there is no free lunch to be had even in open source.
You can trust who you pay properly.
Who entrust the survival of theire family to your continued existence.
Whos bread i take, whos song i sing.
Thats the supply chain.
Turns out there is no free lunch to be had even in open source.
Isn't this what Linux distributions like Red Hat Enterprise Linux are doing? Providing a secure (among other things) platform on which you can build on?
This is only insane in the engineering world because actual human lives are depending on the structural integrity of your cathedral.
Even if you face a supply chain attack, the only value lost may be money for most businesses, and not enough that worrying about supply chain attacks is worth it.
Even if you face a supply chain attack, the only value lost may be money for most businesses, and not enough that worrying about supply chain attacks is worth it.
Actual human lives depend on software development. Hospitals, Airports, Airplanes, missiles, satellites. If any of these are attacked via supply chain lives will be lost and countries could even collapse if the attack was severe enough.
Almost neither of which use a lot of dependencies.
It's still C and C++ world out there, and everything gets shrink wrapped. It's more likely that you would find a logic bug or a well known security issue in a single library than somehow infest the supply chain of these particular applications.
Every piece of off the shelf software is labelled and audited in that world.
It's still C and C++ world out there, and everything gets shrink wrapped. It's more likely that you would find a logic bug or a well known security issue in a single library than somehow infest the supply chain of these particular applications.
Every piece of off the shelf software is labelled and audited in that world.
Or people do realise the risk and are balancing the risk of speed of dev with the likelihood of something going wrong (and them being blamed for it). Imperfect systems often produce a lot of value, even if it means playing a game of spinning plates to keep it going. It's very possible that the value produced by this system overshadows the lost value from an ocassional black swan event as a result of the systems failure modes.
At least github supports 2FA, signing of commits, SSO, etc. It can be a lot more trustworthy than a package distro.
I think it has a lot to do with 2 things:
1. A conscious shift in focus from triaging risks when they occur to stopping threats before they arise.
This is a natural next step from contemporary security and disaster response. Threat response and continuity planning which both incorporate plans that respond to threats were once the primary objective of organizations. They are still valid, but a more modern and proactive approach includes mitigating the risk at the source.
2. A forced increase in material spend toward securing dev and devops ecosystems in a time where they are one of the most targeted parts of the organization.
One only has to watch the news to see this play out... unfortunately after decades of deployment and intranet security emphasis, hackers have recognized that IP and source code are the best way to get money out of a company, and that both are ironically some of the least protected assets.
1. A conscious shift in focus from triaging risks when they occur to stopping threats before they arise.
This is a natural next step from contemporary security and disaster response. Threat response and continuity planning which both incorporate plans that respond to threats were once the primary objective of organizations. They are still valid, but a more modern and proactive approach includes mitigating the risk at the source.
2. A forced increase in material spend toward securing dev and devops ecosystems in a time where they are one of the most targeted parts of the organization.
One only has to watch the news to see this play out... unfortunately after decades of deployment and intranet security emphasis, hackers have recognized that IP and source code are the best way to get money out of a company, and that both are ironically some of the least protected assets.
These days it's always hard to say how much of the hype in areas like supply chain security is driven by the problem actually becoming worse and how much is due to VC funded startups with enormous marketing budgets that try to convince everyone of the nearing apocalypse (but no worries, their tools can save us).
Looking at integrated vulnerability reporting as found e.g. in npm (and soon Golang, I think) it seems that 99 % of alerts are just false positives, as they affect build dependencies (e.g. the tar library) somewhere down the chain that will never run in production or even touch anything sensitive, and that cannot be easily exploited even by sophisticated actors. In my opinion the "high severity" threat category is thrown around way too lightly by these tools, which leads to everyone panicking.
If anything good could come out of this whole trend it might be the realization that open-source software needs better funding to address supply chain issue, my expectation though is that most of the money will go to tool vendors that simple tape over the underlying problems by creating an artificial "security layer" between the maintainers and users.
Looking at integrated vulnerability reporting as found e.g. in npm (and soon Golang, I think) it seems that 99 % of alerts are just false positives, as they affect build dependencies (e.g. the tar library) somewhere down the chain that will never run in production or even touch anything sensitive, and that cannot be easily exploited even by sophisticated actors. In my opinion the "high severity" threat category is thrown around way too lightly by these tools, which leads to everyone panicking.
If anything good could come out of this whole trend it might be the realization that open-source software needs better funding to address supply chain issue, my expectation though is that most of the money will go to tool vendors that simple tape over the underlying problems by creating an artificial "security layer" between the maintainers and users.
The CVE landscape has been broken for ages. In a past life I did a lot of work on a bug bounty program and the amount of "I ran some simple tool that says CVE-XYZ is present in your codebase" reporting we received when the relevant code was either dead, ran in a trusted environment, or ran on trusted input was outrageous. Then people would whine about not getting paid.
I do think that organizations should have a process for identifying and triaging bugs in dependencies. But "there are vulns in dependencies" is not itself a compelling narrative because there are vulns in your business' code too. And people sure as hell are getting updates more frequently when using npm or whatever then in the alternative world of "link in some open source code and never update ever for all time."
To me, the more interesting problem is the "malicious actor takes over an npm account and distributes malware" that is more unique to the npm ecosystem. This one is hard. On the one hand developers are expensive and a robust library ecosystem is basically a superpower. On the other hand you really need a trust relationship with your software vendors if you want to be confident you are delivering safe code.
I do think that organizations should have a process for identifying and triaging bugs in dependencies. But "there are vulns in dependencies" is not itself a compelling narrative because there are vulns in your business' code too. And people sure as hell are getting updates more frequently when using npm or whatever then in the alternative world of "link in some open source code and never update ever for all time."
To me, the more interesting problem is the "malicious actor takes over an npm account and distributes malware" that is more unique to the npm ecosystem. This one is hard. On the one hand developers are expensive and a robust library ecosystem is basically a superpower. On the other hand you really need a trust relationship with your software vendors if you want to be confident you are delivering safe code.
Vendors had to start checking the "ran supply chain audit" box since investors demand they do.
Boom, new industry of snake oil.
Not suggesting that the investors have stakes in any companies doing supply chain audits or building automated solutions for it, of course.
Boom, new industry of snake oil.
Not suggesting that the investors have stakes in any companies doing supply chain audits or building automated solutions for it, of course.
There was never a supply "chain" to begin with.
If you're looking for a more precise metaphor, think of it as a large pot of slightly overcooked spaghetti and meatballs.
Where anyone can throw stuff into the pot.
If you're looking for a more precise metaphor, think of it as a large pot of slightly overcooked spaghetti and meatballs.
Where anyone can throw stuff into the pot.
I think you're oversimplifying spaghetti and meatballs. There are supply chains that go into that as well. Nothing in the modern world is built from raw materials. In theory, if it weren't for supply chain security, someone could poison you in your kitchen by going to a chicken farm, feeding toxins to chickens, who lay contaminated eggs which are sold to a factory that makes pasta, which is sold to a grocery store where your coworker bought it to make spaghetti and meatballs that he brought to your potluck. Even I'm over simplifying it. The point is, without supply chain security, anyone already can throw stuff into any pot.
At the top level, that sounds apt.
But people have automated the throwing in of stuff, and even when the original stuff was excellent, stuff sources are sometimes bought out by bad actors.
And sometimes the stuff has ingredients, where the ingredients are also so managed, so I would say that there is still a chain.
But people have automated the throwing in of stuff, and even when the original stuff was excellent, stuff sources are sometimes bought out by bad actors.
And sometimes the stuff has ingredients, where the ingredients are also so managed, so I would say that there is still a chain.
[deleted]
The hardwiring of dependencies in a Nixpkg-based deployment seems to be good for generating SBOMs.
Cf. https://discourse.nixos.org/t/generating-software-bill-of-ma...
Cf. https://discourse.nixos.org/t/generating-software-bill-of-ma...
Assuming you're using only Nixpkg and not doing pip install, npm install, cargo install and so on like many people suggest and do.
It used to be that adding library dependencies to a project had a lot of fraction, and if you were writing a library yourself, then imposing indirect dependencies on users was something you'd try fairly hard to avoid.
Then npm came, along with some ideology about code-reuse, and the friction went away. But the friction was serving an important function: if adding a library is annoying, you'll add a small number of large libraries rather than a large number of small ones, and you avoid getting an ecosystem where major libraries have hundreds of tiny dependencies. This is important because the friction of adding a dependency is only a small portion of its true cost: the main cost is that you're trusting too many different developers and developers' computers.
Then npm came, along with some ideology about code-reuse, and the friction went away. But the friction was serving an important function: if adding a library is annoying, you'll add a small number of large libraries rather than a large number of small ones, and you avoid getting an ecosystem where major libraries have hundreds of tiny dependencies. This is important because the friction of adding a dependency is only a small portion of its true cost: the main cost is that you're trusting too many different developers and developers' computers.
This is a good framing of it.
Can we keep the benefits of automatic dependency resolution while adding some other, artificial deterrent? What if NPM hosted packages for free if they have less than five (or whatever number) of transitive dependencies, but above that charged some number per additional one? Even if it were a small amount, this might put significant downward pressure on the size of dependency trees.
Can we keep the benefits of automatic dependency resolution while adding some other, artificial deterrent? What if NPM hosted packages for free if they have less than five (or whatever number) of transitive dependencies, but above that charged some number per additional one? Even if it were a small amount, this might put significant downward pressure on the size of dependency trees.
Security snake oil merchants figured out a new way to make a quick buck.
Of course they will, but the actual problem is there.
I've been delivering systems where we had to deliver the source environment (ie what is used to build production) into escrow with a lawyer as part of the contract. It effectively required us to prove that what we delivered (both hardware and software) into escrow, would be able to reproduce what was in production, reliably and provably.
Reproducible builds are really hard, especially with people encoding CVS/SVN/git tags into the build, compilers and linkers not producing equivalent results between runs (eg caches warm/cold can make a difference) etc etc.
It's a real problem and a real issue.
I've been delivering systems where we had to deliver the source environment (ie what is used to build production) into escrow with a lawyer as part of the contract. It effectively required us to prove that what we delivered (both hardware and software) into escrow, would be able to reproduce what was in production, reliably and provably.
Reproducible builds are really hard, especially with people encoding CVS/SVN/git tags into the build, compilers and linkers not producing equivalent results between runs (eg caches warm/cold can make a difference) etc etc.
It's a real problem and a real issue.
I run around the office with scissors air gapping your cables, pay me or i shall destroy productivity to the point security is the least of your concerns. Once i get your employees to work around me, violating contract, your problems are "over".
I'll take the advice of this article lacking substance, and complain.
Well, companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern.
> Open source won because everyone worked together.
No, ``open source'' has ``won'' because it helped corporations defang Free Software and get gratis labor.
> Supply chain security will happen because everyone works together. If you try to do this alone, you will fail.
This isn't food, but perfectly uniform applied mathematics. There's no ``working together'' with corporations. Doing it alone is the only reasonable option, which means aggressively reducing the size of this foul shit is necessary.
> Thank goodness others had the foresight to create our current SBOM formats.
Yeah, MicroSoft totally *<3* ``open source'' guys.
> Long ago all the software supply chain work was done by hand.
Oh, right, all of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.
Well, companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern.
> Open source won because everyone worked together.
No, ``open source'' has ``won'' because it helped corporations defang Free Software and get gratis labor.
> Supply chain security will happen because everyone works together. If you try to do this alone, you will fail.
This isn't food, but perfectly uniform applied mathematics. There's no ``working together'' with corporations. Doing it alone is the only reasonable option, which means aggressively reducing the size of this foul shit is necessary.
> Thank goodness others had the foresight to create our current SBOM formats.
Yeah, MicroSoft totally *<3* ``open source'' guys.
> Long ago all the software supply chain work was done by hand.
Oh, right, all of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.
Business depends on trust, there is risk involved in doing business. There is always chance somebody will either not pay or not deliver, or deliver lower quality products than ordered (like this is the case with supply chain attack). And there is not much you can do about it. Later you can sue them or not do business with them anymore, but you don't know that in advance.
How about all the throubles with hijacked NPM packages? These attacks are still "small", but have had big effects, multiple times.
At a previous job we were building something significant in the Java stack, and as part of the CI/CD we have a jar checking stage which flagged that an emoji library, that wasn't an emoji library, suddenly was coming in to our build transitively. Presumably somewhere down the line, someone wasn't using a fixed version in gradle or maven.
I quit a gig recently because the CI/CD pipeline had automatic scanning of dependencies for vulns (yay!) and no tangible process for dealing with them.
It was up to the dev to "fix" this "defect" before code could be merged.
There was no pinning policy. There was no actual security review (where do we use this? are we vulnerable? why do we use this? are there alternatives?).
Meanwhile it's running in production. (have we discussed this with our MSP? can I discuss this with our MSP? has our MSP notified us of any issues?)
Although we say ignorance is no excuse, having spent a decade in cybersecurity and 20 years before that as a quality-focused consultant, I'd say ignorance lets people get away with a lot. But I know better and I would never be allowed to claim "I vass only followeeeng zee orders!" if the shit hit the fan.
It was up to the dev to "fix" this "defect" before code could be merged.
There was no pinning policy. There was no actual security review (where do we use this? are we vulnerable? why do we use this? are there alternatives?).
Meanwhile it's running in production. (have we discussed this with our MSP? can I discuss this with our MSP? has our MSP notified us of any issues?)
Although we say ignorance is no excuse, having spent a decade in cybersecurity and 20 years before that as a quality-focused consultant, I'd say ignorance lets people get away with a lot. But I know better and I would never be allowed to claim "I vass only followeeeng zee orders!" if the shit hit the fan.
Because it was bound to happen eventually.
I felt clickbaited after reading the article. It's interesting to read, but if you're actually looking for the answer to the question then it's not in the article.
log4j introduced supply chain security to the wider corporate world where most production run on Java and a lot more people finally realized how complicated it is to do incident response for these kind of situations.
[deleted]
There is more money to be earned in exploiting the supply chain.
npm install
It all started with glib/compilers/LLVM/Windows toolkit not being able to reproduce a build, byte-for-byte.
As a QA can see, the inability to procure a reproducible executable checksum values allowed all sorts of non-validated (and un-validatable) things to creep into production.
These folks are working on fixing that problem:
https://reproducible-builds.org/ https://bootstrappable.org/
https://reproducible-builds.org/ https://bootstrappable.org/
Of course, timestamp is the biggest offender of recreatable (deterministic) build.
https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...
https://blog.conan.io/2019/09/02/Deterministic-builds-with-C...
Indeed, tons of other issues prevent deterministic builds though:
https://reproducible-builds.org/docs/
https://reproducible-builds.org/docs/
What difference does it make if the build is compromised in an identical way byte-for-byte?
• There used to be a focus on attacking endpoints directly, but eventually OSes, browsers, and servers stopped being easy targets, so attackers looked for another way in.
• OS (especially Linux) distributors acted as gatekeepers. Getting into a distro requires having some reputation, and the risk of getting caught with a backdoor is a deterrent (although I don't think distros are as much of security barrier as people assume, since maintainers already have too much work to do and their source code review is at best at "LGTM" level).
• New tooling gave more visibility into dependencies. Transitive and build-time dependencies in precompiled libraries typically aren't visible, so it's easy to think you have one or two deps, but transitively there are still tons of them: https://wiki.alopex.li/LetsBeRealAboutDependencies
• Exploits became easier to monetize. Thanks to crypto miners and crypto wallets any exploit is almost a direct payout. Way easier and safer than trying to sell a botnet, adware or even stolen CC numbers, so it lowered barriers to entry to this "business" as well as made it more attractive.