Time for developers to address the chaos of Node.js and NPM(techrepublic.com)
techrepublic.com
Time for developers to address the chaos of Node.js and NPM
https://www.techrepublic.com/article/why-its-finally-time-for-developers-to-address-the-chaos-of-node-js-and-npm/
13 comments
How is it different from maven deps?
Usually you don't use maven deps directly. You have a registry in your company that caches maven packages for you and runs vulnerability scans on the binary. So the binaries you end up using are "tested" internally.
Is that how people use NPM?
Also maven central does NOT support unpublishing of binaries. So if you have version 1.2 of something and you have scanned it once for vulnerabilities, it should always be the correct one. Nobody can publish again 1.2 in the central repo
Is that how people use NPM?
Also maven central does NOT support unpublishing of binaries. So if you have version 1.2 of something and you have scanned it once for vulnerabilities, it should always be the correct one. Nobody can publish again 1.2 in the central repo
In every somewhat serious company I've worked they used a local npm cache like jfrog (usually same one that's used for maven) that can get scanned for security. So yeah, that's how people use npm.
Unpublishing within those companies is usually not allowed, in the official npm registry you can only unpublish within the first 24 hours of publishing, after that you need to request it by email.
Unpublishing within those companies is usually not allowed, in the official npm registry you can only unpublish within the first 24 hours of publishing, after that you need to request it by email.
>In every somewhat serious company I've worked they used a local npm
If that was the case for everybody, then left-pad would not have caused a problem for anybody. People would not notice anything wrong with their builds. The amount of github comments where people say that their production build just broke because of a missing package implies that people are NOT using NPM with a local registry
>in the official npm registry you can only unpublish within the first 24 hours of publishing
I don't believe this to be the case https://news.ycombinator.com/item?id=16087024
If that was the case for everybody, then left-pad would not have caused a problem for anybody. People would not notice anything wrong with their builds. The amount of github comments where people say that their production build just broke because of a missing package implies that people are NOT using NPM with a local registry
>in the official npm registry you can only unpublish within the first 24 hours of publishing
I don't believe this to be the case https://news.ycombinator.com/item?id=16087024
Just because it's a shitty common practice doesn't mean there isn't a proper way to do it (local repo). For example, there are tons of people who check in their secrets into public and private Github repos (including Uber eng), then they get jacked and complain. If they were smarter than that, "it would not have caused a problem for anybody". Doesn't really prove that what a lot of people do must be automatically good.
True, but the fact that NPM seems to be the only repository where packages can be "re-published" further exacerbates the bad practices.
https://status.npmjs.org/incidents/41zfb8qpvrdj
The fact that 9 packages could be "published over" _after_ the left-pad fiasco, shows lack of attention.
Tools should try to foster good practices, instead of worsening bad ones.
https://status.npmjs.org/incidents/41zfb8qpvrdj
The fact that 9 packages could be "published over" _after_ the left-pad fiasco, shows lack of attention.
Tools should try to foster good practices, instead of worsening bad ones.
[deleted]
I don't know if it's still the case, but one of the first things I learned about JavaScript was how it was inherently secure. Then I actually started using the language, and learned what utter bullshit that is.
> inherently secure
Who told you that? I don't think "inherently secure" is a thing in the tech industry.
Who told you that? I don't think "inherently secure" is a thing in the tech industry.
The ECMAScript spec doesn't include details about its environment. You can run it in varying environments, each of which has different security criteria. For example, you can run a fully isolated JS VM that has no interactions with the real world, which is certainly quite safe. Node has full access to pretty much all the user's system resources, while the browser is a bit more sandboxed and restricted in what it can access.
Except in the case of Spectre/Meltdown...
Related: 3m installs in 7 days, of a “module” to check if a number is odd.
https://www.reddit.com/r/programming/comments/886zji/why_has...
https://www.reddit.com/r/programming/comments/886zji/why_has...
[deleted]
It's troubling that PHP gets so much shit because it has an extensive if clumsy standard library, where node gets a free pass because you can get whatever doesn't exist from NPM, welcoming any number of security vulnerabilities into applications.
I recently cleaned my NPM cache at work after 9 months, it was 6GB. ES6 is a massive improvement, but Node's standard library is still lacking and the sheer amount of tooling needed to be effective and up to date is too much.
I recently cleaned my NPM cache at work after 9 months, it was 6GB. ES6 is a massive improvement, but Node's standard library is still lacking and the sheer amount of tooling needed to be effective and up to date is too much.
PHP isn't criticized because it has a big and clumsy library. The criticisms for PHP are very clear: https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/
It is badly designed.
It is badly designed.
The fractal blog post will be 6 years old in a couple of weeks. I'm going through it again, and __a lot of it is about the big clumsy library__. Some choice excerpts:
> There’s no clear design philosophy. Early PHP was inspired by Perl; the huge stdlib with “out” params is from C; the OO parts are designed like C++ and Java
There's even complaints about PHP 5.2, but PHP 5.4 had just been released a month earlier:
> If PHP was compiled with --disable-url-fopen-wrapper, it won’t work. [...] Note that this flag was removed in PHP 5.2.5.
> foo()[0] is a syntax error. (Fixed in PHP 5.4.)
> PHP errors and PHP exceptions are completely different beasts. They don’t seem to interact at all. (Fixed in PHP 7)
> There is no finally construct (Fixed in PHP 5.5)
> Methods cannot be named e.g. “list”, because list() is special syntax (Fixed in PHP 7)
I'm going to stop there. Not saying any of it wasn't valid criticism or that some of it doesn't apply to this day. Just that too many people, yourself included, are happy to throw a 6 year old blog post at a language that has continued to evolve since. It's also strange that you say it isn't criticized for a big and clumsy library, because in the link you provided about half of the page is complaints about the library.
> There’s no clear design philosophy. Early PHP was inspired by Perl; the huge stdlib with “out” params is from C; the OO parts are designed like C++ and Java
There's even complaints about PHP 5.2, but PHP 5.4 had just been released a month earlier:
> If PHP was compiled with --disable-url-fopen-wrapper, it won’t work. [...] Note that this flag was removed in PHP 5.2.5.
> foo()[0] is a syntax error. (Fixed in PHP 5.4.)
> PHP errors and PHP exceptions are completely different beasts. They don’t seem to interact at all. (Fixed in PHP 7)
> There is no finally construct (Fixed in PHP 5.5)
> Methods cannot be named e.g. “list”, because list() is special syntax (Fixed in PHP 7)
I'm going to stop there. Not saying any of it wasn't valid criticism or that some of it doesn't apply to this day. Just that too many people, yourself included, are happy to throw a 6 year old blog post at a language that has continued to evolve since. It's also strange that you say it isn't criticized for a big and clumsy library, because in the link you provided about half of the page is complaints about the library.
> I'm going through it again, and __a lot of it is about the big clumsy library__.
And even a lot of that isn't fair because PHP has always integrated C libraries directly; which was a huge strength when it was new. A function like mysql_real_escape_string isn't a PHP function, it's a MySQL function ( https://dev.mysql.com/doc/refman/5.7/en/mysql-real-escape-st... ).
And even a lot of that isn't fair because PHP has always integrated C libraries directly; which was a huge strength when it was new. A function like mysql_real_escape_string isn't a PHP function, it's a MySQL function ( https://dev.mysql.com/doc/refman/5.7/en/mysql-real-escape-st... ).
Please stop quoting this article. One third of it was wrong at the time it was written. One third is just opinion. The last third is somewhat valid but a lot of it has been fixed since then.
The author never actually used PHP and just collected a bunch of "facts" from the Internet without even testing some of these things himself.
The author never actually used PHP and just collected a bunch of "facts" from the Internet without even testing some of these things himself.
I mean....you aren't arguing JS is a paradigm of perfect design, are you?
It might be a bit controversial here, but language faults aside, I think PHP's model of package management through Composer is better in many ways than NPM, at least in terms of namespacing, and not prefering to define every function as a separate package within its own universe.
If it makes you feel better, there are plenty of us out there who hate PHP and Node :)
I agree. Just beginning work on a react project installs 300MB or so of tooling, and when you throw Electron into the mix, expect a huge node_modules folder.
When I was checking out Serverless Framework I was floored that a single Lambda function (that barely did anything) resulted in 15+Mb of data being uploaded to AWS. I had only installed two NPM modules yet when I looked at the "node_modules" directory there were nearly 300 entries in there. Things like Redux and GraphQL, neither of which I ever intended to use in a simple test of Serverless Framework.
When Node and NPM first came out and I thought, while not perfect, they were a really promising set of technologies. Now, NPM specifically is something I will go out of my way to avoid. And by virtue of the fact that NPM is so closely tied with Node, I tend to avoid Node as well. Which sucks because I have always enjoyed programming in Javascript.
If anyone has come out with a curated alternative to NPM, please let me know.
When Node and NPM first came out and I thought, while not perfect, they were a really promising set of technologies. Now, NPM specifically is something I will go out of my way to avoid. And by virtue of the fact that NPM is so closely tied with Node, I tend to avoid Node as well. Which sucks because I have always enjoyed programming in Javascript.
If anyone has come out with a curated alternative to NPM, please let me know.
Though I agree with the article, and can appreciate that somebody just "checking out Serverless" (I'm assuming you were also new to Node), it just takes a little bit of awareness when installing packages.
It took a while to learn this lesson, but it pm;u tales a few seconds in the process of deciding if you want to use a package, to check out it's dependencies, I also check out the git page and license. Is it active, is it open-source, does it have far too many dependencies, is it in regular use.
https://www.npmjs.com/package/serverless
That's serverless. I was surprised to see it uses graphql as a default, and looking at graphql, how many dependencies does it have? Just one, which has no more. So that chain ends quite quickly.
It doesn't take long looking at the dependencies to see that the same packages get used again and again throughout the community. So you'll find you won't keep going down this path once you are comfortable with the packages.
Having said all of that, it is also good practice to look at the source of a package, and decide if you could do it better and easier yourself. It is amazing how many packages are only a few lines of code, and fairly unnecessary.
It took a while to learn this lesson, but it pm;u tales a few seconds in the process of deciding if you want to use a package, to check out it's dependencies, I also check out the git page and license. Is it active, is it open-source, does it have far too many dependencies, is it in regular use.
https://www.npmjs.com/package/serverless
That's serverless. I was surprised to see it uses graphql as a default, and looking at graphql, how many dependencies does it have? Just one, which has no more. So that chain ends quite quickly.
It doesn't take long looking at the dependencies to see that the same packages get used again and again throughout the community. So you'll find you won't keep going down this path once you are comfortable with the packages.
Having said all of that, it is also good practice to look at the source of a package, and decide if you could do it better and easier yourself. It is amazing how many packages are only a few lines of code, and fairly unnecessary.
I'm kind of surprised no one has tried to make a standard library for node - although I guess underscore/lodash come close.
The package situation is a serious problem for distro maintainers, which AFAICT have basically given up packaging any npm packages the traditional way. Luckily Nix[1] and Guix[2] can most likely support automatic importing from npm.
Incidentally, one of the many things I love about typescript is that it has zero non-dev dependencies [3].
[1] https://github.com/svanderburg/node2nix [2] https://lists.gnu.org/archive/html/guix-devel/2017-03/msg008... [3] https://www.npmjs.com/package/typescript?activeTab=dependenc...
The package situation is a serious problem for distro maintainers, which AFAICT have basically given up packaging any npm packages the traditional way. Luckily Nix[1] and Guix[2] can most likely support automatic importing from npm.
Incidentally, one of the many things I love about typescript is that it has zero non-dev dependencies [3].
[1] https://github.com/svanderburg/node2nix [2] https://lists.gnu.org/archive/html/guix-devel/2017-03/msg008... [3] https://www.npmjs.com/package/typescript?activeTab=dependenc...
Is this article truncated? The conclusion or end part seems to be missing. Or is this just oddly written?
There's two distinct problems I see here, which probably plague most package managers:
1. Package naming: All the good names will be gone soon, which'll probably lead to big players trying to get package names of little players. Which will then result in broken dependencies a la left-pad. Afaik the latter has been solved in NPM [1]. The former is probably more of a usability problem, we want installation to be simple and able to install deps just by remembering their name.
2. The power of deps and our reliance on good will: When installing dependencies, we basically grant them the power to wreak any kind of havoc they want. The ways we mitigate the worst: Belief in the magical power of open source, mostly using libs that are popular (hence might've gotten a good read by an IT-sec person) and at enterprise scale some static analysis tools to get an idea of how it interacts with your system. I think we're in need of a technical solution at the package manager level, where a dependency has to explicitly ask for certain capabilities (similar to how Android apps work).
[1] http://blog.npmjs.org/post/141905368000/changes-to-npms-unpu...
1. Package naming: All the good names will be gone soon, which'll probably lead to big players trying to get package names of little players. Which will then result in broken dependencies a la left-pad. Afaik the latter has been solved in NPM [1]. The former is probably more of a usability problem, we want installation to be simple and able to install deps just by remembering their name.
2. The power of deps and our reliance on good will: When installing dependencies, we basically grant them the power to wreak any kind of havoc they want. The ways we mitigate the worst: Belief in the magical power of open source, mostly using libs that are popular (hence might've gotten a good read by an IT-sec person) and at enterprise scale some static analysis tools to get an idea of how it interacts with your system. I think we're in need of a technical solution at the package manager level, where a dependency has to explicitly ask for certain capabilities (similar to how Android apps work).
[1] http://blog.npmjs.org/post/141905368000/changes-to-npms-unpu...
1) seems to plague NPM a lot more than others.
> The former is probably more of a usability problem, we want installation to be simple and able to install deps just by remembering their name.
There's a point beyond which the desire for simplicity creates dangerous abstractions. A developer should expect to put more effort into installing deps than just remembering a package name.
> The former is probably more of a usability problem, we want installation to be simple and able to install deps just by remembering their name.
There's a point beyond which the desire for simplicity creates dangerous abstractions. A developer should expect to put more effort into installing deps than just remembering a package name.
That it plagues NPM more than others might be due to size, NPM has almost 3x as many packages as the next large module repository (Maven) [1].
[1] http://www.modulecounts.com/
[1] http://www.modulecounts.com/
Size does mean the scope of any issues with NPM is greater, but many of those issues seem unique to NPM's design and the extremely fine-grained nature of Node packages as a whole. One developer had a temper tantrum and unpublished one package and the entire Node universe caught on fire... that kind of catastrophic failure doesn't seem to happen elsewhere, even at a smaller scale.
True, that WAS a deficiency, though one that seems to be fixed now.
Given that dependencies are also resolved by name in the Ruby & Python I'd think there's no technical reason for why a similar thing wouldn't happen, is there?
Given that dependencies are also resolved by name in the Ruby & Python I'd think there's no technical reason for why a similar thing wouldn't happen, is there?
>Given that dependencies are also resolved by name in the Ruby & Python I'd think there's no technical reason for why a similar thing wouldn't happen, is there?
I don't know, I'm not familiar enough with those languages, but I would guess that if anything similar did happen, it would probably wind up posted here, although absence of evidence isn't evidence of absence.
It may also depend on implementation features besides package names - the worst that should happen if a remote dependency no longer exists is that future updates fail, but a local copy should always exist and the current build should still keep working. To me, vendoring dependencies should be the default.
I don't know, I'm not familiar enough with those languages, but I would guess that if anything similar did happen, it would probably wind up posted here, although absence of evidence isn't evidence of absence.
It may also depend on implementation features besides package names - the worst that should happen if a remote dependency no longer exists is that future updates fail, but a local copy should always exist and the current build should still keep working. To me, vendoring dependencies should be the default.
One of my least favourite things about the NPM eco-system is that it doesn't have support for name spacing, i.e. package names are global. I think this is one of the things that Composer (PHP's equivalent of NPM) does very well [1].
Personally I'm not to worried about the size of my node_modules since it doesn't reflect what gets transpiled into the target anyway, but yes, if you have a bunch of projects sitting on your hard drive, it can get a bit busy.
[1] https://packagist.org/
Personally I'm not to worried about the size of my node_modules since it doesn't reflect what gets transpiled into the target anyway, but yes, if you have a bunch of projects sitting on your hard drive, it can get a bit busy.
[1] https://packagist.org/
npm has namespacing https://docs.npmjs.com/getting-started/scoped-packages
Only NPM causes troubles. Node.js is stable and is improved quickly since the io.js merge.
You don't need to use NPM to run node programs. The binaries provided by node.js website are standalone (no dependencies): http://www.nodejs.org/dist/
You don't need to use NPM to run node programs. The binaries provided by node.js website are standalone (no dependencies): http://www.nodejs.org/dist/
Clickbait. At least propose what it is that you think needs to be done to address this so-called chaos.
Worth noting that ES2017 added String.prototype.padStart and String.prototype.padEnd. The whole left-pad thing is blown out of proportion.
Second, you shouldn't be relying on npm for deployments. You dig your own grave. If package deletion is a concern, then host a mirror with something like Nexus [0], or pay for a service [1] [2] to do it for you. Over the years there have been a few local npm mirror tools [3], but I don't think any of em have really caught much traction. There's also the option of bundling each release. For that you can use a tool such as Webpack [4], or take it one step further and generate a single binary using something like pkg [5].
Developers also have the option of forking their dependency graph and updating references in package.json. Sooner or later you'll start to find bugs in your dependencies and have to fix them. It's your responsibility, you own all that code. Don't be afraid of maintaining your own fork. If you find a critical issue you'll probably need to be able to deploy your update without depending on a third-party anyway. This is fairly common in other ecosystems. Heck, you can also just check in your node_modules folder into version control.
Don't get me wrong, npm certainly has its fair share of problems, and historically they haven't had the best defaults... But there's usually ways to work around the issues, and they're gradually working towards improving things. It's the developer's responsibility to handle their ecosystem's problems, which is also nothing new in the world of software development.
[0] https://www.sonatype.com/download-oss-sonatype
[1] https://jfrog.com/artifactory/
[2] https://inedo.com/proget
[3] https://github.com/rlidwka/sinopia
[4] https://webpack.js.org/
[5] https://github.com/zeit/pkg
Worth noting that ES2017 added String.prototype.padStart and String.prototype.padEnd. The whole left-pad thing is blown out of proportion.
Second, you shouldn't be relying on npm for deployments. You dig your own grave. If package deletion is a concern, then host a mirror with something like Nexus [0], or pay for a service [1] [2] to do it for you. Over the years there have been a few local npm mirror tools [3], but I don't think any of em have really caught much traction. There's also the option of bundling each release. For that you can use a tool such as Webpack [4], or take it one step further and generate a single binary using something like pkg [5].
Developers also have the option of forking their dependency graph and updating references in package.json. Sooner or later you'll start to find bugs in your dependencies and have to fix them. It's your responsibility, you own all that code. Don't be afraid of maintaining your own fork. If you find a critical issue you'll probably need to be able to deploy your update without depending on a third-party anyway. This is fairly common in other ecosystems. Heck, you can also just check in your node_modules folder into version control.
Don't get me wrong, npm certainly has its fair share of problems, and historically they haven't had the best defaults... But there's usually ways to work around the issues, and they're gradually working towards improving things. It's the developer's responsibility to handle their ecosystem's problems, which is also nothing new in the world of software development.
[0] https://www.sonatype.com/download-oss-sonatype
[1] https://jfrog.com/artifactory/
[2] https://inedo.com/proget
[3] https://github.com/rlidwka/sinopia
[4] https://webpack.js.org/
[5] https://github.com/zeit/pkg
>It's the developer's responsibility to handle their ecosystem's problems, which is also nothing new in the world of software development.
This is where you have to start questioning the benefits of using node/npm/js on the backend. We're all stuck with it on the front end for the foreseeable future.
I've never had any of the problems in node/npm using NuGet/C# for example.
At some point, I raised my hands and emailed my boss about a new project using node, essentially saying "this is dumb. left-pad is dumb. is-number is dumb. is-odd is dumb. the amount of packages depending on these is dumb. And I won't be arsed to fix it because I value my time, so I'd rather just jump back to an ecosystem that has its shit together because the result is the same simple API pushing json either way we go."
node/npm sucks IMO. Sorry, but the problems I have to fix daily are not interesting, and conquering the mountain and being able to say I fixed all those every-day problems with it is not worth much on the job market.
If the amount of web backends suddenly needs node because they suddenly stop pushing json all day long, because node/npm is the only ecosystem that offers that solution, then you have a compelling reason. But other languages do json APIs just fine right now and have less ecosystem paper cuts.
This is where you have to start questioning the benefits of using node/npm/js on the backend. We're all stuck with it on the front end for the foreseeable future.
I've never had any of the problems in node/npm using NuGet/C# for example.
At some point, I raised my hands and emailed my boss about a new project using node, essentially saying "this is dumb. left-pad is dumb. is-number is dumb. is-odd is dumb. the amount of packages depending on these is dumb. And I won't be arsed to fix it because I value my time, so I'd rather just jump back to an ecosystem that has its shit together because the result is the same simple API pushing json either way we go."
node/npm sucks IMO. Sorry, but the problems I have to fix daily are not interesting, and conquering the mountain and being able to say I fixed all those every-day problems with it is not worth much on the job market.
If the amount of web backends suddenly needs node because they suddenly stop pushing json all day long, because node/npm is the only ecosystem that offers that solution, then you have a compelling reason. But other languages do json APIs just fine right now and have less ecosystem paper cuts.
I personally recommend caching package tarballs locally inside your project [0], using either Yarn's "offline-mirror" feature, or if you're using NPM, a tool called Shrinkpack [1]. Might not be the best fit for every project, it but it guarantees reproduceable installs without having to go out to NPM's servers.
[0] http://blog.isquaredsoftware.com/2017/07/practical-redux-par...
[1] https://github.com/JamieMason/shrinkpack
[0] http://blog.isquaredsoftware.com/2017/07/practical-redux-par...
[1] https://github.com/JamieMason/shrinkpack
It's 2018 and still leftpad issue, really?
The article says that you can still unpublish packages from NPM at will. If that's still the case, then yes, leftpad should still be brought up in 2018 before someone unpublishes the "is-string-empty" package or whatever the dumb packages that people put on NPM are.
you can only within 24 hours of publishing. After that you need to send them an email to request it and they determine if it can be unpublished or not.
Not true https://status.npmjs.org/incidents/41zfb8qpvrdj
"We have restored all the package-versions for the 9 packages that were published over"
Full context here https://github.com/npm/registry/issues/255
"We have restored all the package-versions for the 9 packages that were published over"
Full context here https://github.com/npm/registry/issues/255
The problem relies in the obsession with monolithic design even if it ends up being a complete house of cards. This leads many new developers to sacrifice basic logic and first principles when judging what good software is; often resulting in products that no one uses and that just crash anyway. But hey, you get this great toy-like model to show on your next interview.
Security and access rights are sadly not taught as being essential to having a proper monolith because they'd greatly slow down development time. Whereas an engineer or builder has no choice but to confine their creativity within those principles because a product crash for them has much more dire consequences.
Relevant article: https://hackernoon.com/im-harvesting-credit-card-numbers-and...