Let's Package JQuery: A JavaScript Packaging Dystopian Novella(dustycloud.org)
dustycloud.org
Let's Package JQuery: A JavaScript Packaging Dystopian Novella
http://dustycloud.org/blog/javascript-packaging-dystopia/
7 comments
Honest question, not trolling: why would I want to install a front-end JS lib using a system package manager? What is the use-case for doing this?
Self-hosted web apps (the linked post mentions MediaGoblin). The idea is that a user can run 'apt-get/yum install mediagoblin' on his home computer and have a personal media server.
Deduplication, reproducibility, using only a single package manager instead of a handful, ec.
From the article: "Our deployment and build setups have gotten so complicated that I doubt anyone really has a decent understanding of what is going on, really."
There's now a school of thought in the Rust world that it's easier to just build a static executable and ship that.
There's now a school of thought in the Rust world that it's easier to just build a static executable and ship that.
I think Go also follows this philosophy.
> It doesn't help that, for a long time, the
> status quo in all free software web applications
> (and indeed all web applications) was to check
> javascript and similar served-to-client web
> assets straight into your repository.
Operating systems trying to deal with managing applications dependencies has been a huge source of fustration for me. You cant have jquery randomly upgraded for your application without having tested it, that is always going to cause problems no matter how religious you are with semver.
Anyone who has developed erlang on ubuntu can attest to how ridiculous the situation can get (debian split erlang up into seperate packages).
I very much lean towards having my dependencies bundled. I have misgivings with npm but it does a lot of things right in that regard.
I fail to see the problem. Just create a .tar.gz with all the dependencies installed and assets compiled, which you can distribute to users that don't care about or don't want to mess around with the build tools.
If I want to build a native application (like git, which I often do) from source, it's not like the situation is much better. I have to install the compiler obviously, and I have to manually install the dependencies.
If I want to build a native application (like git, which I often do) from source, it's not like the situation is much better. I have to install the compiler obviously, and I have to manually install the dependencies.
You usually don't have to manually install a compiler and all the dependencies. If you want to build a package from source which has already been packaged by the distribution, it's usually quite easy to do so. For example, in Arch Linux you can run
yaourt -Sb git
and it will get the PKGBUILD file for the package and allow you to edit it before building so you can build a newer version or alter build flags. When building a package this way, it will also fetch all of the packages it depends on in order to build and run. You generally don't have to think about dependencies at all.
In Ubuntu/Debian and Fedora/CentOS it is several more commands, although there are probably scripts that do it in a single command. It's still a lot easier than manually hunting dependencies and figuring out how exactly to build a package.
yaourt -Sb git
and it will get the PKGBUILD file for the package and allow you to edit it before building so you can build a newer version or alter build flags. When building a package this way, it will also fetch all of the packages it depends on in order to build and run. You generally don't have to think about dependencies at all.
In Ubuntu/Debian and Fedora/CentOS it is several more commands, although there are probably scripts that do it in a single command. It's still a lot easier than manually hunting dependencies and figuring out how exactly to build a package.
In Ubuntu/Debian and Fedora/CentOS it is several more commands
In Debian & derivatives it's just "apt-get build-dep git && apt-get source --build git".
In Debian & derivatives it's just "apt-get build-dep git && apt-get source --build git".
"265 unique packages, all to build jquery!"
Oh my goodness.
Oh my goodness.
To be fair, Node/Javascript prefers to have many small packages that do just one thing (hopefully well).
Yeah, I have several modules on NPM that are each just a single function (with unit tests!) that I'm interested in using in more than one package. "265 unique packages" could possibly be "265 short functions" on one extreme end.
To what end? It seems like the package overhead would destroy any savings in package size to be able to mix and match rather than just having a larger, more complete package.
The alternative isn't having a single more complete package, it's having a few dozens of more complete, overlapping packages that each don't fully satisfy your needs.
The overhead is negligible: running `npm install` in production is considered an anti-pattern and because of how npm's dependency resolution works each dependency can have different copies of the same (second order) dependency. The likelihood that there will be version conflicts (and thus duplication) is far smaller when there are smaller more specialised packages instead of a few big ones.
It's not about package size. Nobody cares about package size (except for frontend code but it's fairly easy to pare those deps down to the bare minimum).
Also, nearly all of jQuery's deps are development deps not needed in production. Those deps are only interesting for maintainers -- building jQuery yourself is likely the wrong thing to do. It's only necessary because jQuery doesn't yet expose its "submodules" properly.
The overhead is negligible: running `npm install` in production is considered an anti-pattern and because of how npm's dependency resolution works each dependency can have different copies of the same (second order) dependency. The likelihood that there will be version conflicts (and thus duplication) is far smaller when there are smaller more specialised packages instead of a few big ones.
It's not about package size. Nobody cares about package size (except for frontend code but it's fairly easy to pare those deps down to the bare minimum).
Also, nearly all of jQuery's deps are development deps not needed in production. Those deps are only interesting for maintainers -- building jQuery yourself is likely the wrong thing to do. It's only necessary because jQuery doesn't yet expose its "submodules" properly.
What package overhead? With NPM, all a package needs is a "package.json" file which does little more than specify a name and version number.
And what would a more complete package look like? "AgentME's bag of useful functions"? Many of them don't make sense to group together.
And what would a more complete package look like? "AgentME's bag of useful functions"? Many of them don't make sense to group together.
But the packaging burden is the same, regardless of the size of the source code. There's still 265 unique dependencies for a distro to package to build jQuery from source in its entirety. That's rough.
Nope. There's nothing rough about
It would be just as ridiculous to complain about how hard it is to build ffmpeg from source by using npm as the package manager.
npm install jquery
It would be just as ridiculous to complain about how hard it is to build ffmpeg from source by using npm as the package manager.
Having a build process tied to a specific package manager is a big problem.
It's not a specific package manager. It's the package manager. You use Node.js (or io.js, which is effectively the same as Node.js) to build jQuery, the Node.js package manager is NPM. jQuery is an NPM package and uses NPM's package.json to define its development dependencies.
This isn't really up to debate. There are other package managers for JS, but they aren't replacements for NPM. And no, Meteor doesn't count -- Meteor is its own strange quasi-incompatible microcosm. In fact, some projects which aren't node packages use NPM just to manage development dependencies.
This isn't really up to debate. There are other package managers for JS, but they aren't replacements for NPM. And no, Meteor doesn't count -- Meteor is its own strange quasi-incompatible microcosm. In fact, some projects which aren't node packages use NPM just to manage development dependencies.
I don't know how many other ways to say this: Downloading dependencies is not the job of a build system! One should be able to use npm to fetch dependencies, or get them via another means (your system package manager), and the build system shouldn't care. Coupling package managers to build systems is a mistake.
You can use any build system you want, but you need to have the build system's dependencies installed and those dependencies vary from project to project.
If you're using Makefiles, you implicitly depend on make. If you're using Grunt, you're likely implicitly depending on grunt-cli. NPM allows you to make these dependencies explicit (if the dependencies can be resolved to other NPM packages).
Heck, you can install node packages globally and be able to use them as if they were installed locally.
I don't know how many other ways to say this: what is your problem? As long as your package manager knows how to make the packages visible to node, you don't need NPM as a package manager. But you likely won't want to put all NPM packages in your package manager of choice's repository because there's a metric fuckton of them and system-level package managers have a habit of being horrendously out of date for application-level dependencies.
With regard to dependencies, every node project folder is like virtualenv. And as a matter of fact, every Python developer I know considers the system-level Python packages (i.e. those installed with apt-get or your PM of choice) a pain in the ass and only favours them if installing the package with pip would require a lot of native dev libraries (e.g. the MySQL driver or the XML/HTML packages).
The only thing making NPM a "build system" is its "npm run" command which lets you run the scripts defined in your package.json. And for many projects having a couple of aliases is all you really need and Grunt/Gulp/Make/Jake/Rake is overkill.
Either you have no idea what you are talking about or you're very bad at communicating what you take issue with.
If you're using Makefiles, you implicitly depend on make. If you're using Grunt, you're likely implicitly depending on grunt-cli. NPM allows you to make these dependencies explicit (if the dependencies can be resolved to other NPM packages).
Heck, you can install node packages globally and be able to use them as if they were installed locally.
I don't know how many other ways to say this: what is your problem? As long as your package manager knows how to make the packages visible to node, you don't need NPM as a package manager. But you likely won't want to put all NPM packages in your package manager of choice's repository because there's a metric fuckton of them and system-level package managers have a habit of being horrendously out of date for application-level dependencies.
With regard to dependencies, every node project folder is like virtualenv. And as a matter of fact, every Python developer I know considers the system-level Python packages (i.e. those installed with apt-get or your PM of choice) a pain in the ass and only favours them if installing the package with pip would require a lot of native dev libraries (e.g. the MySQL driver or the XML/HTML packages).
The only thing making NPM a "build system" is its "npm run" command which lets you run the scripts defined in your package.json. And for many projects having a couple of aliases is all you really need and Grunt/Gulp/Make/Jake/Rake is overkill.
Either you have no idea what you are talking about or you're very bad at communicating what you take issue with.
build and test jquery
many of those dependencies are testing related including some large dependencies for virtual dom stuff to mock the browser. Plus it uses grunt which, while I don't really like it, it does have the benefit over make that it works out of the box on windows which is something jquery has to support.
many of those dependencies are testing related including some large dependencies for virtual dom stuff to mock the browser. Plus it uses grunt which, while I don't really like it, it does have the benefit over make that it works out of the box on windows which is something jquery has to support.
To always use the same versions of every dependency you'll want to shrink wrap.
And yes, of course dependencies have dependencies. Software depends on other software. All packaging systems have a tree of dependencies.
And yes, of course dependencies have dependencies. Software depends on other software. All packaging systems have a tree of dependencies.
>To always use the same versions of every dependency you'll want to shrink wrap.
Does "shrink wrap" mean "bundle"? If so, I strongly disagree. It's possible to explicitly use the exact same versions of dependencies without resorting to bundling all the source/binaries. Guix and Nix are particularly well suited for this.
>And yes, of course dependencies have dependencies.
But you have to admit that the depth of this dependency tree is insane.
Does "shrink wrap" mean "bundle"? If so, I strongly disagree. It's possible to explicitly use the exact same versions of dependencies without resorting to bundling all the source/binaries. Guix and Nix are particularly well suited for this.
>And yes, of course dependencies have dependencies.
But you have to admit that the depth of this dependency tree is insane.
I mean npm shrinkwrap.
Depth of tree is reasonable: if you install them at the top.
Depth of tree is reasonable: if you install them at the top.
does it mean that if I have 10 modules that depends on exactly the same version of another module (e.g. dependent 1.1.1).. npm will download 1.1.1 10 times? Wouldn't it just use the single package with the correct and share it across? Or the cache is main use to copy the dependent module 10 times.
The same version of the same package isn't ever downloaded multiple times. That's cached. Packages may be installed in multiple locations though.
NPM doesn't reinstall dependencies that are already above it in the tree. If A depends on B and C, and B depends on (an overlapping version range of) C and D, then B and C will be installed first in A's node_modules directory, and then NPM will install the dependencies of B and C in their own node_modules directories, except that it will avoid installing a copy of C in B's node_modules directory because it already exists in a higher folder.
However, if B and C both depend on D, then NPM will install D in each's node_modules folders. You can run `npm prune` (or `npm dedupe`? I can't remember) to make it lift D up a folder so it only needs one copy.
NPM doesn't reinstall dependencies that are already above it in the tree. If A depends on B and C, and B depends on (an overlapping version range of) C and D, then B and C will be installed first in A's node_modules directory, and then NPM will install the dependencies of B and C in their own node_modules directories, except that it will avoid installing a copy of C in B's node_modules directory because it already exists in a higher folder.
However, if B and C both depend on D, then NPM will install D in each's node_modules folders. You can run `npm prune` (or `npm dedupe`? I can't remember) to make it lift D up a folder so it only needs one copy.
I think `dedupe` is now (i.e. in the most recent version of NPM) part of the install process, actually. So the only situation in which you end up with two copies of the same dependency is if two of your dependencies depend on incompatible versions of the same sub-dependency.
It's quite painful trying to package npm modules in a way that's suitable for distributions like Red Hat/Fedora/Debian. The dependency tree just goes on and on and on, and there are versioning issues all over the shop. And if you want to run test suites in the package build process then you'd better package mocha, tap, tape, nodeunit, should, vows, expresso, jasmine, supertest etc. And all of their dependencies too. And make sure every test in the test suites for every module pass. By the end of it all, you're a broken, hollow shell of the person you were before, but at least now it's possible to run "yum install js-jquery" and get version 2.1.3 :-)
[1]: https://fedoraproject.org/wiki/Changes/jQuery