Makefiles for web work(rosszurowski.com)
rosszurowski.com
Makefiles for web work
https://rosszurowski.com/log/2022/makefiles
18 comments
I generally find shell/npm scripts work until you get to the few cases that actually benefit from the dependency management and then you wish you'd just given in to .PHONY, no matter how inelegant it may be.
All of my dependency management issues are handled by npm.
The main problem `make` solved was not wasting work rebuilding dependencies that hadn't changed, but in 2023 on modern computers I don't notice the CPU cycles burned on that kind of thing.
(... if I do start to notice them, I reach for a tool like bazel, because it can handle spaces in filenames).
The main problem `make` solved was not wasting work rebuilding dependencies that hadn't changed, but in 2023 on modern computers I don't notice the CPU cycles burned on that kind of thing.
(... if I do start to notice them, I reach for a tool like bazel, because it can handle spaces in filenames).
[deleted]
In our case we have most of the processes that are run by make tee their output into a project log directory. Then the make rules can use the log file itself as the dependency for the target. Works great in a lot of cases, and, you get the added benefit of being able to easily go back and observe previous process output and compare it across projects.
Oh wow, I can’t believe I’ve never thought of that. Simple and useful and a bit more elegant than having a phone output file that I manually `touch`.
The [Just] task runner can be what you want: it's simpler than Make, with less magic, but it allows to declare task dependencies.
[Just]: https://github.com/casey/just
[Just]: https://github.com/casey/just
I think .PHONY is mostly fine, but as soon as you have any sort of runtime arguments or interactive stdin/stdout you are definitely using the wrong tool.
I think the reason for the .PHONY is just so people don't have to ./
I generally reach for a shell script with "case" instead, though. Can always invoke make from the shell script.
I generally reach for a shell script with "case" instead, though. Can always invoke make from the shell script.
I tried repeatedly to just stick with Makefiles but I kept having to add workaround and hack to do basic stuff. They got uglier and uglier. I checked out go task and cargo make, but eventually gave in to `just`. It has worked just fine and the files are infinitely more readable than the comparable ones w/ make.
What kind of workarounds and hacks? Make's task interface is essentially "anything you can do at the command line", generally with the same syntax (though you have to put extra parens around your variable names).
It's true that more modern tools have features make doesn't (especially with regard to manipulating complicated dependency graphs) or attack parts of the problem that make doesn't (hermetic build environments, language-aware features like package management).
But... I can't see "lack of ability to do basic stuff" as one of its problems. What basic things are you trying to do?
It's true that more modern tools have features make doesn't (especially with regard to manipulating complicated dependency graphs) or attack parts of the problem that make doesn't (hermetic build environments, language-aware features like package management).
But... I can't see "lack of ability to do basic stuff" as one of its problems. What basic things are you trying to do?
I see `.PHONY` as a hack, `MAKEFLAGS += --no-builtin-rules` as a workaround, and many of the builtin functions as being difficult to use, read, and understand. A long string of nested replacements generating the sources for a task is a pain to read and maintain.
What I want out of task runners nowadays is to run tasks. If I can't make a task without writing '.PHONY' - there's a problem with the task runner.
What I want out of task runners nowadays is to run tasks. If I can't make a task without writing '.PHONY' - there's a problem with the task runner.
Meh. OK. Though almost always, "run tasks" implies some level of state. Are you tasks truly all idempotent and parallelizable? If not, maybe they should produce output and be tracked by dependencies? If so, then you probably have bugs in your "task runners" that will show up in weird ways.
It's absolutely true that ".PHONY" looks like a hack; it was sort of meant to. In general you shouldn't be using it, except maybe as a facade layer where you can put an "API" (c.f. the "all" or "clean" targets) on top of things that are themselves proper dependencies.
I'm not going to hold up make as the ultimate expression of a dependency-based build system. But I will say that history has a LONG trail of products that tried to replace it with decidedly mixed results. Categorical statements like yours tend to trigger my "code smell" layer, most of the time attempts to replace make produce worse results.
It's absolutely true that ".PHONY" looks like a hack; it was sort of meant to. In general you shouldn't be using it, except maybe as a facade layer where you can put an "API" (c.f. the "all" or "clean" targets) on top of things that are themselves proper dependencies.
I'm not going to hold up make as the ultimate expression of a dependency-based build system. But I will say that history has a LONG trail of products that tried to replace it with decidedly mixed results. Categorical statements like yours tend to trigger my "code smell" layer, most of the time attempts to replace make produce worse results.
> Though almost always, "run tasks" implies some level of state.
Yes! But isn't one of the major design principles of Make that the host filesystem is the container for the state? (iirc Make decides when to re-run rules based on when the timestamp on a target is older than the timestamp on an input file)
Yes! But isn't one of the major design principles of Make that the host filesystem is the container for the state? (iirc Make decides when to re-run rules based on when the timestamp on a target is older than the timestamp on an input file)
Sure, so where does that state get stored in a pure "task runner" tool? Attempts to muck with this metaphor ("state is what is left behind when the tool is not running") are among the worst mistakes made by build systems. It's a feature, not a bug, and everyone who believes otherwise ends up rediscovering it the hard way.
GNU Make only recently added any support for rules which produce multiple targets.
I haven't played around with the make-alikes, but I have also taken to using make for this sort of thing.
My blog, for example, has make targets for things like making a production build, publishing, cleaning out generated artifacts, and running a dev build (with drafts) + bringing up the test server.
Last year I wrote a little post about how I also use make + nix-shell to supply dependencies for Makefile tasks without needing to have them on my PATH: https://t-ravis.com/post/nix/nix-make/
My blog, for example, has make targets for things like making a production build, publishing, cleaning out generated artifacts, and running a dev build (with drafts) + bringing up the test server.
Last year I wrote a little post about how I also use make + nix-shell to supply dependencies for Makefile tasks without needing to have them on my PATH: https://t-ravis.com/post/nix/nix-make/
Just [0] was discussed on HN. I think I'll give that a shot next time I have to write something like this. .PHONY is lame.
[0] https://news.ycombinator.com/item?id=34315779
[0] https://news.ycombinator.com/item?id=34315779
I strongly recommend `just`. I used it successfully with projects written in OCaml, Rust, Go, Elixir and Lua. Define a few tasks, give them 1-2 character aliases, profit.
Super ergonomic.
Though I have to admit, Go and Rust projects hardly needed `just`; the `go` program and the `cargo` tool are that good. I used `just` in them mostly to have short aliases and common names for tasks.
Super ergonomic.
Though I have to admit, Go and Rust projects hardly needed `just`; the `go` program and the `cargo` tool are that good. I used `just` in them mostly to have short aliases and common names for tasks.
I really like how easy it is with Just to make it self-documenting.
There's some good things about Makefiles, yes, but there's also a lot of weird legacy functionality. Make was meant to create files and directories, and the .PHONY functionality seems to be a workaround added to make it usable as a general-purpose task runner.
There's some good things about Makefiles, yes, but there's also a lot of weird legacy functionality. Make was meant to create files and directories, and the .PHONY functionality seems to be a workaround added to make it usable as a general-purpose task runner.
Make becomes a pretty terrible tool at a fairly low level of complexity. I would never choose it voluntarily to build things.
Yeah, think of how programming languages invent and reinvent stuff to handle the interaction with Make and native libraries, how we have generators of generators of Makefiles... Heck, even Docker's use case is mostly because of the living hell that is C stuff.
Yeah the absurdity of autoconf tooling is enough evidence by itself
I agree. It has its merits, but it's hard for me to get around my visceral reaction to the fact it just bombs on filenames with spaces in them.
You can escape spaces in makefiles.
Sure... but if you have spaces in filenames, and filenames in variables, what happens then?
"can" or "have to"?
It is at least much better than having tasks and their dependencies merely specified as plain strings in a package.json file, as unstructured data, out of which the tool (npm) does not have any knowledge about dependencies and simply runs all strings as commands. This is done by countless web projects and it is embarrassing. I would rather have a package.json script that calls Make, than define my steps in the package.json itself.
My main problem with make: When you want to write sophisticated build processes, the syntax is bizarre and difficult to work with.
True, but having "sophisticated" build processes are a problem in their own right. I understand that there may be special cases, but generally, if you have troubles making a build process work with Make, then using a "better" tool is just paving over the underlying problem.
Just today I was trying to help a friend build some tool with Bazel. While I'm sure there's a reason for Bazels existence, but it's just way to complex. The build failed and debugging the Bazel config was just a major hassle compared to had we just looked at the Java commands in a Makefile.
Similar with Pythons setuptools, even though that is significantly easier with the new pyproject.toml. It's really powerful, but what all I wanted is basically to copy a bunch of files and add a bit of metadata.
Just today I was trying to help a friend build some tool with Bazel. While I'm sure there's a reason for Bazels existence, but it's just way to complex. The build failed and debugging the Bazel config was just a major hassle compared to had we just looked at the Java commands in a Makefile.
Similar with Pythons setuptools, even though that is significantly easier with the new pyproject.toml. It's really powerful, but what all I wanted is basically to copy a bunch of files and add a bit of metadata.
Not only that, but look at any advanced-age project that uses make. There are layers of includes, and defines and rules are smeared across multiple files.
I think many modern web-build tools were made by people who thought "hey, I can do better than make" (or worse, "What's make?" and reinvented the wheel) and then wound up in a similar conundrum.
I think many modern web-build tools were made by people who thought "hey, I can do better than make" (or worse, "What's make?" and reinvented the wheel) and then wound up in a similar conundrum.
The creator of Make actually apologized for it, as far as I have read.
I'm very happy with Make, though I have to agree, the syntax could be better. I don't mind using tabs, but why are spaces not OK as well? It would also be good to be able to not have to break long lines or multiline things with backslashes all the time. Yes you can set that oneshell thingy, but that is inconvenient in other ways, because it is for the whole file instead of a single command or a single target.
So it could be better, definitely, but it does the job of doing DAGs well and it is available on most systems, so I don't have to add some huge tree of dependencies just to run tasks.
I'm very happy with Make, though I have to agree, the syntax could be better. I don't mind using tabs, but why are spaces not OK as well? It would also be good to be able to not have to break long lines or multiline things with backslashes all the time. Yes you can set that oneshell thingy, but that is inconvenient in other ways, because it is for the whole file instead of a single command or a single target.
So it could be better, definitely, but it does the job of doing DAGs well and it is available on most systems, so I don't have to add some huge tree of dependencies just to run tasks.
Totally agree with the author. As someone who jumps between multiple projects, in different languages-I have hard time remembering what build/test/script runner in used in particular project. Hell, even two JS/TS projects can use different tools (one could use npm with jest, the other yarn with webpack, etc).
I written[1] a blog on that as well. Glad to see there are more like minded people out there.
[1] https://www.yieldcode.blog/post/why-you-should-adpot-makefil...
I written[1] a blog on that as well. Glad to see there are more like minded people out there.
[1] https://www.yieldcode.blog/post/why-you-should-adpot-makefil...
This is sweet but `just` is even sweeter - can use multistage build files to elegantly add it to any Containerfile, no?
Edit: I don't see a publicized image - added an issue: https://github.com/casey/just/issues/1497
However, anyone could add a docker just for just for their own project and use it to pull in just for all their projects, avoiding a download command to get just in other Containerfiles.
Edit: I don't see a publicized image - added an issue: https://github.com/casey/just/issues/1497
However, anyone could add a docker just for just for their own project and use it to pull in just for all their projects, avoiding a download command to get just in other Containerfiles.
I realize that I have been using shell scripts for this. I just copy-paste commands and add a selector. It has the benefit that I didn't need to learn a new syntax.
`make` is not a taskrunner, it is for tracking dependencies between files. a .PHONY target that isn't named "all", "clean", or "dist" is a code smell. there are many design decisions in `make` that are the opposite of what you want in a taskrunner:
* each line executes in an isolated environment
* can't pass options or arguments to a make target
* not portable to other operating systems
the author is already using node packages. purpose-built taskrunners are plentiful, and the resulting scripts will be written in the same language as the rest of the codebase instead of adding a second language.
hiding a tool behind a makefile is my pet peeve. a proper taskrunner makes common operations easy without kneecapping the flexibility of the underlying tools.
* each line executes in an isolated environment
* can't pass options or arguments to a make target
* not portable to other operating systems
the author is already using node packages. purpose-built taskrunners are plentiful, and the resulting scripts will be written in the same language as the rest of the codebase instead of adding a second language.
hiding a tool behind a makefile is my pet peeve. a proper taskrunner makes common operations easy without kneecapping the flexibility of the underlying tools.
I've done things this way for many years but now that I'm working with rust I find myself doing more with cargo instead.
Compared to make, one thing I like about cargo is I don't need to worry about the current directory as much. e.g. "cargo check" works from anywhere in my workspace but something like "make check" will fail if the Makefile isn't in the current directory.
Yes I know I could probably create an alias, direnv or something to improve make's behavior but I'd rather not have more things to manage. All this stuff is too complicated as it is. Overall the cargo defaults are much more ergonomic.
Compared to make, one thing I like about cargo is I don't need to worry about the current directory as much. e.g. "cargo check" works from anywhere in my workspace but something like "make check" will fail if the Makefile isn't in the current directory.
Yes I know I could probably create an alias, direnv or something to improve make's behavior but I'd rather not have more things to manage. All this stuff is too complicated as it is. Overall the cargo defaults are much more ergonomic.
Just making a bunch of phony rules is writing a shell script with extra steps.
How about converting electron-forge-webpack-ts to make. That would be epic.
How about converting electron-forge-webpack-ts to make. That would be epic.
I’m unconvinced. I don’t see a reasonable benefit for learning yet another config syntax just to run some scripts.
That's specifically the gain here, you don't need to keep learning new build tools or language-specific tools. Make works for all languages, all kinds of projects, has been around since forever and is not going away. It's the only build tool you need to learn.
`just` does the same but better, is much quicker to learn (we're talking literal 5-10 minutes), and doesn't have decades of weird syntax baggage.
By the time its close to makes popularity, it will.
I went through my Makefile stage. It can be nice but it can also get messy and ugly. If you're doing simple tasks fine but your project still needs to have a lot things anyways that negates its usefulness.
I tried this for a bit until I found out make couldn't deal with spaces in the file name. switched to rake (ruby), which had the advantage of not needing to write separate shell scripts.
Why do you have spaces in the filenames of your project in the first place? They require annoying escapes to work with essentially everywhere, notably including on the web. You have complete control over the filenames in your project... why are actively choosing to use a space character? I can see a better argument for wanting to use an apostrophe in a filename than a space character :(.
Because I wanted the title of my blog posts to match the file name, so I didn't have to remember which markdown file was which post.
Which sounds cool and all, except I bet the URL doesn't have a space character in it, right? And so, now you have a weird mismatch between the files on disk and the site structure.
No, the filename was a sort of composite key.
"yymmdd Name of Article.md"
URL became yymmdd. Name of Article let me know where to find the article at a glance.
"yymmdd Name of Article.md"
URL became yymmdd. Name of Article let me know where to find the article at a glance.
>People routinely point out that npm/yarn scripts are shockingly slow to start
The claim that 157 ms and 126 ms is "shockingly slow" is quite an exaggeration.
The claim that 157 ms and 126 ms is "shockingly slow" is quite an exaggeration.
It's not an exaggeration when you inevitably end up chaining them. The overhead adds up very quickly and becomes noticeable.
Also they take even longer to start in CI/CD containers.
Also they take even longer to start in CI/CD containers.
true, but the moment you add containers to your mix you're functionally admitting that you don't care about startup times. Or possibly, you've been forced to do it by some service provider who doesn't give you a better option...
I mean you're not wrong in general, it's just that if the tool is written in C/C++, Zig, Rust, Golang, OCaml, D, V, and a few others, it still start in maximum 20ms even in containers.
126 ms is already well above the threshold of latency you can expect to notice when typing at a keyboard. By comparison, it'd almost be a long enough time to start caring as a TTFB. It really is a shockingly long time.
Before a couple of years ago, I had never written a Makefile in my life.
I ended up using Make after experiencing web project builder merry-go-round du jour of Grunt, Gulp, npm/yarn scripts, WebPack, etc.
Of course I had read all of the Make FUD… but like almost all things in computing (Vim vs. Emacs, tabs vs. spaces, Mac vs. Windows) you can't take it too seriously.
For me, similarly to when I went through several editors before I discovered Vim, Make has been around long enough and people have used it to address the hairiest build/dependency issues that many of the new tools have yet to address.
Let me address a few common issues.
IMHO the syntax isn't that bad. Like almost any programming language, you can write a clean, well-documented Makefile following best practices or you can slap something together with no planning and have an indecipherable page of junk.
And like many languages, many folks never learn to write Makefiles properly. I get it—most Makefiles in the wild are close to unreadable and are mostly undocumented.
And because Make has been around since the '70s, there's lots of hard-earned wisdom in blog posts and forums readily available going back decades.
O’Reilly's book "Managing Projects with GNU Make" (3rd edition) does a great job at getting someone up and running with Make. And the online documentation [1] is also quite good.
But the most important thing is I'm much more productive using Make instead of messing around with the various JavaScript build tools and their plugins. It's easy to integrate JavaScript, command line, Ruby, etc. tools with Make to get my projects built, linted, tested, packaged and deployed.
And having native support for Make in Vim/Neovim [2] is just the cherry on top. And it's still under active development [3] with bug fixes and new features.
Perhaps the best thing about Make: there's no dependency graph/build web project issue it can't handle. Writing clean, readable and well-documented may take a little more effort but it's worth it IMHO.
[1]: https://www.gnu.org/software/make/
[2]: https://vimways.org/2018/runtime-hackery/
[3]: https://lwn.net/Articles/913253/
I ended up using Make after experiencing web project builder merry-go-round du jour of Grunt, Gulp, npm/yarn scripts, WebPack, etc.
Of course I had read all of the Make FUD… but like almost all things in computing (Vim vs. Emacs, tabs vs. spaces, Mac vs. Windows) you can't take it too seriously.
For me, similarly to when I went through several editors before I discovered Vim, Make has been around long enough and people have used it to address the hairiest build/dependency issues that many of the new tools have yet to address.
Let me address a few common issues.
IMHO the syntax isn't that bad. Like almost any programming language, you can write a clean, well-documented Makefile following best practices or you can slap something together with no planning and have an indecipherable page of junk.
And like many languages, many folks never learn to write Makefiles properly. I get it—most Makefiles in the wild are close to unreadable and are mostly undocumented.
And because Make has been around since the '70s, there's lots of hard-earned wisdom in blog posts and forums readily available going back decades.
O’Reilly's book "Managing Projects with GNU Make" (3rd edition) does a great job at getting someone up and running with Make. And the online documentation [1] is also quite good.
But the most important thing is I'm much more productive using Make instead of messing around with the various JavaScript build tools and their plugins. It's easy to integrate JavaScript, command line, Ruby, etc. tools with Make to get my projects built, linted, tested, packaged and deployed.
And having native support for Make in Vim/Neovim [2] is just the cherry on top. And it's still under active development [3] with bug fixes and new features.
Perhaps the best thing about Make: there's no dependency graph/build web project issue it can't handle. Writing clean, readable and well-documented may take a little more effort but it's worth it IMHO.
[1]: https://www.gnu.org/software/make/
[2]: https://vimways.org/2018/runtime-hackery/
[3]: https://lwn.net/Articles/913253/
(I generally find a small set of shell scripts work, or just three or four rules in the package.json file I can access with `npm run`).