Ask HN: Do we need another build system?
14 comments
It doesn't matter if you need another. You will receive one, anyway. There can never be enough build systems.
It's an inevitable consequence of two basic natural laws. First, every development team will eventually come to dislike some core feature of their current build system. This is true regardless of the build system. Second, developers like to scratch itches.
It's an inevitable consequence of two basic natural laws. First, every development team will eventually come to dislike some core feature of their current build system. This is true regardless of the build system. Second, developers like to scratch itches.
Third, merger/acquisitions will lead to multiple CICD pipelines that nobody will want to consolidate
Yes, we need 83714 build systems, no more, no less. That's the only way to improve the current software situation in the world. Just joking.
Build systems are only worth it if they are tiny and already included in the repository (or so common that everyone has them, like make) so that when I build the project, all code necessary to build the project is built (please no download) and executed.
Something like: build.sh executes `cc build.c -o build && ./build` which then builds the software; it's also pretty important that the build system is written in the same language as the rest of the project so that you don't add another dependency just to be able to build. The question is just whether you need that. Is your software really so complex that you can't just build it with a small script or Makefile?
Build systems are only worth it if they are tiny and already included in the repository (or so common that everyone has them, like make) so that when I build the project, all code necessary to build the project is built (please no download) and executed.
Something like: build.sh executes `cc build.c -o build && ./build` which then builds the software; it's also pretty important that the build system is written in the same language as the rest of the project so that you don't add another dependency just to be able to build. The question is just whether you need that. Is your software really so complex that you can't just build it with a small script or Makefile?
> it's also pretty important that the build system is written in the same language as the rest of the project so that you don't add another dependency just to be able to build
I tend to believe this, but to me the issue with all build systems is the point where you need to build multiple languages in a single project.
For this, there aren’t a lot of great options, and the ones that exist are complex. Maybe they need to be complex, but that lowers the uptake of those tools.
I tend to believe this, but to me the issue with all build systems is the point where you need to build multiple languages in a single project.
For this, there aren’t a lot of great options, and the ones that exist are complex. Maybe they need to be complex, but that lowers the uptake of those tools.
It seems weird to single out this person's project.
I don't think build systems have peaked yet, so it's good that there are people willing to explore and experiment with new or interesting ideas within that domain.
If the author reads this: kudos to you for being willing to try experimenting with your ideas.
I don't think build systems have peaked yet, so it's good that there are people willing to explore and experiment with new or interesting ideas within that domain.
If the author reads this: kudos to you for being willing to try experimenting with your ideas.
"weird to single out this person's project."
Just because it was advertised today; you can take any of these projects with funny names advertised during the last few weeks. Please or Buck or whatever.
Just because it was advertised today; you can take any of these projects with funny names advertised during the last few weeks. Please or Buck or whatever.
General purpose build systems need to make API expressiveness, observability, performance, and correctness tradeoffs in exchange for their platform breadth.
By narrowing breadth, most often by targeting only one ecosystem[1] or making correctness tradeoffs[2], new build systems can be way more expressive and performant. For now.
They’re also fun to write because DAGs are parallelism are cool problems to think about.
[1] e.g. only targeting one language allows you to use that language instead of a generalized DSL to describe your build graph [2] e.g. abandoning sandboxing altogether and using manual invalidation or just trusting the user
By narrowing breadth, most often by targeting only one ecosystem[1] or making correctness tradeoffs[2], new build systems can be way more expressive and performant. For now.
They’re also fun to write because DAGs are parallelism are cool problems to think about.
[1] e.g. only targeting one language allows you to use that language instead of a generalized DSL to describe your build graph [2] e.g. abandoning sandboxing altogether and using manual invalidation or just trusting the user
Competition yields innovation and/or standardization. While it can also fragment, ultimately competition, even friendly or "alternative" varieties, in software usually ends up benefiting the ecosystem in the long run.
Do we need another X??? You could ask the same question about operating systems, file systems, databases, cloud platforms, or programming languages. There will always be someone who is unsatisfied with the status quo and will invent something new in an already established category.
I invented a new kind of database https://www.youtube.com/watch?v=OVICKCkWMZE because I was dissatisfied with existing ones that required you to create many indexes on big tables in order to get reasonable query speeds. That doesn't mean my system will gain instant success over Postgres, MySQL, SQLite, SQL Server, etc.
I invented a new kind of database https://www.youtube.com/watch?v=OVICKCkWMZE because I was dissatisfied with existing ones that required you to create many indexes on big tables in order to get reasonable query speeds. That doesn't mean my system will gain instant success over Postgres, MySQL, SQLite, SQL Server, etc.
I 100% want people to continue spending their free time inventing new open source things. A lot of software I use on a daily basis only exists because some people decided that they could do better than whatever they were using before.
is this just a submarine ad for BUSY? lowkey seems like it. anyway i feel like a user friendly layer on bazel might be a good solution to a lot of problems, maybe along with a port to a compiled language so no JVM dependency.
Do we need a meta build system, to tie together lots of build systems, so that anyone can use the build system they like while also working with other people's build tools?
That would be GYP, which has been mostly ditched by its big adopters.
> Why do people continue to invent new build systems?
Because none of the existing ones solve everything and they want to try do some aspect better
Because none of the existing ones solve everything and they want to try do some aspect better
Probably not.
We might need new infra automation systems though. The commonly recommended "simple" one, ansible, features an intuitive 22-layer precedence hierarchy.
We might need new infra automation systems though. The commonly recommended "simple" one, ansible, features an intuitive 22-layer precedence hierarchy.
Ansible should never have become "the standard". It was always obviously flawed (YAML is an abomination for long configuration and templating; ssh doesn't scale). It's only advantage was that it was easy to start with and easy to remove. Why Red Hat decided to buy it and then everyone decided even more to standardise on it is a headscratcher.
Things I want to be able to do:
1. Use the same build system for C++, Python, Java, Rust, Golang, etc. Any language should easily plug in.
2. I don't want to have to know exactly which g++/javac/cargo/go commands are being run under the hood.
3. I want to build source code and I also want to "build" release packages, docker containers, compressed asset packs, etc. Anything that starts with files and is "compiled" into an output should be expressible.
4. It should understand how testing works. It should be able to abstract "I need to run FooBarTest since I changed FooBar" from me. I just want to know all my tests passed.
4 1/2. We should be able to run tests in other architectures. Something like KVM could be used to start VMs to run tests on arm platforms from a x86 host.
5. I want to know code coverage for my tests in aggregate across all languages.
6. It should be easy to drop into a debugger for a test or binary.
7. I want to be able to zip up my source code, unzip it in 20 years, rerun the build on new computer, and have the build complete regardless of if I have an internet connection or not.
8. I want a package manager that just dumps source code and build instructions into a folder that the build system understands. This way when I open my source tree crosslinks will "just work" and we don't need to implement a new way to obtain source code + docs for every single language in every single IDE.
8 1/2. The package manager should also be able to fetch compilers/toolchains for things. If I want to cross compile all my code for arm and my code is written correctly it should "just work" without me needing to download something special. I shouldn't need to tell someone on windows to install 5 random unix emulation layers just so they can get a windows version of gcc. The package manager should just create a toolchain directory which contains at least bootstrap compilers.
9. It should expose an API so I can write tools that interface with it efficiently. My tools shouldn't care about the underlying language of my code so I can share tools with other people. EX: How many lines of code are in foobar-service? Does <library with CVE> ever get called by foobar-service? The build system's API should be general enough for me to implement this.
10. IDEs should take advantage of the build system's metadata and make better autocompletion suggestions.
The closest thing here is the Bazel/Buck/Pants/Please family of build systems but surrounding tooling for these does not exist yet (vendoring source code, package management, VMs for tests). Until we have a build system that supplies these features and is simple to use and supported by IDEs we'll keep getting new build systems.
1. Use the same build system for C++, Python, Java, Rust, Golang, etc. Any language should easily plug in.
2. I don't want to have to know exactly which g++/javac/cargo/go commands are being run under the hood.
3. I want to build source code and I also want to "build" release packages, docker containers, compressed asset packs, etc. Anything that starts with files and is "compiled" into an output should be expressible.
4. It should understand how testing works. It should be able to abstract "I need to run FooBarTest since I changed FooBar" from me. I just want to know all my tests passed.
4 1/2. We should be able to run tests in other architectures. Something like KVM could be used to start VMs to run tests on arm platforms from a x86 host.
5. I want to know code coverage for my tests in aggregate across all languages.
6. It should be easy to drop into a debugger for a test or binary.
7. I want to be able to zip up my source code, unzip it in 20 years, rerun the build on new computer, and have the build complete regardless of if I have an internet connection or not.
8. I want a package manager that just dumps source code and build instructions into a folder that the build system understands. This way when I open my source tree crosslinks will "just work" and we don't need to implement a new way to obtain source code + docs for every single language in every single IDE.
8 1/2. The package manager should also be able to fetch compilers/toolchains for things. If I want to cross compile all my code for arm and my code is written correctly it should "just work" without me needing to download something special. I shouldn't need to tell someone on windows to install 5 random unix emulation layers just so they can get a windows version of gcc. The package manager should just create a toolchain directory which contains at least bootstrap compilers.
9. It should expose an API so I can write tools that interface with it efficiently. My tools shouldn't care about the underlying language of my code so I can share tools with other people. EX: How many lines of code are in foobar-service? Does <library with CVE> ever get called by foobar-service? The build system's API should be general enough for me to implement this.
10. IDEs should take advantage of the build system's metadata and make better autocompletion suggestions.
The closest thing here is the Bazel/Buck/Pants/Please family of build systems but surrounding tooling for these does not exist yet (vendoring source code, package management, VMs for tests). Until we have a build system that supplies these features and is simple to use and supported by IDEs we'll keep getting new build systems.
What's the workflow for "I need to pass one argument to the linker" in the world where your build tool config doesn't even say which compiler it's invoking but your build fails with a bunch of linker errors?
In Bazel parlance you would write a "rule" or setup a "toolchain" which has the linker args passing setup in it. 99% of code does not need a specific linker arg though and optimizing the happy path for letting you do anything, rather than having a simplified flow that most users care about, makes things more confusing.
If you instead had a way to say "this one specific thing I am building is special and needs `$magic_extra_arguments` added to it" and the rest of your code is "normal" things will be pretty good.
If you instead had a way to say "this one specific thing I am building is special and needs `$magic_extra_arguments` added to it" and the rest of your code is "normal" things will be pretty good.
It's fairly common for me to download some project that already uses some build automation tool I've never used before and find that it only works on the original developers' machines from a linker error message :)
Gradle fits your description pretty closely.
Gradle misses almost all of these cases. In my experience it is a very unreliable build system that relies on magic and state in the daemon it starts. You can not trust Gradle to build something from source, edit it, and build again correctly. I have seen many time where you build, edit, build, edit, build, edit, push to upstream and it fails in CI because it fails on a clean build.
If you need to "clean" the build system it has failed.
If you need to "clean" the build system it has failed.
people trying to carve up a piece of universe for themselves. don't hate the player, hate the game. and use cmake...
and now we throw BUSY (https://github.com/rochus-keller/BUSY/) onto the pile.
Why do people continue to invent new build systems?
Why should we care that BUSY is statically typed?