Show HN: Attune - Build and publish APT repositories in seconds(github.com)
github.com
Show HN: Attune - Build and publish APT repositories in seconds
https://github.com/attunehq/attune
10 comments
It's probably intuitive to anybody who's tried to do this the hard way, but if your target is people without that experience, you might want to dumb it down even more. So this is from your readme:
Also, from your post here (and why are the quick start steps different here?),
attune repo create -u 'http://localhost:9000/debian' -d bookworm
attune repo pkg -r 1 add -c stable/v17 ~/Downloads/teleport_17.4.4_arm64.deb
Also, from your post here (and why are the quick start steps different here?),
attune repo create --uri https://apt.releases.example.com
Totally agree here. The current target is people who are familiar with the process, and we plan to write more documentation as we start thinking more about accessibility to a broader audience. We actually ran into this issue ourselves while building the tool - the Debian documentation is all there, but not the easiest to approach. These docs are one of the rough edges (sorry!) since we wanted to get this out earlier for feedback.
To answer your direct question: on self-hosted repositories, the URI right now is just a string identifier. We are working right now on a cloud hosted service, and there the URI will help us figure out which repository to actually serve.
To answer your direct question: on self-hosted repositories, the URI right now is just a string identifier. We are working right now on a cloud hosted service, and there the URI will help us figure out which repository to actually serve.
Lots more options listed on the Debian wiki, I've added Attune to the list.
https://wiki.debian.org/DebianRepository/Setup
https://wiki.debian.org/DebianRepository/Setup
Thanks, appreciate it!
How is this different to aptly, especially if one already has object storage?
One of the tools we tried was Aptly! A couple issues we had with Aptly:
1. Aptly needs to rebuild the entire repository before it can do any changes. One of our customers builds packages for every Linux OS, and every architecture, and each release contains 10-something separately packaged tools, and they do regular releases. This means their repository of packages is _huge_. When they try to do small maintenance actions like removing a package that has a bug, they need to rebuild their entire repository to do this, which takes upwards of 30 minutes. On Attune, since we have a centralized control plane, it takes about 10 seconds.
2. Aptly doesn't have concurrency controls. This makes it annoying to run in CI. Imagine you're setting up a CI pipeline that runs on every commit and publishes a package. You need to make sure you don't have two different instances of Aptly running at the same time, because their updates will clobber each other if they publish at the same time. Can you ensure that your CI only ever runs one publish at a time? This is easy in some setups, but surprisingly hard in GitHub Actions - concurrency keys don't quite work the way you'd expect them to.
3. Aptly uses its storage as its source of truth for the current state of packages. This is a great design choice for simplicity! However, if your CI job dies halfway through during an Aptly publish, this can leave your repository in a corrupted (in this case, half-updated) state. To recover from this, you need to rebuild the repository state... which you can't, because the ground truth is contained in the storage which was just corrupted! We mitigate this issue by tracking metadata in a separate transactional database so CI crashes don't cause corruption.
These are some examples of little cuts that build up over time with all of the tools we've found so far. Lots of the open source tooling is great at low scale, but the cuts start to build up once you are pushing a meaningful amount of packages to production.
1. Aptly needs to rebuild the entire repository before it can do any changes. One of our customers builds packages for every Linux OS, and every architecture, and each release contains 10-something separately packaged tools, and they do regular releases. This means their repository of packages is _huge_. When they try to do small maintenance actions like removing a package that has a bug, they need to rebuild their entire repository to do this, which takes upwards of 30 minutes. On Attune, since we have a centralized control plane, it takes about 10 seconds.
2. Aptly doesn't have concurrency controls. This makes it annoying to run in CI. Imagine you're setting up a CI pipeline that runs on every commit and publishes a package. You need to make sure you don't have two different instances of Aptly running at the same time, because their updates will clobber each other if they publish at the same time. Can you ensure that your CI only ever runs one publish at a time? This is easy in some setups, but surprisingly hard in GitHub Actions - concurrency keys don't quite work the way you'd expect them to.
3. Aptly uses its storage as its source of truth for the current state of packages. This is a great design choice for simplicity! However, if your CI job dies halfway through during an Aptly publish, this can leave your repository in a corrupted (in this case, half-updated) state. To recover from this, you need to rebuild the repository state... which you can't, because the ground truth is contained in the storage which was just corrupted! We mitigate this issue by tracking metadata in a separate transactional database so CI crashes don't cause corruption.
These are some examples of little cuts that build up over time with all of the tools we've found so far. Lots of the open source tooling is great at low scale, but the cuts start to build up once you are pushing a meaningful amount of packages to production.
Thanks for the detailed response!
Maybe I'm wrong, but aren't apt repositories just files in a web root? There are many scripts in the Debian ecosystem to generate and maintain such web directories. Maybe the actual package building would be interesting, but I don't know about just "we'll host your .deb files and keep the index updated"
You are correct. The reason we built this tool is because all of the scripts we found had sharp edges when we tried to use them in production at scale (I talk about some of them here: https://news.ycombinator.com/item?id=43730136).
Most of these scripts were designed for a world where there was A Blessed Deployment Machine that acted as its own de facto centralized control plane. We're designed for a newer world where publishing is just another piece of your CI, so you need more features to handle concurrency control, distributed signing, incremental index rebuilds, etc.
Most of these scripts were designed for a world where there was A Blessed Deployment Machine that acted as its own de facto centralized control plane. We're designed for a newer world where publishing is just another piece of your CI, so you need more features to handle concurrency control, distributed signing, incremental index rebuilds, etc.
What is an example of something that would require "distributed signing"?
Our use case is customers who (1) want to use a managed cloud hosting service that we provide, but (2) are not willing to give us their signing keys. Our design allows them to keep all of their signing keys local to their environment.
To our knowledge, we have not found another provider who supports both of these requirements. It's not some amazing technical innovation, but it is one of those annoying paper cuts that builds up with all the others.
To our knowledge, we have not found another provider who supports both of these requirements. It's not some amazing technical innovation, but it is one of those annoying paper cuts that builds up with all the others.
> when we tried to use them in production at scale
Debian and Ubuntu have been using them in production and "at scale" for decades. What are the "sharp edges" that you're trying to solve?
Debian and Ubuntu have been using them in production and "at scale" for decades. What are the "sharp edges" that you're trying to solve?
That's true! The Debian and Ubuntu folks are also experts at this. In our experience, the sharp edges generally affect teams that don't have a lot of in-house expertise in this, and where release engineering is not a core engineering competency that they want to invest in.
Signing an unsigned apt repository is about three gpg commands though. I don't see how this kind of thing requires a whole separate product.
Yeah, solving this locally for one repository definitely isn't that hard at all. Most of the features we're building become useful when you're trying to build CI integrations for a larger team while also complying with enterprise security requirements (e.g. audit logging, HSM key protections, etc.).
[deleted]
As a FOSS project which publishes debs, it's a real hassle using current tooling. The tools are heavily biased towards the needs of distro maintainers, forcing projects into overly complex workflows which are overkill for a small numbers of packages. Features that would benefit smaller repositories with more frequent updates are limited or missing.
The other options is to DIY without tooling (essentially, write custom tooling as scripts or whatever), which is messy and full of pitfalls to the inexperienced.
The other options is to DIY without tooling (essentially, write custom tooling as scripts or whatever), which is messy and full of pitfalls to the inexperienced.
Yup, this was similar to our experience as well. If publishing is still a source of toil for you, feel free to reach out at [email protected] - we'd love to see if we can do any dev work or hosting pro bono for your open source projects.
Curious how different is this from mise and ubi? https://github.com/jdx/mise
I have a few pain points with that for installing cua (https://github.com/trycua/cua/issues/27), so if it can remove the initial friction happy to chat!
I have a few pain points with that for installing cua (https://github.com/trycua/cua/issues/27), so if it can remove the initial friction happy to chat!
I've never heard of mise used for creating and managing APT repositories. Where does that figure in?
I'm not familiar with mise and ubi, but these look like they're tools that the end user runs to install binaries. In contrast, we're building something that sets up a server so that end users can use their existing `apt-get install`.
Looks neat!
What would be really valuable for us would be future support for other distros/repos.
A more unified interface for synchronized publishing across say dep/rpm/archlinux/alpine when we have multi-distro packages to build and publish.
What would be really valuable for us would be future support for other distros/repos.
A more unified interface for synchronized publishing across say dep/rpm/archlinux/alpine when we have multi-distro packages to build and publish.
Yeah, multi-distro and multi-arch is something on our roadmap! We're especially interested in automating the more annoying OS-specific parts of releasing, like MacOS notarization.
Happy to test this out, because I'm really unhappy with aptly quirks
Sorry I missed this - let us know if there's anything we can do to help! Feel free to email me at [email protected].
very cool stuff!
Yes, it's Rust ;). Sign of quality to me.
Excited to try. I ran a reprepo for a couple of years & it wasn't the worst to run, but not fun either. My confidence in efficiently publishing packages was super low, never improved much.
It also mandated running a web server (configuring my nginx to point at the repo filesystem), not very batteries included.
https://wiki.debian.org/DebianRepository/SetupWithReprepro
Excited to try. I ran a reprepo for a couple of years & it wasn't the worst to run, but not fun either. My confidence in efficiently publishing packages was super low, never improved much.
It also mandated running a web server (configuring my nginx to point at the repo filesystem), not very batteries included.
https://wiki.debian.org/DebianRepository/SetupWithReprepro
Thanks for trying, and sorry in advance for the rough edges :)
We're also working on a hosted service! If you'd like a sneak peek, send us a message at [email protected] (or email me directly at [email protected]). I'm happy to talk about your specific needs and see if we can build something for them.
(And yes, it is Rust. I keep trying to find projects where I get to stretch my Haskell wings again, but unfortunately I keep working on things with silly requirements such as "easily understandable performance" and "a cross-compilation story" and "not making my collaborators sit through another monad tutorial".)
We're also working on a hosted service! If you'd like a sneak peek, send us a message at [email protected] (or email me directly at [email protected]). I'm happy to talk about your specific needs and see if we can build something for them.
(And yes, it is Rust. I keep trying to find projects where I get to stretch my Haskell wings again, but unfortunately I keep working on things with silly requirements such as "easily understandable performance" and "a cross-compilation story" and "not making my collaborators sit through another monad tutorial".)
Will you be rewriting the Golang part in Rust?
We don't plan to do that right now. The CLI isn't that complicated, and the choice for that was driven more by Go's excellent cross-compilation story than anything else.
It would be a fun thing to do if we had the resources to get equally good cross-compilation in Rust, but we're focused on building functionality right now.
It would be a fun thing to do if we had the resources to get equally good cross-compilation in Rust, but we're focused on building functionality right now.
It looks like Golang to me?
Edit: the CLI part is Golang and the rest(?) is Rust.
Edit: the CLI part is Golang and the rest(?) is Rust.
.. for the repo github pages can be used as well, although the disgust when pushing huge .deb files up there is real.
[deleted]
Previously, we worked at other startups building open source developer tools that ran on our customers’ CI and development machines. For many of them, being able to `apt-get install` our tools was a requirement.
When we went to actually set up APT repositories, we were really surprised by the state of tooling around package publishing. The open source tools we found were old, slow, and difficult to figure out how to run in CI. The commercial tools we found were not much better. The cloud-hosted vendors required us to provide our signing keys to a cloud vendor (which was a non-starter), while the self-hosted vendors required us to operate our own specialized hosting servers.
We just wanted something simple: sign locally, run quickly, be easy to use, and deploy to managed object storage.
We couldn’t find it, so we built it. If you want to try it out, you can create a repository with three commands:
You can get the tool at https://github.com/attunehq/attune. There are a lot of rough edges right now since it's so new - sorry in advance, we're working on sanding those down.
It’s fully open source under Apache 2. We’re also working with some early customers to build enterprise features like audit logging, RBAC, and HSM integrations, and we’re thinking about building a managed cloud hosting service as well.
We’d love your feedback on whether this is useful for you, and what you’d like to see next. We’re well aware that publishing is a small piece of CI/CD, but we think a lot of the tooling in this area (publishing, artifact registries, package repositories) could really use some love.
What do you think? Comment here, or email us at [email protected].