DevSpace is a great tool but it’s bummer you didn’t like Garden.
Admittedly, documentation and stability weren’t quite what we’d like and we’ve done a massive overhaul of the foundational pieces in the past 12 months.
If you want to share feedback I’m all ears, my email is in my profile.
The way we do things is that we build everything in the cloud and store in a central container registry. So if I trigger a build during dev, the CI runner can re-use that, e.g. if it’s needed before running a test or creating a preview env.
Similarly if another dev (or a CI runner) triggers a build of one of our services, I won’t have to next time I start my dev environment. And because it’s built in the cloud there’s no “works on my machine”.
Same applies to tests actually. They run in the cloud in an independent and trusted environment and the results are cached and stored centrally.
Garden knows all the files and config that belong to a given test suite. So the very first CI run may run tests for service A, service B, and service C. I then write code that only changes service B, open a PR and only the relevant tests get run in CI.
And because it’s all in prod-like environments, I can run integ and e2e tests from my laptop as I code, instead of only having that set up for CI.
Your pipelines should mostly be CI provider agnostic and runnable from anywhere, including your laptop during development.
If you ever need to change a CI provider, you just move your pipelines.
I'm definitely biased, I'm working at Garden[0] which allows you to do just that but it's mostly applicable for teams using Kubernetes. I paused at your comment because Garden has sometimes been called "a makefile for the cloud".
Whatever tooling you use, having portable pipelines that you can iterate on and debug from your laptop is the only sane approach imo.
We did indeed have a reputation for being a bit more complicated then some of the other tools out there (and in turn more flexible). But we've put enormous effort into simplifying Garden itself and also improving documentation.
If you already have your stack containerised you should be good to go but experience with Kubernetes is definitely a plus. Most of our users use K8s in production and use Garden for developing and testing in production-like environments in "inner loop" dev and CI.
To add to what's already been said: If you think about it, CI pipelines are typically a complete description of how your system is built, tested, and deployed.
Which is pretty fantastic except for how walled off they are. You can't really re-use these descriptions for e.g. development, they're not vendor agnostic, and they only way to run them is by pushing your code.
Maybe it's a silly analogy but it's almost like being a web dev that doesn't have a browser and needs to send their code to a friend who can tell them if that font size looks good.
I think we're way over due for freeing these "blueprints" of our system from the confines of CI and making them portable and flexible. And containers are the technology that's enabling that.
Full disclaimer (as always): I work at Garden[0] where we're also solving that problem but taking a slightly different approach to Dagger (it's still a DAG). Garden config is declarative and the jobs (we call them actions) have a semantic meaning. You can e.g. have a Build action of type container or a Deploy action of type Helm and Garden will figure out what to do with it.
We've also put a lot of emphasis on the inner loop development story with hot reloading functionality, log streaming and more.
Just to add to what Tao was saying, our pipelines are not only portable but also "smart".
Instead of having to specify every step of a job—either in code or config—you instead tell Garden that it's e.g. a build of type container, or a deploy of type Helm, and our plugin system figures out the rest (of course with escape hatches when needed).
We also track the files that go "into" each job and cache the results, so the same job never has to run twice if the code doesn't change. So if you have a large distributed system and a change only affects a small part of it, you don't need to re-run everything. It can save _a lot_ of time.
Unrelated, but love that the Cicada team created a Treesitter grammar for Neovim!
Me too. In fact Garden (dev tooling for the Kubernetes)[0] is a Berlin start-up with three Icelandic founders.
And if I'm not mistaken, two of us worked briefly with @halldorel (commented below) at an earlier Icelandic start-up. It's a small world (if you're Icelandic).
Edit: Scratch that, three Gardeners worked with @halldorel.
We use Mutagen for Garden's hot reloading mechanism. (Garden is a dev tool for K8s and hot reloading enables users to sync changes directly to a prod like dev environment as opposed to doing a rebuild and re-deploy).
It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms.
It's zero effort which is important for a small team like ours. Allows us to focus on the content as opposed to bikeshedding design.
Overall I'm happy with the look and feel of things and the support is typically good.
That being said, they recently shipped changes that essentially made the docs site impossibly slow for a few days. They've been working on fixing that and it's better, but not as snappy as before. I also preferred the previous look (it's very similar but the new one is a bit more clunky imo).
We do have a lot of long code examples (YAML reference docs) which I think may contribute to the "sluggishness".
But overall I'd recommend if you want to minimise effort and maintenance. In any case it's easy to give it a spin and see if it works for you.
Our mission is to keep developers productive and happy in the cloud-native era.
Docker, Kubernetes and other cloud-native technologies have made us better than ever at operating our systems in production, but the day-to-day development experience has been left lagging behind.
Developers spend less time in flow, and more time waiting for builds, debugging scripts, or otherwise fighting their tools. We're here to fix that.
Our platform—which includes our open source core product[2]—allows developers to work on distributed systems with remote, production-like dev environments, while enjoying the same fast and frictionless feedback loops we've come to expect when developing a single service locally.
It's a platform that democratizes the kind of advanced developer tooling that only the largest software companies have the resources to build and maintain.
We're still a small team and you'd be joining at a time where you can have a huge impact on our product and our culture. If you've ever thought to yourself that something's not right with our modern development workflows, now's your chance to help fix it!
Garden[1] is a tool we built (yes, I'm affiliated :)) that has a lot of this functionality built-in. Might fit your use case.
We recently re-wrote the hot reload functionality to use Mutagen[2] under the hood and it's insanely fast (<200ms anecdotally). It also does two way sync which can be useful. The old implementation used rsync but a lot of our Windows users struggled with that. So I figured I'd share in case that sounds familiar.
What happens after a sync event depends on the stack, but we've had pretty good success with Entr[3]. We often have it watch a single file so that multiple watchers in a shared dev cluster don't eat up all the node's resources.
In that case you might also be interested in Garden (https://github.com/garden-io/garden). It's an open source tool made for developing distributed systems / micro services. It abstracts most of the gnarly parts of K8s away and allows you to opt into them as needed. Full disclosure, I am affiliated so apologies for the shameless plug. But figured it might of interest to you.
But to your point, abstractions are hard, and we've tried to design Garden such that it takes care of the _what_ and _when_ but allows the users to decide the _how_.
> I'd also like to see something that takes it a step further--build the docker image AND a set of kubernetes manifests that reference that docker image
You might want to check out Garden for that (https://docs.garden.io/basics/how-garden-works). It thinks about your stack as a graph and manages build and deploy dependencies. And it supports the use case you're describing, i.e., you can combine a Docker image and a set of K8s manifests into a single "module" and the manifests can reference the Docker image.
Depending on where you are in the transition, Garden (https://docs.garden.io/basics/how-garden-works) might be a good fit. It definitely touches on all of the surrounding problems such as managing dependencies, integration tests, and running arbitrary tasks (think DB migrations)
You define one part of your system at a time and Garden compiles that into a graph of modules that it knows how to build, test, and deploy. This means you can adopt it piecemeal and use it to run an entire CI pipeline from any given context, e.g. your local machine, VM or actual CI for that matter.
Full disclosure, I'm affiliated with project but your comment describes a problem we see from a lot our users so I couldn't help but replying.
Hey, Garden co-founder here! Flux is a GitOps tool that focusses on the deployment of your services through the use of operators (usually to deploy to staging/production environments). Garden instead focuses on the cloud native testing process and spinning up production-like preview environments. We actually have a number of users that are using Garden to automate their development / PR review and Flux to deploy to staging/production. I think they complement each other quite well.
I have no experience with the platform myself, just bumped into it by coincidence when my OSS project (written it Typescript) was featured