While the way secrets work in Swarm seems weird when compared to Kubernetes, this is usually pretty easily solved by a quick overriding entrypoint in the docker stack file that does essentially this:
Speaking as a Co-Founder of NeuroForge here, with some exciting news: we're proud to announce a new alpha release for our project, Swarmgate - version 0.7.0. This tool, which started as a simple experiment within our team, has evolved into an essential project aimed at addressing the complex challenges of Docker Swarm management. Whether it's juggling multiple users in a single swarm or optimizing resources across various swarms, Swarmgate 0.7.0 is our latest answer to these issues, offering enhanced features and improvements. You can find it over at:
What's Swarmgate?
Swarmgate is our innovative Docker Socket Proxy designed to provide a tenant-specific view onto a Docker Swarm. It supports all necessary operations for deploying stacks, along with managing volumes, secrets, configs, networks, and more. This allows multiple teams to work collaboratively within the same cluster without interfering with each other, thanks to unique labels that filter requests based on resource ownership. It's about making Docker Swarm environments as efficient and user-friendly as possible.
Under the Hood:
We've built Swarmgate using Node.js and Express, with the dockerode (and docker-modem) library for Docker interaction. This technology stack ensures robust performance and seamless integration with Docker's API, providing a reliable and effective management tool.
What's New in 0.7.0?
Docker Registry Auth Verification: Enhanced security through Docker Registry authentication checks to prevent unauthorized access to images.
Security Enhancements: Removal of the :version/swarm endpoint to protect against potential security vulnerabilities by exposing swarm join tokens.
Simplified Proxying: Introduction of the proxyRequestToDocker function for straightforward proxying of requests without the need for filtering.
Resolved Log Issues: We've tackled service/tasks log parsing issues to ensure compatibility and ease of use with the Docker CLI.
A Gentle Reminder:
As enthusiastic as we are about Swarmgate 0.7.0, it's important to remember that this is still ALPHA software. It's primarily a defense against accidental disruptions within clusters. While we're diligently enhancing security features, it's advisable to use Swarmgate in environments where there's a high level of trust among users.
What's up next:
A better tutorial on how to set this up
More ideas from https://github.com/neuroforgede/swarmgate/issues/1
I love DRF for CRUD apis. It just gets the job done and you can Focus on data modelling.
We built our data hub/data Integration solution on top of it. [1] It was a good choice.
By far the most extensible and overridable library I have worked with so far. Even when you need to ressort to hacks, they never seem to break when upgrading a Version.
If you want to stick on one machine, you can always just use a single node Docker Swarm to get the fully automated zero downtime deploys you want with Docker Compose:
I have implemented this for our tool NF Compose that allows us to build REST APIs without writing a single line of code [0]. I didn't go the route of triggers because we generate database tables automatically and we used to have a crazy versioning scheme that was inspired by data vault and anchor modelling where we stored every change on every attribute as a new record. This allowed for simple point in time queries.
Sounded cool, but in practice it was really slow. The techniques that are usually employed by Data Vault to fix this issue seemed too complex. Over time we moved to an implementation that handles the historization dynamically at runtime by generating historizing SQL queries ourselves [1]. We now use transaction time to determine winners and use an autoincrementing column to determine who wins on ties. A lot of brainpower went into ensuring this design is concurrency safe. On a sidenote: Generating SQL in python sounds dangerous, but we spent a lot of time on making it secure. We even have a linter that checks that everything is escaped properly whenever we are in dev mode [2].
Curious, have you tried speeding things up with e.g. cube.js? We used it in a fully custom project and it was a Performance life saver. It works quite well with Superset actually.
I saw your comment and am wondering what in particular you are struggling with.
I recently fixed one of my biggest pet peeves with docker swarm - the inability to directly exec into a service without first SSHing to the host the task is running on.
Maybe your issue is in this ballpark? Happy to exchange notes on this. If you are looking for a community of Swarm users, check out https://devops.fan (that's a discord hosted by Bret Fisher)
CSI support for Swarm is in beta as well and already merged in the Hetzner CSI driver (https://github.com/hetznercloud/csi-driver/tree/main/deploy/...). There are some rough edges atm with Docker + CSI so I would stick with docker-volume-hetzner for now for prod usage.
We use Docker Swarm for our deployments, so I will answer the questions based on that.
We have built some tooling around setting up and maintaining the swarm using ansible [0]. We also added some Hetzner flavour to that [1] which allows us to automatically spin up completely new clusters in a really short amount of time.
deploy from source repo:
- We use Azure DevOps pipelines that automate deployments based on environment configs living in an encrypted state in Git repos. We use [2] and [3] to make it easier to organize the deployments using `docker stack deploy` under the hood.
keep software up to date:
- We are currently looking into CVE scanners that export into prometheus to give us an idea of what we should update
load balancing:
- depending on the project, Hetzner LB or Cloudflare
handle scaling:
- manually, but i would love to build some autoscaler for swarm that interacts with our tooling [0] and [1]
automate backups:
- docker swarm cronjobs either via jobs with restart condition and a delay or [4]
maintain security:
- Hetzner LB is front facing. Communication is done via encrypted networks inside Hetzner private cloud networks
CSI support for Swarm is in beta as well and already merged in the Hetzner CSI driver (https://github.com/hetznercloud/csi-driver/tree/main/deploy/...). There are some rough edges atm with Docker + CSI so I would stick with docker-volume-hetzner for now for prod usage.