I'm working on adding features to the snakemake aws batch executor plugin. The existing plugin supports execution on AWS Batch by dynamically creating job definitions based on rule resource configuration, but was missing support for features like using different containers for different rules, consumable resources, secrets, etc. Two approaches:
1) https://github.com/radusuciu/snakemake-executor-plugin-aws-b... (my fork). Just add the features to the batch job building code
2) https://github.com/radusuciu/snakemake-executor-plugin-aws-b.... This is more experimental and not yet fully working. I wanted to try a few things. a) can we rely on existing job definitions (managed through IaC instead). b) can we implement a fire-and-forget model where the main snakemake process runs on Batch as well? c) Can we slim down the snakemake container by stripping off unnecessary features.
Example: you want to set your local docker context to the production environment, so that when you type `docker system prune --volumes` you delete your production data.
docker swarm is also a decent solution if you do need to distribute some workloads, while still using a docker compose file with a few extra tweaks. I use this to distribute compute intensive jobs across a few servers and it pretty much just works at this scale. The sharp edges I've come across are related to differences between the compose file versions supported by compose and swarm. Swarm continues to use Compose file version 3 which was used by Compose V1 [1].
You've explained your value proposition vs. celery, but I'm curious if you also see Hatchet as an alternative to Nextflow/Snakemake which are commonly used in bioinformatics.
Example: you use a large plotting library, but it's only needed on some pages. Chunking allows you to selectively load that code only where it's needed.
My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.
The term translational is frequently used to describe basic research (eg. with cells or mice) that has applications to human health - findings in lab that make it to the clinic.