If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly.
One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler
Metaflow tracks all artifacts and allows you to build dashboards with them, so there’s no need to use MLFlow per se. There’s a Metaflow integration in Weights and Biases, CometML etc, if you want pretty off-the-shelf dashboards
Metaflow was started to address the needs of ML/AI projects whereas Airflow and Dagster started in data engineering.
Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - including dependency management - and local experimentation, which is out of scope for Airflow and Dagster.
Metaflow has basic support for dbt and companies use it increasingly to power data engineering as AI is eating the world, but if you just need an orchestrator for ETL pipelines, Dagster is a great choice
If you are curious to hear how companies navigate the question of Airflow vs Metaflow, see e.g this recent talk by Flexport https://youtu.be/e92eXfvaxU0
A big deal is that they get packaged automatically for remote execution. And you can attach them on the command line without touching code, which makes it easy to build pipelines with pluggable functionality - think e.g. switching an LLM provider on the fly.
if you want to see similar tricks applied in Python (with a JIT compiler for query-time optimization), take a look at this fun deck that I presented a long time ago: https://tuulos.github.io/sf-python-meetup-sep-2013
we were able to handle trillion+ datapoints with relatively modest machines - definitely a useful approach if you are ready to do some bit twiddling
That's an interesting hypothesis but the words we use to express agreement and disagreement haven't changed much.
We don't try to retrieve articles/topics from the model, which would be affected by the cutoff, just asking it to analyze the sentiment or summarize the content provided in a prompt
Metaflow integrates with AWS Batch which many folks use for serious HPC. Internode scheduling happens through the multinode scheduling supported by AWS Batch. networking via EFA etc.
Like Stefan mention in the OP, Hamilton works well with tools like Metaflow which can help with many other concerns you mentioned. How you define your data transformations for ML is an open question that Hamilton addresses neatly.
Outerbounds | Systems / Full-Stack engineer | SF / Remote | Full-time
Outerbounds commercializes Metaflow, a widely used open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/full-stack/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, Kubernetes, or low-level systems programming.
One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler