the topic of managing large dependency chains for ML/AI workloads in a reproducible has been a deep rabbit hole for us. if you are curious, here is some of the work in open domain
Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.
Outerbounds | Systems dev | SF / Remote | Full-time
Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.
We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.
I think another very important piece of the puzzle is *when* to make the transition from a monolith service to a microservice stack. I have seen many startups getting distracted by committing to a microservices transition rather too early in their lifecycle, and on the flip side - big organisations postponing and accruing a huge transition tax.
Metaflow was built to assist in both developing ML models and deploying/managing them in production. AFAIK, TFX is focused on the deployment story of ML pipelines.
Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessary IAM permissions to ensure data integrity. Using Metaflow, you can then write your workflows in Python/R and Metaflow will take care of managing your ML dev/prod lifecycle.
Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well.
I wouldn’t qualify metaflow as anti-UI. For model monitoring, we haven’t found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like.
As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.
https://docs.metaflow.org/scaling/dependencies https://outerbounds.com/blog/containerize-with-fast-bakery