An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.
If you are feeling overwhelmed with yet another machine learning pipeline automation framework, you should check out Kedro (https://github.com/quantumblacklabs/kedro).
Kedro has the simplest, leanest, functional-programming inspired pipeline definition and also spits out AirFlow and other formats readily + comes in with an integrated visualisation framework which is stunning & effective.
This seems to lend more credit to ReasonML. Compile times are faster and it doesn't bring the associated complexity of object types while still retaining the benefits of static typing.
Tell them you're looking to join a company precisely to get mentorship, see how a successful company operates, learn to work in larger product lines, etc. That way you're pre-emptively going around the Founder damage issue.
One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.
Airflow is an incredibly powerful framework to use in production, but a little unweildy for anything else.
You can use something like Kedro (https://github.com/quantumblacklabs/kedro) to get started building pipelines with pure Python functions. Kedro has its own pipeline visualiser and also has an Airflow plugin that can automatically help you generate airflow pipelines from Kedro pipelines.
I had a similar problem until I knew I had to work on a start-up and pick a market to go help. It took me ~2 years to choose that market but I haven't looked back (crypto developer tools).
Look back, not forward.
Take all the things you have spent the past 5-10 years being interested about. For me, it was coding, programming languages, developer tools, design & minimalism, products, B2B, mathematics, economics, investing, productivity. Crypto dev tools is the intersection of those - a new field allowing me to keep doing the things I'm already provably interested in rather than making bets about the future.
Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage:
- pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing
- sequential and parallel runners are built-in (don't have to rely on Airflow)
- io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn
- flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl
- there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements