My experience is quite similar, except my issue was recurring severe unexplained pain with scarring of the joints. I've come to believe the same conclusion that chronic fight or flight is the biggest factor and have healed myself within a year.
The difference is that I've come at it from a different approach, where this fight or flight is a neural pathway or habit of the unconscious mind that you can train yourself out of. Although trauma can probably be a significant factor on how people get to this state, "the way out" (title of the book by Alan Gordon that explained these ideas) is probable the same and I believe not all people with these issues have significant trauma, although in some it may be necessary to address as an underlying cause.
As for how you can train yourself, it's explained in the book mentioned above as well as the free podcast "Tell me about your pain". Since the podcast is free so the techniques are public knowledge, I believe I'm allowed to summarize my take:
1. Constant "problem solving" and overthinking about your health can put you in a state of stress that heightens your perception of your own body which in turn convinces you that something is seriously wrong (eg: you may hear your own heart beating very loudly even though it's beating normally, or you may feel pain even though the nerves are transmitting normal signals to your brain). If you've been to many doctors and they can't find an explanation, there's a chance they all missed a serious problem, but there's probably a bigger chance that there's nothing physically wrong with your body but your nervous system is causing the altered perception and over reaction. It can be freeing to realize you don't need to try a new diet, or spend all your free time on pubmed, or go see an expensive doctor that prescribes expensive supplements. If you have a "neuroplastic" issue these things will not only not help, but actually hinder your recovery.
2. Catching the "what if" thoughts. You can't stop yourself from going to a worst case scenario, but you can stop yourself from buying into it. Every time you think something like "what if I actually have an injury that the MRI and X-Ray missed"? You can reply to yourself how unlikely that is. The nervous system is part of the subconscious so making yourself feel calm will probably, over time, calm it down.
3. Somatic tracking, which is the action of paying attention to a painful, strange or otherwise uncomfortable sensation in your body through a lens of safety. Eg: wow, it's incredible that my brain can usually filter the sound of my heartbeat but now I'm able to hear it at full volume. If you understand that the sensation is safe and you have nothing to fear, paying attention to that sensation will slowly make it go away over time (not necessarily in one session, which although it can happen is not the goal). Your nervous system highlights pain or sensations because it believes they're dangerous and wants you to feel them to avoid further injury. When you pay attention to them in a lens of safety it understands that there's no danger so they get deactivated.
4.Paying attention to things that feel good and being mindful in the moment about them. Whether it's feeling your breath in mindfulness meditation or taking a walk in the park and consciously being present and enjoying the experience.
5. Filtering what you consume. Media and content nowadays tries to appeal to our most basic instincts, especially fear. Reading news can put you in a state of alert. Watching shorts or tiktok can overstimulate you. Try to be comfortable with stillness and doing nothing. You don't need to fill every free moment of time with your phone.
6. Have you ever though: if I was better I would do X, or when I get better I'll do Y? Being sick can bring apathy that can stop you from doing things you enjoy even after being physically capable. Actually doing them and learning to enjoy them again isn't just a possibility, it's a necessary part of recovery.
I'm sure there's more, but this is what worked for me to go from bed-bound with full body pain (with 3 surgeries to remove internal joint scarring without any injury) to being 98% recovered in less than one year. If any of this resonates with someone reading this I'd recommend listening to the podcast, it's free and got me 70% of the way there even without the book.
Monocultures of any kind are always fragile, you need diversity to have a resilient ecosystem. These projects should benefit from an understanding of permaculture, which is a discipline that aims to create the right conditions for healthy systems. Everything from succession (pioneer leguminous species that can fix nitrogen and improve soil, slowly replaced by other species), trying to slow down and catch water where it falls to prevent soil erosion and runoff and much more. I've heard (unsubstantiated) claims that initiatives in China have already started to take these into account and have succeeded where other monoculture forests failed.
A side effect is that you can end up with productive species. Imagine forests where many trees bear fruits, others have acorns that pigs can feed on, fruit vines and understory herbs that animals can graze on, large lakes with edible fish. This is the future I'd be excited for and it's all currently possible with the right policies.
I have my own opinion on Airflow's pain points and created Typhoon Orchestrator (https://github.com/typhoon-data-org/typhoon-orchestrator) to solve them. It doesn't have many stars yet but I've used it to create some pipelines for medium sized companies in a few days, and they've been running for over a year without issues.
In particular I transpile to Airflow code (can also deploy to Lambda) because I think it's still the most robust and well supported "runtime", I just don't think the developer experience is that good.
Pattern matching is supported in 3.10 and can match classes structurally, as well as other types (edit: I see that you mean you don't like that it's a statement, which I agree with). The typing system supports structural types with Protocol. Personally what I miss most are multi-line lambdas.
A lot of people hate on SQL. I used to be one of them but I've come to think that for data transformations it's hard to beat. My current favorite is DuckDB, which is like a SQLite but columnar. It has great performance and it's easy to call it from python and even run SQL on pandas dataframes.
Sounds like he's still in the apple ecosystem. What i wanted to do was use it for scripting and server stuff (swift for server and Linux) and I think AppCode was not compatible with that at the time.
A lot of the time they do ETL/ELT mostly with airflow nowadays.
Some use little more than SQL and if they're lucky something like DBT to template and schedule the queries.
A lot of SQL now runs on data warehouses of which snowflake and Google bigquery stand out, but it's SQL nevertheless.
A few lucky ones get to use Spark which is more interesting as it's written in scala or python, but from my experience 99% of the time businesses are better off training business analysts to use SQL and writing the queries in snowflake.
Another interesting area is real time data, for existing e with Kafka. This can be challenging and fulfilling except a lot of the time businesses people will tell you it's cool but they don't want the data to keep changing so could you please aggregate it every day... So you end up with an overly complicated batch system anyways.
For me the sweet spot of practicality right now is Airflow/DBT/python/snowflake.
Probably pretty easily if you're a good software engineer. I think they don't require as much technical knowledge, though I can only speak on data engineering from first hand experience. A lot of data engineers aren't very good programmers (some use little more than SQL) though there's always outliers.
I love swift but don't do any OS X development and don't want to use XCode. Not sure how the LSP plugin is doing these days but last time I tried it it was pretty unstable (also why can't they have it in the store so you won't need to install manually?). I wish swift for tensorflow had really taken off and put it in a position to compete with python as it's a very enjoyable language.
I think a lot of people are in the same boat and wish there was a nice ethical way to promote projects that worked without pushing them hard on people. I hope there is.
It's commonplace in data engineering circles, so if that's where the author comes from it's understandable that it wouldn't occur to him that it's not the case in every discipline of programming.
I think your theory is flawed because of the Bohr effect. CO2 in your blood increases the affinity of oxygen for hemoglobin. Higher oxygen and less CO2 means that most oxygen can't combine with hemoglobin and just ends up oxidizing somewhere which is definitely not ideal. You can easily test this by hyperventilating, which despite taking in much more oxygen and breathing out much more CO2 will cause less oxygen to reach your brain until you pass out.
The only way to breathe out more CO2 and lose weight would be to make more CO2 by increasing your metabolism, for example by taking thyroid hormone. This brings us full circle to your theory as it shows that increased CO2 breathing isn't the cause of weight loss, but a consequence of a faster metabolism which will in turn burn more calories and create weight loss.
I think there's space for an open source library that can help with what you described by using just python and YAML. We originally created https://github.com/typhoon-data-org/typhoon-orchestrator to orchestrate ETL workflows, which would be a superset of the use cases you described. Our next goal is to allow deployment to AWS lambda which can be a good compromise between getting locked in with SAAS and hosting your own infrastructure.
Also check out Zappa's scheduled tasks that have a similar goal and inspired our library. We used it initially as a backbone and ran into a series of issues that forced us to write our own version of it, but depending on your goals it could be enough.
Ordinarily I would not recommend airflow over a script for a one off like this and you are right that it could be done with much less ceremony. I do however think it was a good choice in this case due to the sheer amount of data downloaded and how long it took because airflow gives you the following advantages:
- Parallelization: this may be the most minor advantage in the list, but you don't need to set up thread pools to parallelize your work since the scheduler parallelizes tasks.
- Observability: you easily see which tasks failed and look at the logs.
-Reliability and task isolation: you can set up automatic retries for jobs that fail, and they won't affect the rest of the flow. You can easily relaunch the tasks that fail independently without restarting the whole load.
The difference is that I've come at it from a different approach, where this fight or flight is a neural pathway or habit of the unconscious mind that you can train yourself out of. Although trauma can probably be a significant factor on how people get to this state, "the way out" (title of the book by Alan Gordon that explained these ideas) is probable the same and I believe not all people with these issues have significant trauma, although in some it may be necessary to address as an underlying cause.
As for how you can train yourself, it's explained in the book mentioned above as well as the free podcast "Tell me about your pain". Since the podcast is free so the techniques are public knowledge, I believe I'm allowed to summarize my take:
1. Constant "problem solving" and overthinking about your health can put you in a state of stress that heightens your perception of your own body which in turn convinces you that something is seriously wrong (eg: you may hear your own heart beating very loudly even though it's beating normally, or you may feel pain even though the nerves are transmitting normal signals to your brain). If you've been to many doctors and they can't find an explanation, there's a chance they all missed a serious problem, but there's probably a bigger chance that there's nothing physically wrong with your body but your nervous system is causing the altered perception and over reaction. It can be freeing to realize you don't need to try a new diet, or spend all your free time on pubmed, or go see an expensive doctor that prescribes expensive supplements. If you have a "neuroplastic" issue these things will not only not help, but actually hinder your recovery.
2. Catching the "what if" thoughts. You can't stop yourself from going to a worst case scenario, but you can stop yourself from buying into it. Every time you think something like "what if I actually have an injury that the MRI and X-Ray missed"? You can reply to yourself how unlikely that is. The nervous system is part of the subconscious so making yourself feel calm will probably, over time, calm it down.
3. Somatic tracking, which is the action of paying attention to a painful, strange or otherwise uncomfortable sensation in your body through a lens of safety. Eg: wow, it's incredible that my brain can usually filter the sound of my heartbeat but now I'm able to hear it at full volume. If you understand that the sensation is safe and you have nothing to fear, paying attention to that sensation will slowly make it go away over time (not necessarily in one session, which although it can happen is not the goal). Your nervous system highlights pain or sensations because it believes they're dangerous and wants you to feel them to avoid further injury. When you pay attention to them in a lens of safety it understands that there's no danger so they get deactivated.
4.Paying attention to things that feel good and being mindful in the moment about them. Whether it's feeling your breath in mindfulness meditation or taking a walk in the park and consciously being present and enjoying the experience.
5. Filtering what you consume. Media and content nowadays tries to appeal to our most basic instincts, especially fear. Reading news can put you in a state of alert. Watching shorts or tiktok can overstimulate you. Try to be comfortable with stillness and doing nothing. You don't need to fill every free moment of time with your phone.
6. Have you ever though: if I was better I would do X, or when I get better I'll do Y? Being sick can bring apathy that can stop you from doing things you enjoy even after being physically capable. Actually doing them and learning to enjoy them again isn't just a possibility, it's a necessary part of recovery.
I'm sure there's more, but this is what worked for me to go from bed-bound with full body pain (with 3 surgeries to remove internal joint scarring without any injury) to being 98% recovered in less than one year. If any of this resonates with someone reading this I'd recommend listening to the podcast, it's free and got me 70% of the way there even without the book.