Let's say your immutable DAG has four different 'transformations' (think either a map or a fold on the data). Beam or Spark will take a chunk of your data, partition it somehow (so you get parallelism), and do a transform on the data.
Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot. So each step is usually saved to disk via checkpoint and then moved along to the next transformation in batches, so if the subsequent transfomation is where it dies, you have a safe spot to restart that particular computation from.
In addition, your (chunk of) data might need to be shipped to an additional machine to be transformed so you're not so much 'updating' data as you are creating new data in a space efficient way of the changes and shuffling those along the various steps.
It's awkward to work with fully nested structures. Think about having a map of customer objects which have a list of addresses and you need to update everyone's first address to be capitalized for some raeson. You'd really want a fully fleshed out lens library and maybe even macros for auto generating them on your concrete data structures to make it easy to 'update' (via a persistent structure) without having to deal with all the cruft of doing it by hand.
If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next edge and so on.
In a single standalone system, many argue that a purely immutable conceptual approach has pros and cons. In the Big Data world, I've never heard anyone argue for an imperative approach. Immutability is necessary, and as soon as you need immutability you want all the other goodies that facilitate it in a clean way like monads, lenses, and the like.
I don't have time (nor the endurance) for a 90 minute run, so could you combine the two? I often do a 14 hour fast and then go for a 20 minute run combined with some calisthenics on my lunchbreak before eating. How can you tell your body is in a ketogenic state?
I can definitely say the first few times I exerted a ton of energy after not eating for 12+ hours I felt like I had the flu for a few hours, but now I don't really get that.
Would love to get your take on if python is popular in large organizations because of the existing libraries or if it is the language asethetic itself?
To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated.
Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas.
Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python.
If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging.
This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.
When I would commute from Orange County so Santa Clara the flight was often ~40 minutes. Combine with takeoff and landing it didn't make a lot of sense to pull out an addition device but it was easy enough to watch a short TV show on my phone.
Depends on the language (I don't use it for Java), but for Scala and python I definitely prefer it. I'd definitely recommend using TMux with Vim, I've mapped my tmux navigation to mirror Vim's window navigation, and I can keep a terminal and REPL open and easily copy-paste between Vim and my tmux windows.
>Sure, that's possible, but the opposite is also possible: it might be wrong for most programs we actually write.
We could confirm this though! It's not like we can't find out if a given program halts or is inconsisent. Godel talks about it in his letter to Von Nuemann.
>Only one thread actually mutates anything, and its the same one over and over
Sounds great in theory and usually results in fantastic average case, but you get a hot partition and suddenly you can't work share and things go south. It's not a perfect solution.
I think it's important to point out that you can do calculus and arithmetic and only construct proofs that are provable. The possibility of deriving unprovable but true statements in an axiomatic system doesn't mean they're common or you can't prove all the satatements one has encountered thus far.
> The day rates look high, but they have to pay self-employment taxes and health insurance
Most SWEs doing 'contract work' aren't actually 1099 contractors, what it means is they work for an intermediary and then 'contracted out' to a big company with a recognizable name (Google/BofA/Verizon/IBM). These jobs are W2 so you don't pay self-employment tax, they usually have mediocre insurance options, and you lose your job as soon as the work dries up.
It's simply a way for a company to staff up quickly without going through as strict of hiring committee and having any obligation to keep said staff around if plans change.
It's often easier for a department to get a req for a senior/staff contractor (and make salary exceptions) than a FTE, so what happens is departments will offer 200, 250 an hour for temp work but only able to hire engineers for 150k a year because their 'employee' req is slotted lower.
Both at my current startup job and at my previous Verizon job, a lot of my time is scaling out existing services and also boostrapping core infrastructure projects that prioritize reliability over adding features or addressing rapidly changing requirements.
Part of that is because I'v become very comfortable diving into very complex, established code bases because of my OSS experience, and enabled by the fact that I've had coworkers who didn't enjoy the deep dive/scaling part of coding compared to adding new features.
A lot of developers really enjoy the spotlight of having added the new killer feature that everyone recognizes, and sometimes that's even different skill set.
So my long winded answer is that there are two main ways to accomplish your goal:
1. Be able to refactor, address technical debt, scale out solutions, debug any problems, and find a team that complements your skillset, and get good at making sure your non visible contributions are known and appreciated. This often means having clear and easily available metrics that can show over time things like: less errors, more tests, faster build times, and so on. Really anything that you can point to that makes it easy to understand your value to those outside your team/org.
2. Become a team lead. A lot of times the tech debt/refactoring work isn't celebrated or recognized, but necessary. The job of a team lead is to both give guidance and enable your engineers to accomplish their tasks. Making code more maintanable, easy to read, and robust ensures your team will quickly and happily progress on their task, and you'll get to share in their successes even if you aren't the one contributing features. Unlike scenario 1, you don't necessarily even have to show as many metrics, as your value proposition is much simpler: Make your team productive and happy.
I think he means that the Fed can directly and indirectly increase the money supply, which disproportionately benefits people and corporations leveraged into assets. Hence, increasing the money supply makes the rich even richer.
I'm postulating that any modern CRUD app framework has Monads, Monoids, and other categorically inspired structure, even if they don't call it that or have a way to abstract over it.
>I was on a team building good old crud apps using monads, monoids, categories
Here's the thing, if you look at a basic Spring crud app, you are also using Monads, Monoids, Categories, Traverse, etc. but you aren't expressing it in the type system. Seriously go look at modern Spring's flux stuff, it's all there minus the type classes.
I've seen teams that tried to over engineer Spring, teams that tried to over engineer Node applications, and yes there are teams that over engineer Functional style Scala.
All the teams I worked with (and managed) at Verizon leaned pretty heavily into FP style Scala without much over engineering and the experience was extremely pleasant. The only production issues in my three years there I remember were performance related, finding out how to get more throughput or lower latency out of FP Scala. I literally can't remember any 'bugs' that made it to production.
Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot. So each step is usually saved to disk via checkpoint and then moved along to the next transformation in batches, so if the subsequent transfomation is where it dies, you have a safe spot to restart that particular computation from.
In addition, your (chunk of) data might need to be shipped to an additional machine to be transformed so you're not so much 'updating' data as you are creating new data in a space efficient way of the changes and shuffling those along the various steps.