A quote from the article I would object to is "for large datasets and complex transformations this architecture is far from ideal. This is far from the world of open-source code on Git & CI/CD that data engineering offers - again locking you into proprietary formats, and archaic development processes."
No one is forcing you to use those tools on top of something like Snowflake (which is just a SQL interface). These days we have great open source tools (such as https://www.getdbt.com/) which let you write plain SQL that you can then deploy to multiple environments, perform automated testing and deployment, and do fun scripting. At the same time, dealing with large datasets in a spark world is full of lower level details, whereas in a SQL database it's the exact same query you would run on a smaller dataset.
The reality is that the ETL model is fading in favour of ELT (load data then transform it in the warehouse) because maintaining complex data pipelines and spark clusters make little sense when you can spin up a cloud data warehouse. In this world we don't just need less developer time, those developers don't have to be engineers that can write and maintain spark workloads/clusters, they can be analysts who are able to do transformations and have something valuable out to the business faster than the equivalent spark data pipeline can be built.
SQL has definitely become the defacto tool for a lot of data processing. This model of working is generally referred to as ELT as opposed to ETL.
For small/medium scale environments Fivetran/Stitch with Snowflake/BigQuery using getdbt.com for modelling is an insanely productive way to build an analytics stack. I consider this the default way of building a new data stack unless there's a very good reason not to.
For larger scales Facebook has Presto, Google has Dremel/Procella/others, and a lot of data processing is done using SQL as opposed to writing code.
The only downside really is that it tends to be fairly focussed on batch pipelines (which are fine for 95% of workloads). But even that is becoming less of an issue with Beam/Spark so you can use SQL for both batch and streaming.
Source: Solution Architect at an analytics consultancy.
I can highly recommend this talk by Respawn, Multiplay, and Google on how Titanfall 2 does multiplayer server management. It's geared more at the infrastructure side as opposed to actual dev but worth a watch: https://www.youtube.com/watch?v=p72GaGq-B_0
While the move seems to be towards NewSQL databases (Spanner/CockroachDB), the answer to relations in NoSQL is to model the data differently. This can generally resolve a lot of the problems inherent in the need for relations.
Lambda will create a function instance to handle concurrent requests, i.e. if 100 requests happen at the same time there will be 100 function instances. It will then however keep them "alive" for a few minutes allowing it to reuse already running instances. Additionally, a function instance won't handle multiple requests at a time.
Given this, using beam is a bit of a waste in terms of individual instance scalability. That being said however, you might be able to use shared actor pools (e.g. caches, etc.) across all your functions. I want to emphasise the might as rapidly adding and removing nodes from the beam cluster might not work well or at all.
At the end of the day, the lambda model kind of supplants the actor model as your unit of messaging and concurrency, and so trying to mix the two isn't the best idea. If you want to use the beam on AWS I'd recommend sticking to ECS/Fargate/EKS. That being said, Elixir might be a nice match due to developer ergonomics, just don't expect to be able to drag and drop actor reliant features.
In my experience the reason for a JSON object per line is because a tool can then split the entire file by newlines and have a list of objects to start parsing/processing in parallel, which avoids having to parse the entire file up front to get usable data, and lets the tool start processing things while some data is still being parsed.
One of the major reasons that would influence a language like Jai is that once you do that, even if your generated C/++ is fairly optimal, you still have to go through another slower compiler. Jai's compiler is fairly optimised for speed and I think that would be an unacceptable trade off.
On the target platform side, a video game tends to have a fairly well known set of platforms it will run on (new PC's, PS4, Xbox), so I think this is an acceptable trade off.
And just a final note, the generated C/++ would be very unreadable anyway, unless it was a direct transpiler, at which point you lose a lot of the benefits of having a new language.
The app can still use environment variables and be completely independent (12 factor), but the Dockerfile/Kubernetes config still needs to provide those things, so there is a clear distinction between ops and dev, they're just more mingled than previously.
There are generally two levels of autoscaling involved with kubernetes.
Firstly, kubernetes is able to create multiple instances of your app as load increases and scale them out over all the nodes.
Secondly, the nodes your kubernetes cluster is running on can also autoscale. With terraform you can set up an AWS autoscaling group for example to automatically increase the size of your cluster as load increases.
There doesn't seem to be any part of that package that requires ES6 to use. You can use it as an ES6 module, but the commonjs version should work fine as well.
Building distributed systems makes you aware of how unreliable things are at the large scale, e.g. the network. The parent comment implies that Google's network is so fast and reliable that it becomes tempting to ignore best practices and work as if it's a non distributed system.
In regards to Purescript/Typescript, they're both statically typed and that results in friction when trying to integrate with the existing JavaScript ecosystem/libraries. Erlang/OTP might be different, but there will probably be situations where the type system is either incompatible with a certain library, or the type system is made less strict (e.g. an any type).
While I agree that the OTP perhaps is not as easily statically typed, since it was built with Erlang in mind, I do think that static typing adds a layer of robustness to distributed systems, especially if you design it that way upfront. In my experience the problem comes when you try to apply static typing to a dynamic system.
The node level autoscaling doesn't need to be integrated with kubernetes, all it needs to do is create a new instance and register it as a node through normal channels.
Even if you don't need autoscaling, I'd suggest still using autoscaling groups and setting it to a fixed number of instances, so that instances will automatically get restarted if they go down.
I don't see why not, Haskell has green threads, channels, and everything that Go has. The only argument I can see made is that the laziness and current GC implementation aren't suited, and even then, those problems can be solved with strictness pragmas, and a different GC respectively.
Since languages like Rust/C++/Swift compile reference counting into machine code already, there's no real reason why they shouldn't be able to compile it to WebAssembly.
Erlang's processes are green threads, they're just called "processes". Neither Erlang, nor Akka run 1 thread or process per actor, they just schedule multiple actors over N native threads inside the single process (ignoring multi-node situations).
The semantics of Erlang and Elixir are very similar, as they're both dynamically typed functional languages, so the performance difference should be negligible.
No one is forcing you to use those tools on top of something like Snowflake (which is just a SQL interface). These days we have great open source tools (such as https://www.getdbt.com/) which let you write plain SQL that you can then deploy to multiple environments, perform automated testing and deployment, and do fun scripting. At the same time, dealing with large datasets in a spark world is full of lower level details, whereas in a SQL database it's the exact same query you would run on a smaller dataset.
The reality is that the ETL model is fading in favour of ELT (load data then transform it in the warehouse) because maintaining complex data pipelines and spark clusters make little sense when you can spin up a cloud data warehouse. In this world we don't just need less developer time, those developers don't have to be engineers that can write and maintain spark workloads/clusters, they can be analysts who are able to do transformations and have something valuable out to the business faster than the equivalent spark data pipeline can be built.