TLP [1] covers this use case pretty well, although I confess I didn't expect this to still be an issue these days (until I bought my latest laptop and had to run TLP on Kubuntu 20.04 to ensure battery life on par with the factory specs, and TLP almost tripled it)
BOINC tasks are validated by running multiple executions on different hosts/users and comparing results; credit, and hence GRC, is only attributed to tasks with valid results: https://boinc.berkeley.edu/trac/wiki/CreditNew
All the tools you mention are great for their specific use cases, e.g. Snowflake BigQuery and RedShift are great for analytics over big data, the very common COUNT/SUM/AVG/PARTITION/GROUP BY analysis. But as with any other tool, they're not a good/perfect fit for diverse analysis methods of diverse data types, which I think is what a data lake aims at. Analyzing a large JSON dataset on Snowflake is possible, but either too slow or expensive when compared with more appropriate tools (e.g. Elasticsearch or a Python notebook running PySpark).
Having a data lake - which I understand as a repository of raw data of diverse types, regardless of the tools - structured in a tool like S3 is very useful when you have multiple use cases over data of different kinds.
For example, you could store audio files from customer calls and have them processed automatically by Spark jobs (e.g. for transcript and stats generation), structure and store call stats on a database for analytics, and do further analysis via notebooks on data science initiatives (e.g. sentiment analysis). This is akin to having a staging area for complex and diverse data types, and S3 is useful for this because of its speed, scale and management features.
Teradata or Snowflake aren't a great fit for use cases like these, but they are great if the use case is to get answers to questions like "top 3 operators per team in volume of calls, by department and region, in last quarter" if the volume of calls is big.
If I understood correctly, I think your comment was more focused on why use new tools when the existing are mature, but I think big data tools have had to become more specialized and targeted for specific use cases. But if the question is "why build more than one data lake", the only reason I can see is organizational: teams or different areas of an organization either need their own data lake because they have specific needs (which is rare) or won't/can't collaborate with others to have a shared asset.
>> Controlling cost is the hard part. You may only need a cluster for 1 hour per day for a nightly aggregation job. Kubernetes clusters are not easy to provision and de-provision, so you end up paying for a cluster for 24 hour days and use it for only 1 hour.
What is the benefit of using Kubernetes to deploy Spark jobs then? Is that approach meant to achieve independence from the hardware?
I'm asking because that is fairly trivial to achieve using, at least, a provider like AWS: you can build a CloudFormation template (or use the AWS API or the web UI) to launch AWS EMR clusters with specific hardware and run any spark jars, and you can use services like DataPipeline or Glue to schedule and/or automate the whole process. So you can use AWS services to set up a schedule that will periodically spin up a cluster with whatever machines you need to run a Spark app and decommission it as soon as its done.
In this case, the EMR cluster comes with the myriad of Hadoop tools and services (and Spark, and other relevant software) preinstalled and ready to use. And most relevant Spark settings are already optimized for the cluster's hardware; but not for the Spark app itself, which is what this solutions seems to address.
I've used Firefox since I cannot recall when, on Ubuntu 16 and 18, and Windows 7 and Windows 10.
Although I've experienced very rare crashes, like I have with Chrome/Chromium, my experience is that Firefox is stable and does everything I need, including some basic web dev/debug.
This is actually something that I'm missing in this analysis: walkthroughs and greeting messages are presented as a "must have", but unlike other conclusions in the report, I didn't see any data supporting these assumptions.
It's hard to quantify that "risk" in simple terms, but --disabled-password is a basic must as it seriously mitigates popular attack vectors like brute forcing.
Filtering ssh connections at firewall level helps, and certainly reduces log entries for port scans and can halt less sophisticated attackers, but it doesn't mitigate attack vectors like a DDoS or a well funded actor.
It stems from automated port scans sweeping IP ranges looking for vulnerable/exploitable systems.
A simple way to address it is to use something like fail2ban to ban repeat offenders. Blacklist services like abuseipdb.com may help further, but ban with caution.
Try to adapt the business processes to the ERP, instead of adapting the ERP to the custom business processes.
While I don't have hard data on this (and it would be hard to establish causality anyway), most of the ERP implementation failures I've witnessed and know about stem from line of business people who firmly believe their company's processes - e.g. collection, billing, etc - are very special and unique, when it is very rarely the case; and when it is, it shouldn't be, the process is likely overly complex due to inertia or lack of will/skills to improve or just legacy. Then, the customizations that need to be implemented to support the "special and unique processes" are so complex that the project's schedule and budget inevitably explode...
ERP implementations could actually be seen as a great opportunity to revisit, optimize and document current processes before the ERP is implemented...
More than 90% of the enterprises I've worked with use one of the enterprise tools listed in Gartner's magic quadrant, typically from the Leader quadrant, sometimes from the Niche quadrant, here are a few examples: https://duckduckgo.com/?q=gartner+magic+quadrant+data+integr...
What's actually mind boggling for me, and I wonder if it's a bit of over-engineering, is people going for complex setups (oh, it's just Airflow scripts with k8s and a little bit of SystemD services and configs plus some shell scripts) when there are COTS tools that do more for less engineering cost. Yes, these carry a price tag, but it's usually quite less than paying for engineers to babysit a tool with a ton of moving parts...
Not sure what you mean by "BI/data science architecture" but Airflow is essentially a scheduler and orchestrator for data processing jobs.
These activities are usually managed by cron and more often by advanced scheduler tools (depending on the vendor), so it's quite a core part of any architecture that needs to e.g. load/reload/refresh data periodically.
If the requirement is simply to connect notebooks to a data lake, then the only scheduling required is to load the data lake, and something like Airflow may be overkill for this, depending on what/how the data is processed and loaded.
I partially agree with you; unless the goal was to "do this in Go", the choice of tools seems odd/inneficient.
Spark would have a been a simple option to do this kind of processing, with less lines of code, and could also run on "spare compute". Same goes for the "How does one process 10 million JSON files taking up just over 1 TB of disk space in an S3 bucket?": there are appropriate file formats for storing and querying big datasets, text/json is simply the least efficient option and likely the cause of the "$2.50 USD per query" number...