I deployed a Lambda function behind CloudFront which rendered a simple HTML page with the query results from executing some SQL over the dataset. I served millions of page views for next to nothing because most pages were already in the cache.
I don't know where you get this expectation that people should anticipate that a crawler might try every possible combination of query parameters, thereby missing the cache on each one. Most people consider it a bitter and arrogant perspective, which is why this got downvoted.
Yes, if I wanted to put a nice HTML interface over the query results then I would still end up with the same problem where some combinatoric explosion of query parameters to the `/search` endpoint, most of which are cache misses, leads to many many TBs of network egress.
Completely agree, I've worked at several companies where I saw the AWS bill balloon from five to six figures, usually because of pointless over-provisioning. However, it all became worth it when I saw Jeff Bezos go to space. [1]
They were both more exhaustive and more frequent. I don't remember the exact numbers, but it was definitely over 100x the traffic from search engines. By the way, search engines were allowed under robots.txt - I did want all 11.5 million loans to be individually indexed so they would pop up in Google search results, and I actually did receive/forward multiple tips about fraudulent loans because someone searched a business name on Google. All of this traffic was barely a blip, and my AWS bill for my hobby data science account was only ~$50-$100/month.
When I looked at the logs after getting the billing alert, 99.99% of the requests were to the "/search" endpoint with virtually every permutation of ~10-12 facets in the query parameters. There was only one scraper, but it triggered an enormous amount of network egress since it ended up missing the cache on the majority of queries.
A couple years ago, I was running a website which allowed the public to view all the US government handouts to small businesses during the COVID-19 pandemic. It also tracked all the fraudulent loans being prosecuted by the DOJ, and allowed anyone to run structured queries over the public dataset. There was a "donate" button which took in ~$2k in donations over the lifetime of the site, and you could download the entire underlying dataset (around 10 GB uncompressed) for free directly on the site.
Despite the "download all data" link being prominently placed on the front page, the AI scrapers decided it would be more efficient to download terabytes upon terabytes of raw HTML by paginating through every possible facet on the search endpoint. Even with CloudFront caching results and a fairly efficient backend setup, the monthly bill ended up with around $1k just going toward network ingress/egress, so I shut down the site the following month.
If Boeing claimed a plane was airworthy, but it crashed because basic engineering controls were skipped, we have collectively put our faith in the NTSB to preserve evidence, run an independent technical investigation, etc. There is no such authority for software - most security auditors (SOC2, HITRUST, etc) are just looking at self-reported data.
Just take a look at the recent Epic vs. Health Gorilla lawsuit to see how nonexistent the protection is around exchanging your medical records, one of the most sensitive types of PII.
Missing from the article - the hacker first compromised Resolv Lab's AWS account, took a private key from KMS that was used to control minting, then managed to extract $25 million into ETH before all protocol functions were suspended.
English is full of animal-based metaphors, and that's a pretty innocuous statement. "I herded everyone into a room" does not automatically imply that one perceives those people as animals.
The most telling sign of a human commenter is brevity.
Consequently, I hardly ever spend the time to write out long and detailed HN comments like I used to in the pre-LLM era. People nowadays have a much harder time believing that an Internet stranger is meticulously crafting a detailed and grammatically-airtight message to another Internet stranger without AI assistance.
The Puya PY32 series MCUs found in most vapes have 3kb of RAM and 24kb of ROM, whereas Doom requires at least 4MB of RAM. Assuming Moore's law also applies to the computing power inside a disposable vape, we should be seeing that post in around a decade :)
The fact that this occurred in Bombay is important context. In India, the culture amongst older people is to have a clear sense of where you fit in the hierarchy. You might be verbally abusive to those who you consider below you, but you will remain silent and deferent to those who are considered economically/socially superior. This manifests as a certain class of people who have never been called out on any of their obnoxious behavior, because their economic/social status has shielded them from criticism for their entire lives. Meanwhile a majority of society is perfectly accustomed to being verbally abused, to the point where someone like me saying "please" and "thank you" makes it clear that I am of the Indian diaspora.
By the way, I've noticed that the younger crowd in India leans much more toward egalitarianism and tends to reject bizarre social constructs like caste. The fact that a young guy also thought of this solution speaks to their ingenuity as well.
I wouldn't call CareQuality a "data broker" as much as it is an interoperability framework. It's essentially a big distributed system of participants who agree to instantly share patient records - CareQuality maintains a central list of participants and the URLs where they can be reached. Since the technical requirements to actually participate in this network are fairly complex (far more than a hospital IT can or should manage), there are companies like Health Gorilla which serve as QHINs (Qualified Health Information Networks) which query the network on behalf of their customers (i.e. doctors engaging in some form of care).
There are many gray areas to this - for example, a value-based care program or ACO can pull records en masse, for the purpose of "care coordination" (i.e. checking if a particular patient requires intervention). However, what Health Gorilla has done is certainly no gray area as some articles on this matter suggest - if the allegations are true, then they have engaged in outright criminal behavior along with their co-conspirators (RavillaMed, LlamaLab, and others). Thankfully, this situation has completely eroded all trust in Health Gorilla and prompted a massive customer exodus.
If this actually happens, get ready for an avalanche of AI-generated garbage code that exists for the sole purpose of boosting a scammer's metrics, so they can maximize their slice of the pie with the minimum amount of effort. Spotify is dealing with this same issue around AI-generated music [1].
Thank you for the excellent feedback. I had this realization a while back that I'm a mobile user during "consumption" (e.g. browsing HN late at night), but a desktop user for "production" - now I see how it applies to this side project as well. Also, I still need to figure out some React performance issues which make it virtually unusable on pre-2020 machines :(
This comment actually invigorated me to try the site from my phone and improve the experience, so I sincerely thank you for the motivation.
I love the Typescript handbook, but wanted the examples to be "runnable". It turns out that the TypeScript compiler runs pretty fast in the browser for trivial code snippets, so I threw together https://ts.coach (TypeScript handbook with code examples that execute in the browser + instant type checking)
You are effectively saying that the indiscriminate slaughter of the Japanese civilian population was justified, due to the actions of a few Japanese leaders. In my opinion, there is no justification for violence against civilians.
Notice that we have a holiday for MLK, and Indians have a holiday to celebrate Gandhi. Something deep inside all of us knows that pacifism is “correct”.
In my reply above, I evoked the memory of Jeanette Rankin, who was the lone dissenting vote against the Pacific War after the Japanese attack on Pearl Harbor (somewhat analogous to the October 7th attack).
It is a natural human tendency to desire that the people who inflict pain upon others to also feel pain inflicted upon them. This has been the human condition since ancient times, and yet the most revered figures in human history have been the pacifists who consistently advocate against violence (e.g. Gautama Buddha, Jesus Christ, Lao Tzu, Gandhi, MLK, etc).
https://duckdb.org/2021/10/29/duckdb-wasm