I think they presented it wrong for impact. If instead of "killed" and "eliminated" they were saying that the internet made it possible to proceed without a sales team and that cloud computing made it possible to not need a bare metal IT team, it makes a little more sense. That is to say, it's not going to be the most effective way, but it went from impossible to at least viable. Similar to music recording. It used to be you needed competent instrumentalists and access to a commercial recording studio. Now you can do it all at home on a laptop and it is viable, people are successfully doing it. But in large part the most successful music is still done using the same professionals that existed before.
In a similar vein, Instagram was the first platform where I actually took a pause and realized someone had actually figured out how to give me appealing advertising over the internet. I don't use Facebook to know how they differ, it may be practically identical.
My experience is that I use my Instagram account not to interact with my social group but to engage with content I like (primarily art content like sculpture, painting, light/projection, music, etc). It's actually a really pleasant experience and I'm doing my best to protect my groove in the recommendation algorithm. I basically get an effortless feed of art events in my area without me needing to subscribe to a newsletter. And it also engages me with a lot of independent artists selling unique items that I like.
> If you do it by taxing the rich, you wind up hobbling and punishing your most productive citizens
We may have different definitions of "rich" here. I don't think you can call them the most productive citizens. I think most extractive would be the more accurate label.
And the stimulus checks were short lived, not universally distributed, and came in under $1T. The majority of the money spent was on other sources, even if you want to include the boosts to unemployment in the "UBI" category. As others mentioned it's also conflated with other factors like supply chain failures and massive business loan fraud.
The 30 days part is most likely a legal compliance bit to cover their asses if their backing data systems ever take a big dump.
They need to use an asynchronous system to be tracking when your chat becomes "finished" and then likely queue it up to a system looking to propagate deletes. They have to choose some kind of SLA on that and probably went with a common data privacy user data deletion window of 30 days.
Also a lot of things at Trader Joes are just white labeled versions of other products. As in, literally the same product, just inside packaging that says "Trader Joe's <Similar Name to Other Product>"
Was MBS (mortgage backed securities) a typo here? My understanding was SVB had a huge long term treasury position on the books from before interest rate hikes that was in the red against the market. And their second largest tier was direct loans to things like startups that weren't asset backed but instead default to warranties on the companies taking the loans (I'm not close enough to this to really say that more specifically).
Why does high school have to go first in the morning? When I was in elementary school I would wake up at 5:30-6am naturally and watch TV before school. Come high school and I was constantly not getting enough sleep with the earlier start times and setting my own bed times. I think high school aged children have more need for a later start than elementary school aged children.
I would have to defer to one of my colleagues for most of the details on running data infrastructure on Kubernetes, I’m not that close to that domain. The major ones we run are HBase, Vitess (MySql), Kafka, Elasticsearch, Memcached, and Zookeeper.
As for the call sampling/interception, that did not factor into discovering the high cost buckets in the logging case study. It was mostly relevant to generally describing how we track costs and it ends up being useful in other scenarios. For example it could be used to assess the estimated unit economics of customers subscribed into a specific product tier.
We also have the death star microservice model, so even relatively simple attribution can be helpful when you want to run a query like “for my team which owns 30 applications, tell me the monthly attributed cost grouped by resource” and that will be able to return all the associated database and cloud costs.
Your observations are correct. I wouldn’t portray it as an ideal system, just best effort. In the end we care more about the finer details being good funnels to follow up on rather than being exact. We know our real costs of resources, which is important for finance and budget. And then we have the approximate attributions from the sampling which narrow things down enough to focus diagnosis when needed.
I would have to read more into how it intercepts some of our database calls to confirm if it tries to weight for execution time where it wires into database client code, which is probably useful and could help to a degree to approximate utilization.
I think in practice it’s a bit uncommon for the heaviest user to also be a sparse user in terms of volume. But I can also admit there are quirks to how it samples and I once personally spent a couple days tracking down a surprising cost of an application I owned only to later confirm it was a data flaw in how we were doing this sampled attribution (in this case the heaviest users were un-instrumented infra processes that can’t just wire in our java cost attribution library, making it artificially look like my app was the heaviest user).
You’re right, it’s not an either or for this as we tackle both less total data and making it smaller, although I probably failed to clarify that in this first post.
At 20% of storage costs, the should makes a lot of sense to focus on. Once it becomes 1% of storage costs it’s maybe not as problematic though. The magnitude to which, “let’s compress the logs” changes how much something like “am I logging too much” matters is important. Taking it to the absurd, if logging storage were free why not retain all logs. And if logging is cheap, why invest in complicated guardrails for what qualifies as important logs.
A specific consideration for us is organizational inertia. We have a lot of teams using infrastructure in a lot of ways both intended and unintended. One thing, for better or worse, that has been emphasized for us is developer velocity. Which includes things like abstracting “do I need to log this” from most engineers. We have some guardrails to alert if you do some egregious volume.
I think we do often opt for non-invasive infra solutions first because they have much shorter delivery times and less risk of stalling on long-tail outliers. They avoid very expensive organizational costs of buy in and team-level migration. I’m not suggesting this is the best organizational model, but it also transcends one team’s influence.
That ends up circling back around to the start of the problem. If we can transparently reduce the cost burden of some heavily-used internal infrastructure within the same relative magnitude of applying a paradigm shift on the usage of the same internal infrastructure, the former wins out.
The costs I talk about in the estimate in this post are for the remaining cost of each stored file. We have S3Inventory dumping metadata of all the files in specific buckets weekly, so I had written a job that calculated the exact remaining cost of each file, accounting for lifecycle events like moving to infrequent access storage in S3 and the eventual deletion of the file. So it’s sort of the “potential energy” version of the cost of our stores files. If we take no action they will aggregate to a certain amount of money.
I reckon you may be looking at the monthly cost of storage per gigabyte which is why the number doesn’t seem to make sense. Our retention policy started off at about 2 years, so the remaining lifetime per file amortizes out to much more than 1 month.
Also worth considering that we have a custom AWS contract, so none of our actual numbers are the publicly advertised rates and probably won’t entirely math out if you try to ballpark with those numbers.
That's a fair criticism in the edit. Part 2 will cover that a bit more. I did run analysis on the types of queries users ran against the data and what parts of the timeseries were used, which informed a bit of our solution. I don't want to give away too much, but lifecycle retention adjustment ends up being relatively lower value (but still worthwhile) compared to general space savings.
(Author here). Yes I believe you are correct with regards to tracking application utilization of say EC2 and other AWS resources.
The post fails to mention this system is also tracking internal data dimensions like customer ids, such that we can also use this sampled data to estimate the cost of customers (and joining that with tiers of customers, and so forth).
I'm also not sure if that would allow us to attribute the cost of our datastore utilizations since those are not AWS-hosted versions but ones we run ourselves. The traffic interception lets us be able to say that Application A is using 75% of database cluster XYZ, and therefore that application/product group are most likely responsible for that share of how much the database costs.
The last thing I'll mention is that CloudTrail has the potential to be expensive on its own, I believe at least moreso than us storing the raw data in S3 for something like Athena to read. I don't think I'll be writing about it, but we've also done work this last year to trim down what we track in CloudTrail due to the cost of events (for example tracking everything in S3 ends up being pretty expensive).
It's a bit more complicated though, interest rates are part of that function too.
Low prices & low interest rates: great for new entries to homeownership, but would require a surplus of housing
Low prices & high interest rates: great for those who buy property with cash (wealthy people and corporations)
High prices & low interest rates: great for existing owners
High prices & high interest rates: generally bad for everyone except those able to offload existing housing thy own
Is it really a negative feedback and not just transient though? Sure it may cool things during the fire, but it also just destroyed some carbon sinks and worsened the overall emissions effect. Then the fire ends and it no longer has any cooling effect so its not feeding back.
Stardock released their own spiritual successor already, so give that a try if you want. And as far as I'm aware the original creators are still working on their own successor.