How Ahrefs Gets a Billion Dollar-Worth Infrastructure with a 90% Discount(tech.ahrefs.com)
tech.ahrefs.com
How Ahrefs Gets a Billion Dollar-Worth Infrastructure with a 90% Discount
https://tech.ahrefs.com/how-ahrefs-gets-a-billion-dollar-worth-infrastructure-with-a-90-discount-5edd473b2399
4 comments
https://en.m.wikipedia.org/wiki/Wikipedia:Articles_for_delet... seems to be the most recent deletion discussion, from 2020
This conversation is cringy. Ahrefs is a great tool.
This reads like 99% of the other Wikipedia debates I've seen.
First they ignore you...
Nice writing!
My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation.
On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.
My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation.
On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.
Running your own data center is just a PITA no one wants to do. Non-C-levels just love to ignore the costs you have when doing such a thing by just comparing hardware costs to the cloud costs. There's always the liability aspect, managing less people, externalizing some complexity to another company etc. etc. All worth a lot of money, at least for me personally. I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already.
I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
> I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
Not sure I got this point right - if you need to create you own product you will need to hire programmers as well and get right people. Same for PMs, Security, Infra guys as I see it.
Not sure I got this point right - if you need to create you own product you will need to hire programmers as well and get right people. Same for PMs, Security, Infra guys as I see it.
> if you need to create you own product you will need to hire programmers as well and get right people
SWEs working on a product are part of a different budget (R&D) from IT (Finance & IT).
As a tech company, you can justify high R&D (they build the product) and Sales&Marketing (they sell the product) spend, but you cannot justify Finance & IT spend.
This depends company to company ofc, but for plenty of firms, it's fine to eat the Public Cloud cost because DevOps can be treated as part of the R&D bucket, and there are common metrics that all companies need to follow (some due to experience, others due to inertia).
SWEs working on a product are part of a different budget (R&D) from IT (Finance & IT).
As a tech company, you can justify high R&D (they build the product) and Sales&Marketing (they sell the product) spend, but you cannot justify Finance & IT spend.
This depends company to company ofc, but for plenty of firms, it's fine to eat the Public Cloud cost because DevOps can be treated as part of the R&D bucket, and there are common metrics that all companies need to follow (some due to experience, others due to inertia).
Thank you, interesting.
Then "infra" would be under "IT" in such kind of setups?
From my experience it was bit different - say Infra, Dev, Security departments having their own Head Of Dev/Infra/Sec reporting to the CTO and having their budgeting aligned with CTO. CorporateIT being a bit different thing. Procurement may be shared among all of them of course.
Not saying that's the only possible setup of org structure - just what I've used to more.
Then "infra" would be under "IT" in such kind of setups?
From my experience it was bit different - say Infra, Dev, Security departments having their own Head Of Dev/Infra/Sec reporting to the CTO and having their budgeting aligned with CTO. CorporateIT being a bit different thing. Procurement may be shared among all of them of course.
Not saying that's the only possible setup of org structure - just what I've used to more.
> Then "infra" would be under "IT" in such kind of setups?
Depends what you mean by "infra".
The team that would manage your DC (either self managed or CoLo) would generally be under the CFO or COO (who will generally report to CFO), and as such that team would fall into the Finance procurement bucket.
Even if R&D has budget for multiple racks in your DC, it will still cost money on the Finance&IT side of the house to staff people to maintain it and manage it.
This model is very clunky, because procurement takes time, and this lack of nimbleness can drastically affect delivery roadmaps.
CFOs also don't like this model as much, because their job is to manage cash flow, keep books and accounts ready for audits, build financial projections, etc and time consuming work like managing IT procurement (while critical) is a major slog.
The model you mentioned is more common now (Platform/Infra team acting as an interface w/ Eng), but even they would offload DC and CoLo management to the CorpIT team.
Depends what you mean by "infra".
The team that would manage your DC (either self managed or CoLo) would generally be under the CFO or COO (who will generally report to CFO), and as such that team would fall into the Finance procurement bucket.
Even if R&D has budget for multiple racks in your DC, it will still cost money on the Finance&IT side of the house to staff people to maintain it and manage it.
This model is very clunky, because procurement takes time, and this lack of nimbleness can drastically affect delivery roadmaps.
CFOs also don't like this model as much, because their job is to manage cash flow, keep books and accounts ready for audits, build financial projections, etc and time consuming work like managing IT procurement (while critical) is a major slog.
The model you mentioned is more common now (Platform/Infra team acting as an interface w/ Eng), but even they would offload DC and CoLo management to the CorpIT team.
The structure of the company depends on the needs.
When you have a tiny Corp IT, almost invisible in a company's annual budget, and nobody in the company wants to care about it (a lot of pain but zero rewards) -> it goes to CFO. You have Corp IT as a major cost center which is bigger than all the other spending altogether (mature IT companies) then you want someone knowledgeable and trustable to run it, usually CIO.
Sometimes, Corp IT as internal infrastructure supporting employees, development, backoffice is under the same person running production for the clients. Then standards of the internal IT and external services are the same (and very hard for internal services to achieve).
In other cases, production and corp IT are different departments with not so funny unresolvable collisions, like buying incompatible hardware for testing environment (internal services) and production.
When you have a tiny Corp IT, almost invisible in a company's annual budget, and nobody in the company wants to care about it (a lot of pain but zero rewards) -> it goes to CFO. You have Corp IT as a major cost center which is bigger than all the other spending altogether (mature IT companies) then you want someone knowledgeable and trustable to run it, usually CIO.
Sometimes, Corp IT as internal infrastructure supporting employees, development, backoffice is under the same person running production for the clients. Then standards of the internal IT and external services are the same (and very hard for internal services to achieve).
In other cases, production and corp IT are different departments with not so funny unresolvable collisions, like buying incompatible hardware for testing environment (internal services) and production.
It’s an own goal, why not have infrastructure as a sibling team to the dev team. This is a problem only because I make it a problem, situation
This is an organizational structure that is common industry-wide.
Core IT (not Infra - Infra often means DevOps/Platform which is part of R&D), like any other shared function, is placed under the CFO/COO org because it is a shared internal function that's used to keep the lights on.
These are functions that by definition are a cost center, and as such R&D really doesn't want to own it, as it distracts from the core mission of delivering forward facing features.
The whole point of the DevOps/Platform Eng model was to incorporate operations under R&D, but having R&D own and manage physical assets is a major distraction, and a major reason Public Cloud became popular among plenty of companies.
Cost of compute isn't the only variable that comes to play - depreciation, hiring/firing, security, redundancy planning (because these are physical assets), etc are all significant headaches operationally and financially.
Technology is a tool - you choose the tool that works best. In some cases Public Cloud is the solution, in other cases it's a CoLo. It comes down to your specific organization's needs.
Core IT (not Infra - Infra often means DevOps/Platform which is part of R&D), like any other shared function, is placed under the CFO/COO org because it is a shared internal function that's used to keep the lights on.
These are functions that by definition are a cost center, and as such R&D really doesn't want to own it, as it distracts from the core mission of delivering forward facing features.
The whole point of the DevOps/Platform Eng model was to incorporate operations under R&D, but having R&D own and manage physical assets is a major distraction, and a major reason Public Cloud became popular among plenty of companies.
Cost of compute isn't the only variable that comes to play - depreciation, hiring/firing, security, redundancy planning (because these are physical assets), etc are all significant headaches operationally and financially.
Technology is a tool - you choose the tool that works best. In some cases Public Cloud is the solution, in other cases it's a CoLo. It comes down to your specific organization's needs.
Absolutely. But I can pay someone else to do one specific part very good, so I don't need to manage and hire for that department. I really don't see a use case to do CoLo or DC. Even at ahrefs scale, they probably have a lot more they can optimize before you start going down that avenue.
this appears to be survivor bias from two decades of fast money and those that handled it
Not really, no.
> Running your own data center is just a PITA
I believe in the article they clearly say they just use colo, not running own DCs
> I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already.
This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV.
No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
I believe in the article they clearly say they just use colo, not running own DCs
> I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already.
This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV.
No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
Maybe my point didn't come across properly. Me personally, I prefer to keep the org complexity down and externalize some headcaches if I can. Having to also manage hardware, people that manage this hardware and not being able to keep other people liable for a given SLA. It's just yet another thing that other companies are really good at and for me it's just not worth it to do on our own. If I don't really need it, why would I? And again, there are more costs associated with this approach than just infra vs hardware + people costs.
Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way.
In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way.
In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation.
I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
> Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation.
I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort.
> Second, connecting all the piping gives an illusion of doing work and feeling "important".
This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort.
> Second, connecting all the piping gives an illusion of doing work and feeling "important".
This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
I’ve been doing infrastructure for 27 years.
Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups.
One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility.
Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility.
They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands.
But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money.
So they moved to Amazon.
Their first month’s bill was $50,000. And it only went up from there.
The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups.
One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility.
Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility.
They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands.
But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money.
So they moved to Amazon.
Their first month’s bill was $50,000. And it only went up from there.
The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
> But their new VC-installed CTO was like… No!
Largely the point I'm trying to put here
> Their first month’s bill was $50,000. And it only went up from there.
Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
Largely the point I'm trying to put here
> Their first month’s bill was $50,000. And it only went up from there.
Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
You're mixing yearly salary figures and monthly infra bills.
50k*12=600k, equivalent to 1-3 SWE employees; practically not even a single small team. It's nothing.
Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
I’ve seen ICs lose promotions they were on course for and get wrecked on their bonuses for losing track of a single $50k/mo cluster in AWS and leaving it running though unused. At a FAANG company.
So, ICs who sweat that kind of waste are right to do so. If management sees that someone thinks 50k/mo is no big deal to squander, bet your ass there’s a good chance they’ll take at least a month’s worth out of that person’s annual bonus since that kind of money matters so little to them.
So, ICs who sweat that kind of waste are right to do so. If management sees that someone thinks 50k/mo is no big deal to squander, bet your ass there’s a good chance they’ll take at least a month’s worth out of that person’s annual bonus since that kind of money matters so little to them.
Sure, that money shouldn't be wasted - but thinking it's a significant part of the budget is misguided.
Not sure why you were down moderated for pointing this out. Facts is facts.
I’ll bet this isn’t apples to apples comparison and the bill was for a lot more resources.
Ec2 or Ecs Clusters per developer, leave experimental shiny cloud service running … etc
You’d lose that bet.
Source: I was there
Source: I was there
In my eyes, physical hardware is something non-growth companies do to improve margins.
When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
Exactly. Dropbox was cloud and then later moved to on-prem to save on storage costs.
Anyone running >1000 servers with >80% utilization should consider going partially colocated.
Comma.ai and Tesla come to mind who run their own training clusters.
If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy.
Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.
Anyone running >1000 servers with >80% utilization should consider going partially colocated.
Comma.ai and Tesla come to mind who run their own training clusters.
If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy.
Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.
It's investors and the one-size-fits-all metrics many rely on. They want a company's financial statements to look like the financial statements from other financial instruments (companies) they're familiar with. If your industry has 30% EBITDA, your company needs to have ~30% EBITDA. If your industry is 100% cloud, your company is expected to be ~100% cloud. If your numbers deviate from expectations, they'll demand an explanation. If that explanation isn't your company's secret to success, then it's probably not going to be accepted.
I think these articles are great.
It's about understanding what you do and what infra is appropriate.
I have to wonder how that would look with a negotiated AWS discount. But they might get beaten to death by incoming bandwidth costs anyway.
It's about understanding what you do and what infra is appropriate.
I have to wonder how that would look with a negotiated AWS discount. But they might get beaten to death by incoming bandwidth costs anyway.
Netflix is a flagship example of AWS usage. https://aws.amazon.com/solutions/case-studies/innovators/net...
We don't know what are Netflix discounts. Maybe Netflix uses AWS just for free to make others pay. Maybe AWS even pay to Netflix to continue attracting the other guys.
On the other hand, just read this disclosure in the Netflix 2023 annual financial report: "We have architected our software and computer systems so as to utilize data processing, storage capabilities and other services provided by AWS... Given this, along with the fact that we cannot easily switch our AWS operations to another cloud provider, any disruption of or interference with our use of AWS would impact our operations and our business would be adversely impacted. While the retail side of Amazon competes with us, we do not believe that Amazon will use the AWS operation in such a manner as to gain competitive advantage against our service, although if it were to do so it could harm our business."
On the other hand, just read this disclosure in the Netflix 2023 annual financial report: "We have architected our software and computer systems so as to utilize data processing, storage capabilities and other services provided by AWS... Given this, along with the fact that we cannot easily switch our AWS operations to another cloud provider, any disruption of or interference with our use of AWS would impact our operations and our business would be adversely impacted. While the retail side of Amazon competes with us, we do not believe that Amazon will use the AWS operation in such a manner as to gain competitive advantage against our service, although if it were to do so it could harm our business."
> We don't know what are Netflix discounts
What you do, if you're a VC backed SaaS company with a six or seven figure monthly spend on AWS, is hire away someone from Netflix who does know what their discount is and use that when your next contract negotiation is up with AWS.
What you do, if you're a VC backed SaaS company with a six or seven figure monthly spend on AWS, is hire away someone from Netflix who does know what their discount is and use that when your next contract negotiation is up with AWS.
Why does knowing the discount give you any additional leverage?
Knowing X get Y% discount would give you the opportunity to ask for the same Y% discount. Not just AWS, in enterprise world the service provider will always make the % discount a secret and you can’t tell anyone else.
Almost universally, you get taken more seriously when you demonstrate you understand the exact quantity of leverage you have.
“I know this is possible because we had it at X” is a lot different conversation than “We want X”. Your leverage is even worse when X is nowhere near reality (in either direction).
“I know this is possible because we had it at X” is a lot different conversation than “We want X”. Your leverage is even worse when X is nowhere near reality (in either direction).
What we've seen with AWS (and we only discounted like 2 services) is that if you commit to a monthly spend amount on a service you can get substantial discounts on that service.
There are EC2 plans as well, which I assume means that you can get discounts off the reserved instance pricing - but you'd have to ask your AWS rep about that.
If you spend > 4 figures on an AWS service you should ask for their discount plan. Nobody wants to say what they get discounted on, because they might lose them (and the contracts are confidential). So you have to ask your rep.
There are EC2 plans as well, which I assume means that you can get discounts off the reserved instance pricing - but you'd have to ask your AWS rep about that.
If you spend > 4 figures on an AWS service you should ask for their discount plan. Nobody wants to say what they get discounted on, because they might lose them (and the contracts are confidential). So you have to ask your rep.
from AWS/GCP/Azure/you_name_it's perspective, it may be worth to give that 90% discount extra, just to ensure
1. keep shrinking workforce being able to keep doing such kind of setups
2. avoid such writeups with _numbers_ at all
1. keep shrinking workforce being able to keep doing such kind of setups
2. avoid such writeups with _numbers_ at all
The reason I ask is that we (who are super small) got a massive discount on cloudfront bandwidth by just committing to a $2k/month spend - a discount that was more than 90% off of list.
For a spend their size they should be able to get a much bigger discount.
It would probably behoove them to ask AWS about it, and to report back.
At some point AWS just wants their capacity to be utilized. Unused CPU and memory are like empty seats on the plane: you might as well fill them up because you're losing money on those empty seats when you take off.
For a spend their size they should be able to get a much bigger discount.
It would probably behoove them to ask AWS about it, and to report back.
At some point AWS just wants their capacity to be utilized. Unused CPU and memory are like empty seats on the plane: you might as well fill them up because you're losing money on those empty seats when you take off.
I don't doubt that physical hardware would be cheaper than AWS at their scale, but their graph seems off: They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive. Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing?
> They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive.
That's exactly what is happening when you look at things without depreciation. You pay 3y all upfront. And it's a lot of money: a high step on the graph. On demand will grow steadily and in three years, it will surely be higher than 3y reserved instance. But we are not comparing one instance. We add more ever more powerful servers as they are added in any infrastructure. And we pay upfront. On demand price can't catch up to those all upfront costs on our period of time.
To see it clearly, we fix the infra on the 4th graph. And then on-demand becomes much more expensive. Also, you may look at the last paragraph in the Appendix to see why 3y all upfront is not horizontal between the steps.
> Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing?
Yep.
That's exactly what is happening when you look at things without depreciation. You pay 3y all upfront. And it's a lot of money: a high step on the graph. On demand will grow steadily and in three years, it will surely be higher than 3y reserved instance. But we are not comparing one instance. We add more ever more powerful servers as they are added in any infrastructure. And we pay upfront. On demand price can't catch up to those all upfront costs on our period of time.
To see it clearly, we fix the infra on the 4th graph. And then on-demand becomes much more expensive. Also, you may look at the last paragraph in the Appendix to see why 3y all upfront is not horizontal between the steps.
> Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing?
Yep.
Super interesting article though, and a nice addition to their prior one.