Our digital lives need data centers. What goes on inside them?(washingtonpost.com)
washingtonpost.com
Our digital lives need data centers. What goes on inside them?
https://www.washingtonpost.com/dc-md-va/interactive/2024/data-centers-tour-northern-virginia/
11 comments
From my own adventures in colocation, i came to conclude that data centers are Narnia, in the worst possible way: you enter to perform what feels like 30 minutes worth of work, and somehow 8 hours pass before you see the outside world again.
They've got all the hallmarks of a casino. Zero natural lighting, tons of blinking lights, and the occasional dopamine hit when a server goes down.
> dopamine hit when a server goes down
You mean adrenaline?
You mean adrenaline?
Adrenaline for when it goes down, dopamine for when it's back up again.
But minus the overly ostentatious carpet, and the faint smell of cigarettes.
And this is why I love colocation.
Knowing you're sitting next to a rack of blinking lights of god knows what.
Knowing you're sitting next to a rack of blinking lights of god knows what.
Reminds me of my trips to Fry's electronics in the old days. ;-)
I am simultaneously impressed and depressed by data centers. I still get the same frisson I used to get when I was 20, when I enter the space, with all the security theater, and the noise, and the blinkenlights.
What depresses me now though is thinking of all the energy that it takes to power and cool a DC, and for what? So some sloppy code can hog the CPU 100x more than it needs to? So we can train AI models because everyone else is doing so?
I feel its high time we had a green software revolution, where we strive for code efficiency not just as an intellectual exercise but because it might help save the planet. And save the bottom line of companies that are paying to run their stuff in these environments.
What depresses me now though is thinking of all the energy that it takes to power and cool a DC, and for what? So some sloppy code can hog the CPU 100x more than it needs to? So we can train AI models because everyone else is doing so?
I feel its high time we had a green software revolution, where we strive for code efficiency not just as an intellectual exercise but because it might help save the planet. And save the bottom line of companies that are paying to run their stuff in these environments.
I'm glad someone else feels the same way, it can be kind of isolating to want to be efficient at the expense of speed. There are important features to be shitting out, dont-cha-know.
It feels the same way a neglected pothole does: the city doesn't want to send a crew out for half a day (uh oh it might cost a couple thousand bucks!) to fix a pothole that five thousand cars each and every day are running over, accumulating wear and tear, discomfort to drivers and riders, day after day, week after week. No itty-bitty investment to fix the problem that might benefit so many others, similar to how we could invest a little bit of time (city money) to fix a little inefficiency here and there (that dastardly pothole). It just makes too much sense.
It feels the same way a neglected pothole does: the city doesn't want to send a crew out for half a day (uh oh it might cost a couple thousand bucks!) to fix a pothole that five thousand cars each and every day are running over, accumulating wear and tear, discomfort to drivers and riders, day after day, week after week. No itty-bitty investment to fix the problem that might benefit so many others, similar to how we could invest a little bit of time (city money) to fix a little inefficiency here and there (that dastardly pothole). It just makes too much sense.
Depends on the center and how its run.
Some of them are very human accommodating, and seem to understand that today's engineers might be tomorrows customers.
Others seem to go out of their way to create anti human environments. Hot aisle is literally 38cm wide? Doors wont open? Lifting tools are in an undocumented location? Rack uprights randomly unscrewed? Specific permission required to open a roller door so you have to wait 3 hours for an escalation request to be processed before you can reach the guy on the other side. 24/7 access conditional on the 1 security guard for 3 data centres being available to let you in.
My favorite was the one that had declared bankruptcy with our hardware still inside. No staff(despite 24/7 staff guarantee), but we had some limited after hours access pass.
Between the pass, some keys we "found" + friendly landlord disabling the alarm system for us, we got in, saw fans operating in place of the aircon, dismantled our rack and bailed.
Some of them are very human accommodating, and seem to understand that today's engineers might be tomorrows customers.
Others seem to go out of their way to create anti human environments. Hot aisle is literally 38cm wide? Doors wont open? Lifting tools are in an undocumented location? Rack uprights randomly unscrewed? Specific permission required to open a roller door so you have to wait 3 hours for an escalation request to be processed before you can reach the guy on the other side. 24/7 access conditional on the 1 security guard for 3 data centres being available to let you in.
My favorite was the one that had declared bankruptcy with our hardware still inside. No staff(despite 24/7 staff guarantee), but we had some limited after hours access pass.
Between the pass, some keys we "found" + friendly landlord disabling the alarm system for us, we got in, saw fans operating in place of the aircon, dismantled our rack and bailed.
>Northern Virginia — home to the largest concentration of data centers in the world
Beside other factors, it seems there is decent concentration of nuclear power plants in the area.
And it seems we'll see even more close ties between those technologies - Oracle talks nuclear reactors as a part of the future data centers, and MS partnered to start a Three Mile reactor specifically for MS datacenters.
Beside other factors, it seems there is decent concentration of nuclear power plants in the area.
And it seems we'll see even more close ties between those technologies - Oracle talks nuclear reactors as a part of the future data centers, and MS partnered to start a Three Mile reactor specifically for MS datacenters.
I guess I only use the cloud for my email. No social media either. The climate aspect of these data centers frightens me that I took all my stuff off the cloud and store it locally. Even my email I store locally with thunderbird.
RE climate argument, your marginal impact on the environment is likely going to be worse with storing stuff locally vs. in the cloud. Even including the extra security/maintenance work that you don't need locally, the data center is still going to have more efficient hardware that's managed better, and gets all kinds of economies of scale, including efficiency (cooling) and utilization (via virtualization, if you're just using a service or VM and not colocating).
I say that as someone who prefers local-first and avoid cloud dependencies like a plague.
I say that as someone who prefers local-first and avoid cloud dependencies like a plague.
My photos are on my internal SSD drive. I don’t know how who having them on a cloud server is better ecologically. I don’t access my old photos really, sort of like a photo album under the coffee table.
Taking less or no photos is kind of my thing now anyway.
Taking less or no photos is kind of my thing now anyway.
So you're decreasing your ecological impact by simply not doing what other people do.
In other words it's not that you're local vs cloud. You're using local-S3-glacier and other people are using cloud-S3 + cloud-EC2.
> So you're decreasing your ecological impact by simply not doing what other people do.
Yes. How else am I supposed to do it?
Yes. How else am I supposed to do it?
I think that's a great way to do it! It's just not a cloud vs local argument, more of a more vs less argument
re: the climate aspect, it's actually better to have some services like email operating in a data center. Think about all the idle no-op cycles your home email server is burning through. A million individual email servers will consume orders of magnitude more electricity than those same million+ accounts hosted in a data center. Efficiencies of scale.
I searched and cannot find the answer in their archives: has the Washington Post, or any other nationally prominent newspaper, ever written coverage from this standpoint, but about an oil refinery?
One thing that could explain this dichotomy is that writing such an article for oil refinery would be expensive task. Because data for such research may not be readily available online (not focussing on intentions). Just a guess. You will find more articles about things that are
easy to write about. If an article requires lot of field work, on foot research, talking to many people and synthesize and piece together disparate information in a coherent manner, don't expect it to be commissioned.
I at least expected a description of the conditions inside data centers that portrayals in movies in TV shows always seem to get wrong: how _loud_ and _cold_ they are.
Only colos are cold, though. Private hyperscalers keep theirs nice and hot.
Seriously. Next up from the Post an article about how great and important leaded gasoline is.
Honestly, the Internet as a decentralized global network of computers suggests otherwise. There really shouldn't be any, or very few, datacenters, people should host their own stuff from their own machines, from their own homes.
Unfortunately, culture and technology took a different road.
It might be hard to look beyond status-quo, but it USED to be easy to _SEND_EMAIL_ from your own machine into the world, sure it came with problems, but they could be overcome.
It USED to be easy to host a website on your home computer, simply ask for a static ip, and forward a port or two on your router.
All of this was taken away from us in the holy name of convenience (and corporate greed).
Unfortunately, culture and technology took a different road.
It might be hard to look beyond status-quo, but it USED to be easy to _SEND_EMAIL_ from your own machine into the world, sure it came with problems, but they could be overcome.
It USED to be easy to host a website on your home computer, simply ask for a static ip, and forward a port or two on your router.
All of this was taken away from us in the holy name of convenience (and corporate greed).
It's really an economy of scale thing, and is not that surprising.
The economy of scale is separately at the level of hardware procurement, security, interop, bandwidth, access to electricity, and probably other things.
This isn't a normative claim. Simply that the tendency for these types of things to centralize as complexity and demand increases is natural.
I think it's tough to say that it was taken away, or that there's anybody in particular to blame.
The hardware market moves fast, the capabilities needed evolve fast, etc.
It's still 100% possible to host your own stuff. Many people do, but it's obviously a vast minority. But it's not like there are licensure requirements to host your own website. Some things are disallowed by consumer ISPs, but you can also generally upgrade that allow it.
It's worth noting that when it was simple and unrestricted, it was also vastly more expensive and far fewer people participated. Part of the reason why internet is so cheap now (on a $ / byte basis) is that new offerings were designed that restricted people on the now available cheap plans from doing things that are very expensive.
The economy of scale is separately at the level of hardware procurement, security, interop, bandwidth, access to electricity, and probably other things.
This isn't a normative claim. Simply that the tendency for these types of things to centralize as complexity and demand increases is natural.
I think it's tough to say that it was taken away, or that there's anybody in particular to blame.
The hardware market moves fast, the capabilities needed evolve fast, etc.
It's still 100% possible to host your own stuff. Many people do, but it's obviously a vast minority. But it's not like there are licensure requirements to host your own website. Some things are disallowed by consumer ISPs, but you can also generally upgrade that allow it.
It's worth noting that when it was simple and unrestricted, it was also vastly more expensive and far fewer people participated. Part of the reason why internet is so cheap now (on a $ / byte basis) is that new offerings were designed that restricted people on the now available cheap plans from doing things that are very expensive.
I think I disagree… desktops are pretty inexpensive in the grand scheme of things. I mean most of us could host our stuff on a raspberry pi or something like that. It isn’t like we’d individually need to handle a ton of traffic.
Rather I think the problem is that most server (software) was pretty poorly written, to the point where it requires professional administration and maintainence. Although, this did create a lot of IT jobs. I guess there isn’t anyone, really incentivized to architect things better.
Rather I think the problem is that most server (software) was pretty poorly written, to the point where it requires professional administration and maintainence. Although, this did create a lot of IT jobs. I guess there isn’t anyone, really incentivized to architect things better.
You could run your blog off your machine at home, much like you (hopefully) run your home automation from a machine at home, not a cloud instance.
Serving your collection of videos from home is a bit ore taxing, should anything become popular. And if you produce popular videos, you want to handle subscriptions and comments, accept payments, fend off spam, and weather the occasional DoS attack. Things quickly snowball into a full-time sysop job.
This is to say nothing of businesses that need hundreds of boxes. Putting them all in the office building means you have really hard time moving, e.g. because you grow and need more space, even as we ignore the question of redundant power and internet links in the building.
Datacenters most honestly deserve their place. They make life easier for many people, for many reasons. They are not the only thing that should exist, but very certainly they are not a mistake or aberration.
Serving your collection of videos from home is a bit ore taxing, should anything become popular. And if you produce popular videos, you want to handle subscriptions and comments, accept payments, fend off spam, and weather the occasional DoS attack. Things quickly snowball into a full-time sysop job.
This is to say nothing of businesses that need hundreds of boxes. Putting them all in the office building means you have really hard time moving, e.g. because you grow and need more space, even as we ignore the question of redundant power and internet links in the building.
Datacenters most honestly deserve their place. They make life easier for many people, for many reasons. They are not the only thing that should exist, but very certainly they are not a mistake or aberration.
People don't do this for reliability reasons. All you need is a static IP, a machine at home and a internet connection. It's fine if you don't care about uptime, but people aren't running businesses that way for a reason.
Can you can give an example of what expensive things those cheap plans are restricted from having access to (IPv4 blocks?)
A public IPv4 address for some, general restrictions against using it as a server / hosting any services in the AUP for others.
CGNAT tends to make your life very difficult.
Lots of residential internet plans will block certain ports, even if you do end up with a public ip.
Lots of residential internet plans will block certain ports, even if you do end up with a public ip.
> It's still 100% possible to host your own stuff.
It's absolutely not.
A minority of people have direct access to a network that will let other people reach any service they want to host. You need to carefully select your location to get that. And a lot of people don't even have direct access to a public address.
And that's not even going into the monopolized federated protocols like email, where you can only do anything if big-tech allows.
It's absolutely not.
A minority of people have direct access to a network that will let other people reach any service they want to host. You need to carefully select your location to get that. And a lot of people don't even have direct access to a public address.
And that's not even going into the monopolized federated protocols like email, where you can only do anything if big-tech allows.
>All of this was taken away from us in the holy name of convenience (and corporate greed)
Corporate greed will serve up whatever people will pay for. I (and nearly 100% of people) am willing to pay for the convenience of not running my own email service, so the greedy corporations which sold that solution succeeded while others failed.
Also think of Capex vs Opex. I don't want to drop $4k on an at-home PC to play with new AI toys, but I am willing to spend $20 per month paying for ChatGPT and Claude AI access.
Corporate greed will serve up whatever people will pay for. I (and nearly 100% of people) am willing to pay for the convenience of not running my own email service, so the greedy corporations which sold that solution succeeded while others failed.
Also think of Capex vs Opex. I don't want to drop $4k on an at-home PC to play with new AI toys, but I am willing to spend $20 per month paying for ChatGPT and Claude AI access.
The internet protocols suggest this but the common 1gbps/35Mbps suggest that they want you to use it to consume rather than host.
Distributed hosting is incredibly inefficient.
Distributed hosting is incredibly efficient.
Me browsing a website for my local pizza joint should be a connection from my pc, through a couple of switches/routers, to the PC in their office.
As it stands now I have to route out of town to the nearest datacenter hosting their web platform. Then I need to make more requests out of town to every third party analytic, font, and framework their platform includes. Not to mention the associated energy cost of running all of that code.
Every one of those requests triggers a cascade of dozens of database updates, replications, and algorithm updates across many geographically diverse datacenters.
(Yes I am aware that 'distributed' means many different things depending on the context and is largely a meaningless word)
Me browsing a website for my local pizza joint should be a connection from my pc, through a couple of switches/routers, to the PC in their office.
As it stands now I have to route out of town to the nearest datacenter hosting their web platform. Then I need to make more requests out of town to every third party analytic, font, and framework their platform includes. Not to mention the associated energy cost of running all of that code.
Every one of those requests triggers a cascade of dozens of database updates, replications, and algorithm updates across many geographically diverse datacenters.
(Yes I am aware that 'distributed' means many different things depending on the context and is largely a meaningless word)
Restaurants are a good example of where this breaks down, though, I think.
There are cash-only bars with mechanical registers, and food trucks with no electric hookups, and small third-party kitchens in bars and corner stores that all have websites or at least social media profiles with hours, menu, location, etc.
These businesses have no interest in running a web server. They may not even have internet access onsite beyond employee cellphones.
There are cash-only bars with mechanical registers, and food trucks with no electric hookups, and small third-party kitchens in bars and corner stores that all have websites or at least social media profiles with hours, menu, location, etc.
These businesses have no interest in running a web server. They may not even have internet access onsite beyond employee cellphones.
Exactly right. No one is bemoaning the lack of restauranteurs constructing their own buildings or farming their own food. Why do we want them to host their own website?
You're absolutely right, and yet the romantic in me still thinks there is hope.
I have an excellent example where some friends used to run a wifi mesh in my city. They handed out mesh nodes to small business owners like restaurants and corner shops. These devices were tiny, and only required power since they were mesh nodes.
The point here is that these nodes lasted for years. Sometimes we went around and restarted, or updated firmware, but by and large they worked so well that they were forgotten by the business owner.
This was 10 years ago, today a local web server, or similar device, might work in a similar way.
One idea would be to integrate it into their internet package. They get a router that gives their shop wifi, at the same time it also runs their ordering system.
I think it's possible, anything is possible if you're motivated, but large businesses don't like independence. They prefer that small business and end users are dependent on them.
I have an excellent example where some friends used to run a wifi mesh in my city. They handed out mesh nodes to small business owners like restaurants and corner shops. These devices were tiny, and only required power since they were mesh nodes.
The point here is that these nodes lasted for years. Sometimes we went around and restarted, or updated firmware, but by and large they worked so well that they were forgotten by the business owner.
This was 10 years ago, today a local web server, or similar device, might work in a similar way.
One idea would be to integrate it into their internet package. They get a router that gives their shop wifi, at the same time it also runs their ordering system.
I think it's possible, anything is possible if you're motivated, but large businesses don't like independence. They prefer that small business and end users are dependent on them.
The masses of small businesses yearn to put their local cable monopoly on the critical path of their revenue? This is your belief?
Yeah you're right, it would be better as a separate box next to their router.
The router was just a tempting entrypoint since every pizzeria and falafel joint has one.
The router was just a tempting entrypoint since every pizzeria and falafel joint has one.
Wrong angle.
> Me browsing a website for my local pizza joint should be a connection from my pc, through a couple of switches/routers, to the PC in their office.
I don't think you have a very good picture of the energy costs involved. Half of the energy costs in the total worldwide collection of information systems are in wireless links at the edges of the network (mobile networks and wifi). So if you traverse two wireless links on the way to your pizza joint it is likely that you have caused the worst-case energy usage. An optical link to your nearest cloud datacenter is going to need less energy.
Not to mention the now-always-on nature of the pizza place's PC, which can't be as efficient as a multi-tenant cloud system, has terrible AC/DC conversion losses, spectacularly bad HVAC inefficiency, etc.
I don't think you have a very good picture of the energy costs involved. Half of the energy costs in the total worldwide collection of information systems are in wireless links at the edges of the network (mobile networks and wifi). So if you traverse two wireless links on the way to your pizza joint it is likely that you have caused the worst-case energy usage. An optical link to your nearest cloud datacenter is going to need less energy.
Not to mention the now-always-on nature of the pizza place's PC, which can't be as efficient as a multi-tenant cloud system, has terrible AC/DC conversion losses, spectacularly bad HVAC inefficiency, etc.
Except not every pizza joint can afford their own network engineer, IT Security officer, hardware guy. Small firms that try to roll their own often fail because these things become neglected.
They already franchise in so that the franchiser can perform these central tasks in a more efficient manner.
The franchise looking at the costs of on prem vs hosted vs colo vs cloud will probably find that on prem is the least efficient way they can go about it.
They already franchise in so that the franchiser can perform these central tasks in a more efficient manner.
The franchise looking at the costs of on prem vs hosted vs colo vs cloud will probably find that on prem is the least efficient way they can go about it.
IIRC direct sending of mail wasn't removed in the name of convenience, it was removed in the name of "99% of incoming mail is junk, can we permanently reject the worst offenders?"
Woz was famously quoted as "what is this - biggest datacenter takes all?"
Most of this is people like you and me completely ignoring IPv6 and similar technologies, just because they're not the default
What I would like from the cloud is absolutely positively guaranteed lifetime backups of everything I deem important enough, and heavily encrypted to boot. Everything else cloudish is... extremely optional.
I like to see articles about the hidden systems that make so much of the modern world work, I think it's a positive thing to allow common people to have that perspective if they're interested. However, when I do see stories about datacenters and the like, I just really sort of despair, to be perfectly honest.
I see how little the industry has changed and innovated since the 'modern' datacenter crawled out from the legacy telephone switching centers. You still see the marks (read: scars) left on the old in the new: same sized racks in width, asinine cold and hot aisles shuffling hot air around blown about by fan after fan after fan screaming away endlessly until they throw their bearings. The energy moved about among so many different interfaces, air/water/air/refrigerant/water/air and around and around and back again. It's the same feeling as railroad gauges echoing the ancient wheeled carriage ruts in the hard dirt because we can't be arsed to make anything better.
You'd think that with all of the developments of omg distributed computing that every tech dweeb loves to bandy about that they're experts in that there would be a stampede to, you know, actually make use of some new technology, moving the data and compute closer to the user (and don't talk to me about edge facilities, they're just more of the same), reduce the blast radius of the idiocy of centralization, pare down the eye watering waste of facilities like this. But no, it's still client-server everything, maybe put a copy in another region if you're feeling kinky, but the promise of decentralization and distribution hasn't really materialized. Such a large percentage of people now have phenomenal power they carry with them at all times, but they're just dumbed-down gateways to these huge black holes of resources because the control needs to remain held in the same hands as the capital that continues to build them larger and larger.
I guess what I'm trying to say is that in an ideal world that uses the technology that it develops in a sane way, these places really shouldn't exist anymore, like another comment suggested. The fundamentals are all there, it's just the incentives are so out of whack it's comical. Build bigger and bigger facilities and collect more and more data and fritter away cycle after cycle and maybe you'll increase your year-over-year by a couple dozen basis points. And since the demand is so insatiably fueled by this feedback, nothing will change. You could have minuscule neighborhood nodes scattered around a city that sips resources in comparison that does all of this in concert with the portable supercomputers we all carry, each benefiting a small handful of people at once but the beast must. be. fed.
I see how little the industry has changed and innovated since the 'modern' datacenter crawled out from the legacy telephone switching centers. You still see the marks (read: scars) left on the old in the new: same sized racks in width, asinine cold and hot aisles shuffling hot air around blown about by fan after fan after fan screaming away endlessly until they throw their bearings. The energy moved about among so many different interfaces, air/water/air/refrigerant/water/air and around and around and back again. It's the same feeling as railroad gauges echoing the ancient wheeled carriage ruts in the hard dirt because we can't be arsed to make anything better.
You'd think that with all of the developments of omg distributed computing that every tech dweeb loves to bandy about that they're experts in that there would be a stampede to, you know, actually make use of some new technology, moving the data and compute closer to the user (and don't talk to me about edge facilities, they're just more of the same), reduce the blast radius of the idiocy of centralization, pare down the eye watering waste of facilities like this. But no, it's still client-server everything, maybe put a copy in another region if you're feeling kinky, but the promise of decentralization and distribution hasn't really materialized. Such a large percentage of people now have phenomenal power they carry with them at all times, but they're just dumbed-down gateways to these huge black holes of resources because the control needs to remain held in the same hands as the capital that continues to build them larger and larger.
I guess what I'm trying to say is that in an ideal world that uses the technology that it develops in a sane way, these places really shouldn't exist anymore, like another comment suggested. The fundamentals are all there, it's just the incentives are so out of whack it's comical. Build bigger and bigger facilities and collect more and more data and fritter away cycle after cycle and maybe you'll increase your year-over-year by a couple dozen basis points. And since the demand is so insatiably fueled by this feedback, nothing will change. You could have minuscule neighborhood nodes scattered around a city that sips resources in comparison that does all of this in concert with the portable supercomputers we all carry, each benefiting a small handful of people at once but the beast must. be. fed.
The answer: Beep, Bop, Beep!