We used to build steel mills near cheap power, now that's where we put servers(danluu.com)
danluu.com
We used to build steel mills near cheap power, now that's where we put servers
http://danluu.com/datacenter-power/
8 comments
> Unfortunately, the baseline assumption isn't correct.
To quibble with that quibble, though the steel mills thing is in the title, I wouldn't call it a "baseline assumption" of the longer discussion in the post: the post talks about how power affects datacenter costs, with one sentence at the end mentioning steel mills.
He links the "steel mills" phrase to a page on a Microsoft datacenter in Quincy, WA. I don't know much about the history of that area, but I did read that another Microsoft datacenter was built in The Dalles, OR, which had very cheap power and historically was home to aluminum manufacturers that needed a lot of power for the electrolytic process involved:
http://www.nytimes.com/2006/06/14/technology/14search.html?e... mentions the datacenter and that aluminum smelters were once in the area
http://en.wikipedia.org/wiki/Aluminium_smelting talks more about the process and how it takes "prodigious amounts of electricity"
To quibble with that quibble, though the steel mills thing is in the title, I wouldn't call it a "baseline assumption" of the longer discussion in the post: the post talks about how power affects datacenter costs, with one sentence at the end mentioning steel mills.
He links the "steel mills" phrase to a page on a Microsoft datacenter in Quincy, WA. I don't know much about the history of that area, but I did read that another Microsoft datacenter was built in The Dalles, OR, which had very cheap power and historically was home to aluminum manufacturers that needed a lot of power for the electrolytic process involved:
http://www.nytimes.com/2006/06/14/technology/14search.html?e... mentions the datacenter and that aluminum smelters were once in the area
http://en.wikipedia.org/wiki/Aluminium_smelting talks more about the process and how it takes "prodigious amounts of electricity"
Quincy, like The Dalles, is located near the Columbia River, and is an hour drive to the Grand Coulee, Chief Joseph, Wanapum, Rock Island, and Rocky Reach hydroelectric dams. Electricity in central Washington is very inexpensive. Since that article was written, Dell, Yahoo, Vantage, and Sabey have all constructed data centers in the Quincy area.
Also note that a very large reason for constructing Rocky Reach Dam on the Columbia was to provide power for an Alcoa Aluminum plant that is still heavily dependent on cheap energy prices. Chelan, Douglas and Grant counties all have nearly the cheapest power rates the the country.
We still build steel mills near cheap sources of transportation, but it used to be that they tended to be near sources of coke and iron ore, such as in Le Creusot in Eastern France. Nowadays the big steel mills get their ore and coke through ships. There are also direct reduction mills that don't use coke and are often located near a source of natural gas.
One example of an instance were steelmaking and data centers meet is Facebook's Luleå operation, built in an iron ore port town: loads of hydroelectric power, good infrastructure, and cold.
One example of an instance were steelmaking and data centers meet is Facebook's Luleå operation, built in an iron ore port town: loads of hydroelectric power, good infrastructure, and cold.
A listen through the old Woody Guthrie Columbia River Collection [1] suggests that powering heavy industry was a key justification for building dams on the Columbia River.
[1]: http://en.wikipedia.org/wiki/The_Columbia_River_Collection
[1]: http://en.wikipedia.org/wiki/The_Columbia_River_Collection
Beyond that, I'd say that low-latency abundant bandwidth availability is probably harder to find and bring into a site than power is these days. Not to mention effective cooling.
Then again, there's some very large banking institutions with data centers here in the Phoenix area, so what do I know.
Then again, there's some very large banking institutions with data centers here in the Phoenix area, so what do I know.
I live just across the river from a recently-retired Ford plant in St Paul. The plant had its own dedicated hydroelectric dam on the Mississippi (before you're shocked, the Mississippi just isn't very big in Minneapolis as rivers go, and it was whitewater there before being developed).
I keep waiting for someone to buy up that location to build a giant data center - Amazon, Google, someone like that. It's a beautiful spot with clean dedicated power, in the heart of a metro area full of enterprise operations talent.
I keep waiting for someone to buy up that location to build a giant data center - Amazon, Google, someone like that. It's a beautiful spot with clean dedicated power, in the heart of a metro area full of enterprise operations talent.
FWIW, the site is blocked in Russia, though the most regular proxy works around it just fine, yet.
I wonder what have the author written to trigger the censorship filter.
The halter of censorship is pressing ever stronger. Пора валить :(
I wonder what have the author written to trigger the censorship filter.
The halter of censorship is pressing ever stronger. Пора валить :(
Solar panel prices have dropped to a point where we are about to enter an era of energy cost deflation. This is an incredible human achievement, and the climate and economic benefits will be enormous.
The US was muddling its way out the 2008 crash repercussions, with incremental improvement in the economy. It wasn't until the oil-choke collar on the economy was removed that we saw a return to the low unemployment levels we are seeing today.
Energy costs are a tax on everything we do, and we are in for a global energy tax cut.
The American Southwest has tremendous solar resources, and could be the cheapest place to run data centers in the states if we could develop:
1. Lower cost electricity storage, compressed air, new batteries, or superconducting magnetic energy storage could do it.
2. Develop economical solar thermal air conditioning systems. If you can generate heat, you can generate cold. http://en.wikipedia.org/wiki/Solar_air_conditioning
The US was muddling its way out the 2008 crash repercussions, with incremental improvement in the economy. It wasn't until the oil-choke collar on the economy was removed that we saw a return to the low unemployment levels we are seeing today.
Energy costs are a tax on everything we do, and we are in for a global energy tax cut.
The American Southwest has tremendous solar resources, and could be the cheapest place to run data centers in the states if we could develop:
1. Lower cost electricity storage, compressed air, new batteries, or superconducting magnetic energy storage could do it.
2. Develop economical solar thermal air conditioning systems. If you can generate heat, you can generate cold. http://en.wikipedia.org/wiki/Solar_air_conditioning
Your economic references are incorrect.
1) Unemployment levels are not low, yet. U6 is about 11%. It was 8% in 2007, and 7% in 2001. The labor force participation rate is at a 40 year low. Full-time employment has seen zero net gain in over a decade, despite a population increase of 30 million people.
2) US GDP growth has seen zero improvement. The GDP growth rate the last five years is lower than it has been in all of US history outside of recessions (the average rate of growth from the 1940s to 2007 was about 3.5% +/-). There has been no acceleration.
Growth rates - 2010: 2.5%; 2011: 1.6%; 2012: 2.3%; 2013: 2.2%; 2014: 2.4%
These growth rates required 0% interest rates, $8 trillion in new public debt, and trillions in Fed stimulus.
This latest quarter required a -0.1% inflation adjuster (only the second time that has happened in 65 years...) to keep the economy from printing another negative GDP growth quarter.
The oil and fossil fuels collar has not been removed from the US economy. Electricity prices are at all-time record highs. And when it comes to oil and gasoline, an extraordinarily small % of vehicles run on electricity; Tesla hasn't boosted GDP growth what-so-ever yet, their scale simply is not big enough.
I see no evidence that an oil collar has been removed, nor that solar is meaningfully boosting GDP growth at this point.
1) Unemployment levels are not low, yet. U6 is about 11%. It was 8% in 2007, and 7% in 2001. The labor force participation rate is at a 40 year low. Full-time employment has seen zero net gain in over a decade, despite a population increase of 30 million people.
2) US GDP growth has seen zero improvement. The GDP growth rate the last five years is lower than it has been in all of US history outside of recessions (the average rate of growth from the 1940s to 2007 was about 3.5% +/-). There has been no acceleration.
Growth rates - 2010: 2.5%; 2011: 1.6%; 2012: 2.3%; 2013: 2.2%; 2014: 2.4%
These growth rates required 0% interest rates, $8 trillion in new public debt, and trillions in Fed stimulus.
This latest quarter required a -0.1% inflation adjuster (only the second time that has happened in 65 years...) to keep the economy from printing another negative GDP growth quarter.
The oil and fossil fuels collar has not been removed from the US economy. Electricity prices are at all-time record highs. And when it comes to oil and gasoline, an extraordinarily small % of vehicles run on electricity; Tesla hasn't boosted GDP growth what-so-ever yet, their scale simply is not big enough.
I see no evidence that an oil collar has been removed, nor that solar is meaningfully boosting GDP growth at this point.
The price of oil went from $100 a barrel to $50 a barrel right before we saw the u3 unemployment level drop. Yes, it hasn't been removed but its grip has lessened.
I think we will see the labor participation rate increase now that we are just starting to see some nascent pressure on wages to increase.
Solar isn't meaningfully boosting gdp now, but I'm following the trend to its logical conclusion.
Lower energy prices do fuel growth, and solar is poised to to lower electricity prices globally. Our future is one of abundance and prosperity.
I think we will see the labor participation rate increase now that we are just starting to see some nascent pressure on wages to increase.
Solar isn't meaningfully boosting gdp now, but I'm following the trend to its logical conclusion.
Lower energy prices do fuel growth, and solar is poised to to lower electricity prices globally. Our future is one of abundance and prosperity.
Where can I source panels that cheap, if I'm not in the US.
Genuine question, I'd really want to know.
Genuine question, I'd really want to know.
My parents in Australia just paid $10k AUD for 5KW fully installed including the inverter and "smart meter".
(the government later reimbursed them $5k of that, though that's not really relevant)
(the government later reimbursed them $5k of that, though that's not really relevant)
China.
Actually that is still a pretty good location for steel mills, as well as other industries, like Bitcoin mining farms (perhaps those fall under datacenters...).
++luu for an interesting read.
One thought on hardware acceleration: it may be kind of happening, piece by piece. AES-NI, carry-less multiply, and CRC32 instructions are kind of like smallish (in gate area) accelerators of operations that go much easier in hardware. AMD talked about a compression accelerator for their (now-delayed) A1100 ARM server chip. These are marketed as performance features, not as power-saving, but if you can do something in fewer clock cycles without a ton of area, you are often saving on power as well.
It's also possible that some savings come through finding places where slower chips will do the job because RAM and/or disk and/or bandwidth is the limiter (storage, caching, maybe some proxies and scale-out DBs now running on faster chips). Intel has server Atoms (Avoton, Rangeley) and there are experiments with ARM servers (Baidu cloud storage tried them out: https://gigaom.com/2013/02/20/first-arm-based-servers-in-pro...). There's lots of work on ARMs and Atoms on the consumer side and some of the benefits may spill over to low-power servers. Don't know how widespread this will be, though; I've been rooting for the concept of servers on low-power chips for years and haven't seen much yet.
You also pay for higher power dissipation with lower density. If you're Amazon or some huge provider, you can be talking about a lot of space in absolute terms.
It's interesting to eyeball a latest-gen Facebook Open Compute server with high-wattage CPUs in it, and how much of its height seems to be due to those heatsinks and fans: http://images.anandtech.com/doci/9138/leopard-mob.jpg
SeaMicro sadly didn't succeed, but it's interesting they thought people wanted density enough to make it a key part of their marketing, and AMD thought enough of the idea to buy them: http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-S...
Of course, companies using and selling blade servers and the popularity of <1U configs (e.g., SuperMicro's twin servers: http://www.supermicro.com/products/nfo/1UTwin.cfm, Microsoft's blade-y variation on the idea: http://www.anandtech.com/show/9138/open-compute-hardware-tri..., even that Facebook Open Compute layout which fits 3 boxes in 2U) demonstrate some interest in density.
One thought on hardware acceleration: it may be kind of happening, piece by piece. AES-NI, carry-less multiply, and CRC32 instructions are kind of like smallish (in gate area) accelerators of operations that go much easier in hardware. AMD talked about a compression accelerator for their (now-delayed) A1100 ARM server chip. These are marketed as performance features, not as power-saving, but if you can do something in fewer clock cycles without a ton of area, you are often saving on power as well.
It's also possible that some savings come through finding places where slower chips will do the job because RAM and/or disk and/or bandwidth is the limiter (storage, caching, maybe some proxies and scale-out DBs now running on faster chips). Intel has server Atoms (Avoton, Rangeley) and there are experiments with ARM servers (Baidu cloud storage tried them out: https://gigaom.com/2013/02/20/first-arm-based-servers-in-pro...). There's lots of work on ARMs and Atoms on the consumer side and some of the benefits may spill over to low-power servers. Don't know how widespread this will be, though; I've been rooting for the concept of servers on low-power chips for years and haven't seen much yet.
You also pay for higher power dissipation with lower density. If you're Amazon or some huge provider, you can be talking about a lot of space in absolute terms.
It's interesting to eyeball a latest-gen Facebook Open Compute server with high-wattage CPUs in it, and how much of its height seems to be due to those heatsinks and fans: http://images.anandtech.com/doci/9138/leopard-mob.jpg
SeaMicro sadly didn't succeed, but it's interesting they thought people wanted density enough to make it a key part of their marketing, and AMD thought enough of the idea to buy them: http://www.brightsideofnews.com/Data/2012_9_11/AMD-Expands-S...
Of course, companies using and selling blade servers and the popularity of <1U configs (e.g., SuperMicro's twin servers: http://www.supermicro.com/products/nfo/1UTwin.cfm, Microsoft's blade-y variation on the idea: http://www.anandtech.com/show/9138/open-compute-hardware-tri..., even that Facebook Open Compute layout which fits 3 boxes in 2U) demonstrate some interest in density.
Being near a river wouldn't hurt for heat dissipation either.
Not great for the wildlife, however.
Modern datacenters are not cooled that way, and some of them don't even have substantial makeup water inputs.
[deleted](1)
Actually, we built steel mills near cheap transportation, not cheap power.
Why? Because the main power source for primary steelmaking is coal (coke, actually), not electricity. And you need cheap transport for the power source (coal), raw materials (iron ore and such), and the finished steel. And cheap transportation, water, helps with getting rid of heat.
Electricity is mostly used in secondary steelmaking where you are taking already existing steel, remelting it, and adjusting its properties.
Now, aluminum production is very much tied to cheap electricity. I seem to remember that Iceland does a tremendous amount of aluminum refining because of the cheap geothermal power.