Bucket Brigade Hauls Diesel Fuel up 17 Stories to Keep NYC Data Center Online(datacenterknowledge.com)
datacenterknowledge.com
Bucket Brigade Hauls Diesel Fuel up 17 Stories to Keep NYC Data Center Online
http://www.datacenterknowledge.com/archives/2012/10/31/peer-1-mobilizes-diesel-bucket-brigade-at-75-broad/
87 comments
Unfortunately, physics dictates this setup. To get to a top floor generator (which is generally necessary for ventilation), you have to pump from the bottom. For most buildings, that means from the basement, where the fuel tanks are placed for other obvious reasons. (Consider if one started leaking, would you want it dripping down to all of the floors below it?)
These data centers are well designed, but it's impossible to cover every disaster scenario.
It's easy for you to sit here after the fact and snipe at them. If you really think you can do better, go start designing them yourself. If you succeed, you'll do quite well for yourself.
These data centers are well designed, but it's impossible to cover every disaster scenario.
It's easy for you to sit here after the fact and snipe at them. If you really think you can do better, go start designing them yourself. If you succeed, you'll do quite well for yourself.
Fuel pumps can be installed such that they are resistant to flooding. A water-tight seal on the fuel tank, sealed power line from the generator, and other straightforward water-proofing measure would have kept Peer 1's generator working as long as the building's fuel supply held out.
Manhattan is a low-elevation coastal island in a region of the country prone to hurricanes. Flooding is an obvious disaster scenario. It's not like we're asking them to plan for nuclear war, here.
Manhattan is a low-elevation coastal island in a region of the country prone to hurricanes. Flooding is an obvious disaster scenario. It's not like we're asking them to plan for nuclear war, here.
Your one complaint about diesel fuel dripping is not a big deal in the short term.
Diesel fuel is quite safe and hard to ignite. THere are plenty of places/situations where significantly more toxic substances are stored in tanks that are not in basements. Your logic would apply more there (could you imagine if acetone was dripping from the ceiling?). As a practical matter, anything leaking through floors is really bad, but the problem is solvable and not a good reason to keep fuel tanks in a basement.
So what other reasons are fuel tanks placed in basements that outweigh the risk of putting them in the basement (everything except tornadoes would seem to be a con for basements)?
Additionally, "Flooding" would seem to be a fairly typical disaster scenario. Again, what does putting them in the basement prevent against that makes it a good alternative.
Plenty of people are designing better datacenters and using them, there is no need for the parent to go out and do anything. If customers hold internap/et al accountable, the problem should take care of itself.
(of course, they won't, but that's another matter :P)
So what other reasons are fuel tanks placed in basements that outweigh the risk of putting them in the basement (everything except tornadoes would seem to be a con for basements)?
Additionally, "Flooding" would seem to be a fairly typical disaster scenario. Again, what does putting them in the basement prevent against that makes it a good alternative.
Plenty of people are designing better datacenters and using them, there is no need for the parent to go out and do anything. If customers hold internap/et al accountable, the problem should take care of itself.
(of course, they won't, but that's another matter :P)
It's easy to say diesel fuel is "quite safe and hard to ignite." But if the NYC Fire Code says you have to store it in the basement, then guess what? You have to store it in the basement
Then either
1. Stop claiming they are disaster tolerant, because they aren't. Flooding is a common case.
The exact rule i'm aware of is "406.5.3 Storage tanks. Motor fuel storage tanks shall be installed below ground, except as authorized by the rules of the Fire Department"
So get the fire department to authorize it.
2. Datacener folks should lobby new york city. Bloomberg would likely care.
3. Find something other than motor fuel, which is what is restricted :)
If all else fails:
4. Stop building disaster tolerant datacenters in new york. This is why a lot of financial folks who aren't HFT's build in Seacaucus, etc. There are very few people who need to place a DC in new york, and can't afford to be across the river.
(I'm very keenly aware of the bandwidth situation in both places, FWIW)
I'm sure if datacenters start moving across the river, Bloomberg would care.
Building codes and other restrictions are taken into account by the big boys (Google, Apple, Facebook, etc) when they build large datacenters.
1. Stop claiming they are disaster tolerant, because they aren't. Flooding is a common case.
The exact rule i'm aware of is "406.5.3 Storage tanks. Motor fuel storage tanks shall be installed below ground, except as authorized by the rules of the Fire Department"
So get the fire department to authorize it.
2. Datacener folks should lobby new york city. Bloomberg would likely care.
3. Find something other than motor fuel, which is what is restricted :)
If all else fails:
4. Stop building disaster tolerant datacenters in new york. This is why a lot of financial folks who aren't HFT's build in Seacaucus, etc. There are very few people who need to place a DC in new york, and can't afford to be across the river.
(I'm very keenly aware of the bandwidth situation in both places, FWIW)
I'm sure if datacenters start moving across the river, Bloomberg would care.
Building codes and other restrictions are taken into account by the big boys (Google, Apple, Facebook, etc) when they build large datacenters.
Regardless of whether they can be designed better, the data centres were assuring everyone that everything will be fine because they have backup generators, ignoring the probability that flooding would take out those generators.
If I had a website or service hosted at a data centre likely to go offline, I'd be annoyed if I were reassured that everything was taken care of right before the system went offline.
If I had a website or service hosted at a data centre likely to go offline, I'd be annoyed if I were reassured that everything was taken care of right before the system went offline.
There's no reason the fuel tanks and pumps can't be in the basement and still withstand a flood if they're waterproof, which would be relatively easy to achieve. I suppose these systems are typically designed to mitigate blackouts though rather than natural distasters, and floods were not considered as a possibility. As you say, it's easy to be critical after the fact.
How about coming to the reasonable conclusion that a data center in a 17 story tall building is not practical? Most purpose built data centers are only a few stories tall and don't have these type of issues as they are not built on islands close to sea level.
For those companies hosting production services on an island a few feet from sea level without a DR plan, I simply laugh. You deserve our scorn, not our production data.
For those companies hosting production services on an island a few feet from sea level without a DR plan, I simply laugh. You deserve our scorn, not our production data.
As someone who designs these sorts of things for a living, there are a host of problems with having diesel fuel on the room. For one thing, fire codes limit the amount of diesel fuel you can store on the roof (for obvious reasons). For another thing, the trucks that carry in the diesel fuel to refill the tanks are located on the street. One way or another you're going to have to pump fuel from the street up to the roof, which is going to leave the pumps that do so susceptible to flooding.
Would a closed, pressurised air system work as a one-off "push all diesel from this tank to the roof" system?
So you could store:
- tank full of diesel
- some compressed air cylinders
at a low level, and use them in the "everything is submerged" event.
So you could store:
- tank full of diesel
- some compressed air cylinders
at a low level, and use them in the "everything is submerged" event.
500 gal diesel = 1622 kg
17 stories = 46 meters
Energy required = 1621 * 46 * g = 732 kJ
Integrating PV=nRT, W = -nRT*ln(V₂/V₁)
assume you're going to compress the air to 150 psi, or 10 atm, so V₂/V₁ = 10. Assume T=10 degrees C. Solving for n, you need 880 moles of air.
Putting this back through PV=nRT, we get a modest 190 gallons of air (compressed).
So you need a 200 gallon tank with 150 psi storage. Look, here's one on craigslist for $400.
http://eugene.craigslist.org/gms/3306858286.html
17 stories = 46 meters
Energy required = 1621 * 46 * g = 732 kJ
Integrating PV=nRT, W = -nRT*ln(V₂/V₁)
assume you're going to compress the air to 150 psi, or 10 atm, so V₂/V₁ = 10. Assume T=10 degrees C. Solving for n, you need 880 moles of air.
Putting this back through PV=nRT, we get a modest 190 gallons of air (compressed).
So you need a 200 gallon tank with 150 psi storage. Look, here's one on craigslist for $400.
http://eugene.craigslist.org/gms/3306858286.html
Given a 200 gallon tank with 150 psi pressure you would no longer be able to pump the diesel to the roof after 460 gallons of the diesel is pumped, assuming temperature is constant and line friction losses are negligible.
Also, I think you have made an error in your arithmetic.
Also, I think you have made an error in your arithmetic.
One can buy pneumatic submersible pumps. I could see having a compressor on the upper levels of the building drive the pump in the basement.
http://www.atlascopco.us/usus/products/navigationbyproduct/p...
Of course, if the diesel has water in it it's no good anyway. And the submersible pumps are mainly sludge or slurry pumps and (I suspect) not rated to handle fuel.
http://www.atlascopco.us/usus/products/navigationbyproduct/p...
Of course, if the diesel has water in it it's no good anyway. And the submersible pumps are mainly sludge or slurry pumps and (I suspect) not rated to handle fuel.
My expertise is on the electrical side, rather than the plumbing side, but I suspect the amount of compressed air you would need to propel 500 gallons or so of diesel up 17 stories would be impractical.
I would have thought Tropical Storm Allison in 2001 was the teaching moment for the data center business, when all the underground infrastructure of downtown Houston was completely flooded. Apparently planners had not realized that critical backup power systems (or research animals, or symphony archives, or ...) shouldn't be below the level of potential flooding. Houston is 25 miles from the Gulf, so this wasn't a storm surge, just a huge steady rain event. I guess the lesson will be repeated until building management learns.
http://en.wikipedia.org/wiki/Tropical_Storm_Allison
http://en.wikipedia.org/wiki/Tropical_Storm_Allison
It's hard to call the diesel generators "useless" when this company is successfully using them to power its facilities. The bucket brigade is clearly not an optimal solution, but it is a solution that is working so far.
Do you have better suggestions? (Not being a troll - you make a fair point and I'm genuinely interested to hear what you have to say if you would have done it differently)
No, I don't. But I think this shows a lack of understanding on the part of the data centers. Datagram said they'd tested their generators prior to the storm, but apparently they were unaware that the electrical switchgear was located in an area that would readily flood. During a flood, what good is a generator if the electrical equipment can't be used? Would a possible solution be to have the generator output run directly to the datacenter, and to have the ability to isolate it from the building ("mains") power?
Also, see my sibling response to Pyrodogg - it was kind of a joint response to both :)
Shouldn't it be possible to make the fuel pump and it's power supply a closed, water proof system? Such that the fuel tank could be flooded and the pump would still run?
Probably a good idea, but I can't help but think what other weakness that system might turn out to have in the next disaster. I don't know if it's just me, but I was definitely very surprised by the degree of flooding - I would have expected debris, structural damage and power outages, but even as a former New Yorker, I was surprised it all happened.
To reply to mjwalshe ... who from the looks of it is hellbanned:
> Submesible pumps or rig a block and tackle out of a window and haul jerry cans up that way safer thna carying open buckets by hand.
[deleted]
True but a slipping on spilled diesel and falling down a flight of stairs is one danger still think rigging block an tackle would be much more efficient.
[deleted]
Makes me think about http://xkcd.com/705/ .
Here is some information on the setup @ 75 Broad St:
"The 17th and 18th floors of 75 Broad have been reserved for generator farms that can accommodate as many as 40 machines. Big doors will be installed in the facade of both floors so the generators can be rigged into the building.
A 41,000-gallon fuel tank is being installed in the basement, with a separate generator and three redundant pumps to supply the generators on the 17th and 18th floors. Each tenant will own its own generator -- E-Spire already has one installed outside on the setback on the 17th floor -- but the building will sell them fuel."
http://www.nytimes.com/1999/10/10/realestate/commercial-prop...
"The 17th and 18th floors of 75 Broad have been reserved for generator farms that can accommodate as many as 40 machines. Big doors will be installed in the facade of both floors so the generators can be rigged into the building.
A 41,000-gallon fuel tank is being installed in the basement, with a separate generator and three redundant pumps to supply the generators on the 17th and 18th floors. Each tenant will own its own generator -- E-Spire already has one installed outside on the setback on the 17th floor -- but the building will sell them fuel."
http://www.nytimes.com/1999/10/10/realestate/commercial-prop...
Squarespace is one of the companies in that data center. See here for their story (and some photos):
http://blog.squarespace.com/ http://status.squarespace.com/
http://blog.squarespace.com/ http://status.squarespace.com/
Thanks for the link! It had exactly what I was hoping for from the actual article / OP.
This shows great dedication on behalf of the team to provide a temporary solution to a more permanent problem. Well done.
More importantly, though - and not to discredit any of the hard work that's been done - hopefully the companies take a look at why the problem was created in the first place. For instance: why were the generators on the 17th floor? Why were the pumps below ground? Why was the datacenter built in a floodzone in the first place?
This is not unlike a lot of problems we face in software - developers bearing the consequences of poor planning.
More importantly, though - and not to discredit any of the hard work that's been done - hopefully the companies take a look at why the problem was created in the first place. For instance: why were the generators on the 17th floor? Why were the pumps below ground? Why was the datacenter built in a floodzone in the first place?
This is not unlike a lot of problems we face in software - developers bearing the consequences of poor planning.
I use Fastmail.fm for email and they're hosted at NYI, which seems to be fine. I wonder what their facilities are like?
http://www.nyistatus.com/
http://www.nyistatus.com/
Particularly important is that they're just in Zone C, i.e. apocalyptic flooding required. They also kept hefty onsite fuel reserves, i.e. at one point a reported 30 hours before they needed their first delivery, then 5 days....
This site, the/a main one for the Huffington Post (Datagram), they're all in Zone A, when Zone B flooding was considered to be likely :-( http://project.wnyc.org/news-maps/hurricane-zones/hurricane-... ). I'm sure Manhattan Island datacenter space in Zones A and B cost less, but....
(I too use them for email and their siting has always been one of my biggest concerns.)
This site, the/a main one for the Huffington Post (Datagram), they're all in Zone A, when Zone B flooding was considered to be likely :-( http://project.wnyc.org/news-maps/hurricane-zones/hurricane-... ). I'm sure Manhattan Island datacenter space in Zones A and B cost less, but....
(I too use them for email and their siting has always been one of my biggest concerns.)
Why are the backup generators on the 17th floor and not the 3rd floor? Assuming there is a very good reason for that,
Why wasn't there an additional pumping room on the 3rd floor, pre-built, with a legal amount of diesel in reserve, and a additional pumps to take over from the basement pumps when those fail, thus saving your bucket brigade 14 floors of climbing?
Why are you carrying diesel in the open in 5 gallon buckets and not in fuel containers that were purchased years ago?
All in all seems somewhat half-assed.
Why wasn't there an additional pumping room on the 3rd floor, pre-built, with a legal amount of diesel in reserve, and a additional pumps to take over from the basement pumps when those fail, thus saving your bucket brigade 14 floors of climbing?
Why are you carrying diesel in the open in 5 gallon buckets and not in fuel containers that were purchased years ago?
All in all seems somewhat half-assed.
Generators are placed on the top floor to simplify the exhaust path, which must terminate at the roof.
Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel.
Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel.
"Generators are placed on the top floor to simplify the exhaust path, which must terminate at the roof."
Thank you.
"Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel."
The proper design would seem to have two pumping stages. One from basement to 3rd floor, the other from 3rd floor to 17th floor.
If fire codes are such that one can't safely store 24 - 72 hours of fuel above flood level, don't advertise that your data center has reliable emergency power backup.
Thank you.
"Pumps are placed next to the fuel because pumping liquid over any significant vertical distance requires the pump to "push" rather than "pull". The fuel is placed in the basement because nobody wants to sit next to a tank full of diesel."
The proper design would seem to have two pumping stages. One from basement to 3rd floor, the other from 3rd floor to 17th floor.
If fire codes are such that one can't safely store 24 - 72 hours of fuel above flood level, don't advertise that your data center has reliable emergency power backup.
Can you use the elevators? If the generator doesn't have the extra power to run them, offer some customers credit and a mention in the post-mortem if they'll let you shut them down temporarily to power the elevators. Then you can bring fuel up in drums instead of buckets.
edit: Nope. http://news.ycombinator.com/item?id=4723814
:(
edit: Nope. http://news.ycombinator.com/item?id=4723814
:(
The elevator equipment is also in the basement, which is flooded with sea water and diesel fuel. Even if they had power, there's no running them until everything is cleaned up.[1]
[1] http://news.ycombinator.com/item?id=4720894
[1] http://news.ycombinator.com/item?id=4720894
How much does it cost to rent a helicopter in NYC?
The elevators are down for safety reasons, IIRC.
And I doubt anyone who's in that building needs "credit" or "a mention in the post-mortem" more than they need the uptime they've paid for.
And I doubt anyone who's in that building needs "credit" or "a mention in the post-mortem" more than they need the uptime they've paid for.
[deleted]
Yeah, seriously. Hauling drums up 17 flights of stairs is both foolishly risky and inefficient. I suspect there's a little marketing going on here.
Apparently the important bits of the elevator are also underwater.
Ah. That's depressingly poor design. It won't change though :-(
It probably has a hydraulic elevator, which would necessitate having all the mechanical stuff at the bottom.
Hydraulic elevators are only used in buildings with 2-5 (in rare cases up to 8) floors. These are certainly traction elevators but the controls and wiring are still likely located in the basement.
I'm surprised to see so many building experts on HN.
With all of this effort put into keeping the data center running, I've been wondering about a few things.
Was it actually connected to the outside world throughout the storm?
I have a hard time imaging that with the power out in large sections of the city some key router on the line wouldn't have also lost power.
If that's the case, the effort was put in just to keep the computers warm to prevent unplanned shutdown, not to actually provide uninterrupted service to the customer?
I'm not familiar with data center operation. If you're already cut off from the larger network at what point does it make sense to keep the machines running vs. shutting them down?
Or perhaps i'm just mistaken and they were actually connected throughout. In which case I find it amazing that the water knocked out pumps and necessitated other shutdowns but their network wasn't damaged in some way.
Was it actually connected to the outside world throughout the storm?
I have a hard time imaging that with the power out in large sections of the city some key router on the line wouldn't have also lost power.
If that's the case, the effort was put in just to keep the computers warm to prevent unplanned shutdown, not to actually provide uninterrupted service to the customer?
I'm not familiar with data center operation. If you're already cut off from the larger network at what point does it make sense to keep the machines running vs. shutting them down?
Or perhaps i'm just mistaken and they were actually connected throughout. In which case I find it amazing that the water knocked out pumps and necessitated other shutdowns but their network wasn't damaged in some way.
Yes, it's still connected to the outside world, and has been continuously thus far. Example of a site still being served from machines at Peer1: http://blog.squarespace.com/
(I am a Squarespace employee)
(I am a Squarespace employee)
Fiber can be underwater with no problems so it's really just the end points you have to worry about. Honestly, I would be surprised if most networks had much issue with this storm as they generally last until the first time you need to refill the generators. That said available bandwidth is probably significant issue, I had a lot of network issues though the storm, but slow is often a lot better than down.
This story reminds me of the efforts one guy in New Orleans went through to keep his data center running after Katrina. There were some tales of diesel hauling in that blog as well.
http://interdictor.livejournal.com/57475.html http://interdictor.livejournal.com/40720.html
http://en.wikipedia.org/wiki/Interdictor_%28blog%29
http://interdictor.livejournal.com/57475.html http://interdictor.livejournal.com/40720.html
http://en.wikipedia.org/wiki/Interdictor_%28blog%29
Some numbers: a 1 megawatt generator burns 70 gallons / hour. If someone can carry 10 gallons (60 lbs), they need to make 7 trips / hour up 17 stories. I think one soldier could manage it.
Highly optimistic. Some numbers:
Columbia Tower in Seattle, Firefighter Stairclimb event (I think you could agree a firefighter is probably on par with a soldier for fitness) - 63 stories carrying 50ish pounds of gear, average finish time, 48 minutes.
7 trips an hour up and down 17 stories = 119 stories.
Oh, and the firefighters are exhausted, drenched in sweat, require cooling down and up to an hour in rehab for each climb, with legs near collapse, burning like fire...
If I had to do this, I'd be going with the bucket brigade, every time (spoken as someone who has completed that stairclimb event).
Columbia Tower in Seattle, Firefighter Stairclimb event (I think you could agree a firefighter is probably on par with a soldier for fitness) - 63 stories carrying 50ish pounds of gear, average finish time, 48 minutes.
7 trips an hour up and down 17 stories = 119 stories.
Oh, and the firefighters are exhausted, drenched in sweat, require cooling down and up to an hour in rehab for each climb, with legs near collapse, burning like fire...
If I had to do this, I'd be going with the bucket brigade, every time (spoken as someone who has completed that stairclimb event).
Was it worth it? Really?
Compared to the amount of work involved after unscheduled power loss in a data centre yes. Been there done that, and I would defiantly haul buckets of diesel for 12 hours rather than spent the next months sorting out systems that had not restarted cleanly.
I don't think so. At work we set up some servers on the west coast to take over in case our main provider on the east coast went down.
I'd say the chance of one of their people getting hurt isn't really worth anyone's uptime.
Also all the single site prep in the world doesn't help if that one site is taken out completely. Keeping multiple servers in multiple areas is a must if 100% uptime, even during events like this, is key.
I'd say the chance of one of their people getting hurt isn't really worth anyone's uptime.
Also all the single site prep in the world doesn't help if that one site is taken out completely. Keeping multiple servers in multiple areas is a must if 100% uptime, even during events like this, is key.
If you know of some way to swap out servers with 100% uptime, I'd like to hear it. Even Stack Exchange, which had that sort of plan in place, had to go 'static' for about a half hour.
Awesome dedication, but it makes me wonder why geo-redundancy isn't in place for companies the size of these.
The long-term correct solution to this problem is cloud infrastructure with multi-provider failover. If you have a server in California hosted by Amazon and a server in Texas hosted by Rackspace it's unlikely that you'll find yourself hauling diesel fuel up a staircase.
This got mentioned on an irc channel a few hours ago and my response was: "Why don't they build a pulley? They're nerds, they have the skills". Obviously shifting diesel about has some risks involved but a basic pulley system would help save a lot of time.
Actually a pulley might be slower when you think about it, lets say you have 10 guys trudging up the stairs each with 4gal in a bucket, that is 40gallons going up stairs, as opposed to one 4gallon bucket being pulled up, then emptied, then dropped down, then pulled up. So the question is how long the load / unload cycle is relative to the bucket brigade cycle.
When constructing my earthquake preparedness kits I spent some time looking at what happened when folks were in earthquakes both in reasonable infrastructure places (Chile) and less reasonable (Haiti) and non-existant (Turkey). That led to having a 'suture kit' in my day pack because one of the common themes was that there were emergency personnel around who were trying to help but they were often without or short on basic supplies to treat severe lacerations. I have no illusions about being able to suture myself up if I needed it, but I do have hope that I could find someone with the skills to do so. And by having distributed emergency supplies with lots of people, it means that as more people collect the better supplied the resulting group will be.
So back to our flood, a number of buildings include a lift system on the roof for moving heavy things in and out of the building that can't go by elevator. One preparedness solution would be to have a way to utilize that system with a bunch of buckets that would let people do this without carrying up the water directly. If it could be made part of the regular gear that they have on site for doing lifts, that might be a positive thing overall.
When constructing my earthquake preparedness kits I spent some time looking at what happened when folks were in earthquakes both in reasonable infrastructure places (Chile) and less reasonable (Haiti) and non-existant (Turkey). That led to having a 'suture kit' in my day pack because one of the common themes was that there were emergency personnel around who were trying to help but they were often without or short on basic supplies to treat severe lacerations. I have no illusions about being able to suture myself up if I needed it, but I do have hope that I could find someone with the skills to do so. And by having distributed emergency supplies with lots of people, it means that as more people collect the better supplied the resulting group will be.
So back to our flood, a number of buildings include a lift system on the roof for moving heavy things in and out of the building that can't go by elevator. One preparedness solution would be to have a way to utilize that system with a bunch of buckets that would let people do this without carrying up the water directly. If it could be made part of the regular gear that they have on site for doing lifts, that might be a positive thing overall.
A pulley system would save manpower but manpower wasn't the limiting factor last night. It was buckets and throughput.
Pulley system wouldn't work inside. There isn't a gap in the stairwells and it's not a straight shot up. Going over the side of the building would potentially work but there'd need to be a boom to get it off the side of the building and without some sort of power winch it's a matter of hauling one or two buckets up manually and I think that'd be lower throughput than the brigade system where you get 1-2 buckets in the time it takes to walk up two flights of stairs.
Pulley system wouldn't work inside. There isn't a gap in the stairwells and it's not a straight shot up. Going over the side of the building would potentially work but there'd need to be a boom to get it off the side of the building and without some sort of power winch it's a matter of hauling one or two buckets up manually and I think that'd be lower throughput than the brigade system where you get 1-2 buckets in the time it takes to walk up two flights of stairs.
Only if they have a place to rig it and the tools to do it. Neither are guaranteed (and if they don't, they won't be getting them quickly).
Depends how advanced and fancy they wanted to get it, there's probably enough bits of random hardware in a data centre to improvise something.
It'd be more a 'thereifixedit.com' solution than something you'd use on a daily basis, but it could (possibly) work.
It'd be more a 'thereifixedit.com' solution than something you'd use on a daily basis, but it could (possibly) work.
Who keeps 17 stories of rope in a data center? Do you know how much 17 stories of rope weighs?
A data centre with a really comprehensive disaster recovery plan.
Don't necessarily need 17 stories depending on internal layout.
Don't necessarily need 17 stories depending on internal layout.
Building a pulley system would take some significant time and requires finding the necessary tools and resources. The bucket brigade is much simpler and immediately available.
Not nearly as much fun, though.
Not nearly as much fun, though.
Nowhere as much fun, but the brigade probably has more of a sense of camaraderie involved so it might balance.
[deleted]
Any bets as to whether they will still be singing that common 'implementation efficiency doesn't matter, you can always scale horizontally' tune afterwards?
Is that dangerous?
I imagine that spilling diesel down the stairs would be hazardous. At the very least, it's extremely viscous (slippery). In addition, that's probably one of the fire escape routes.
Appropriate username is appropriate.
At the minimum it's against fire code. But I think the chances of an inspection are pretty low at this point.
Not nearly as dangerous as gasoline would be. Diesel's flash point is like 60°C.
[deleted]
Is there an advantage of 17th floor vs. say 3rd?
That's where the generator is.
What dedication!
Amazing!!!
[deleted]
I was just reading about the Fukushima accident, and how most of the EDG's failed because they were water-cooled, and while the EDG's were located out of harms way, the pumps for providing cooling water were located on low ground and were damaged by the tsunami. http://fukushima.ans.org/report/accident-analysis
Yesterday I saw a story about the Datagram data center in NYC having to shut down. From their reports about the damage (http://www.datagram.com):
"As of 5pm on October 29, 2012, Datagram had thoroughly tested its emergency systems at 33 Whitehall, NYC fully staffed and awaiting the storm to hit Manhattan's shores. Once ConEd lost power to Lower Manhattan, Datagram's emergency systems kicked on maintaining power to Datagram's datacenter. Unfortunately, within a couple hours of the storm hitting Manhattan's shores, the building's entire basement, which houses the building's fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby. Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage."
It's pretty obvious that, despite all the disaster planning done in the past, Datagram (and TEPCO, and others) have really neglected to appreciate the potential modes of failure for their backup power systems. In both cases, they misunderstood the threat to critical backup power infrastructure. If your EDG is on the roof but the fuel pumps and electrical switchgear is in the basement, what will happen during a flood? If your EDG's are on high ground but the pumps to cool them are not, what happens during a tsunami?