Your computer room will overheat next weekend(everythingsysadmin.com)
everythingsysadmin.com
Your computer room will overheat next weekend
http://everythingsysadmin.com/2010/05/your-computer-room-will-overhe.html
9 comments
"overhead"?
Do you mean "overheat"? It's not too late to fix it ...
Do you mean "overheat"? It's not too late to fix it ...
We had a cooling failure in the room in our office that had the development servers (production server are in a data center).
It was well over 40C - the noise was incredible, sounded like the racks were trying to take off.
It was well over 40C - the noise was incredible, sounded like the racks were trying to take off.
I've been surprised just how long "amateur" server rooms survive cooling failure. Mostly becuase the densities tend to be lower, but also becuse you just get lucky with the way they're built. At our current office we had a server room total cooling failure (due to copper piping stolen off the back of the building...) and although it got very hot (and very loud) in there, we didn't get any machine failures or thermal shutdowns. But you could feel the heat radiating out of the walls the rooms around and above the server room were noticably hotter. If we'd been an older building or the server room had been built for physical security, the thicker walls would have trapped the heat and it would be a different story.
Not my 'server' room. Granted, its actually my laundry room, so the heat escapes to the rest of the house, even on hot days. You can see graphs here - http://www.digitemp.com
If you are one of the unlucky then you should be setting up temperature monitoring and alerting before you exceed the melting point of your Pb free solder...
If you are one of the unlucky then you should be setting up temperature monitoring and alerting before you exceed the melting point of your Pb free solder...
When our department was buying a moderately large cluster I remember being surprised by the fact that one of the considerations for the server room was that in the event of cooling failure you had to make sure there was enough time for the 1600 cores to shut down before the temperature went totally through the roof. I guess the power density of these packed installations is something not easily imaginable if you don't have experience with them...
This really should be a "duh" for any system administrator. If not, time to find a new job :)
Unless you work in a large organization, the "system adminstrator" often isn't actually a System Administrator
That doesn't make sense. The larger the organization, the more often they have large system administrator teams. In fact, most large organizations have compliance requirements, which often map people with proper skills to proper titles and responsibilities in order to get the correct liability protection.
I can not imagine Google allowing some junior programmer to setup a little closet and run servers in there without contemplating cost, heat/cooling, network allocation, and priviledge management (i.e. what services are running where, why, and who's managing it).
However, the smaller the organization, I can see this happening. 8 man startups without a system administrator... Now that is certainly reality.
I can not imagine Google allowing some junior programmer to setup a little closet and run servers in there without contemplating cost, heat/cooling, network allocation, and priviledge management (i.e. what services are running where, why, and who's managing it).
However, the smaller the organization, I can see this happening. 8 man startups without a system administrator... Now that is certainly reality.
I just now realized what you meant... The guy who doesn't really have talent can slip into a larger organization, since his work scope is often narrowed or siloed.
The zillions of little companies with this kind of problem disagree.
It's truly common sense for anyone that has racked a server to consider heating conditions in a room. If not, they shouldn't be allowed to rack anything!
Not really. In the future there will be a whole crop of system administrators who don't ever deal with hardware and data center issues thanks to cloud computing. There will still be plenty to do and plenty to worry about.
Indeed. It'll be smaller teams of hardware architects and administrators. Just like developers today hardly ever touch hardware.
We used to co-locate our servers with another software/services company that had a good setup in their office. On a cold winter morning our oldest server restarted and wouldn't come back up so I went over to their office. The server room was probably 120 degrees and the server was stuck on a temp warning on boot. Turned out their AC in the attic froze up.
And/Or switch to 32nm. Runs much cooler at load.
I wish AMD would hurry the heck up with their 32nm as Intel is too pricey.
I wish AMD would hurry the heck up with their 32nm as Intel is too pricey.
Air conditioning shuts down on the weekend? THAT'S NOT HOW BUILDING HEAT REGULATION WORKS.
You clearly have not leased space in a building owned and run by accountants.
• 4:00pm? Shut it off. Leave the blowers going at least. (Had negotiate life support systems until 6pm in the next lease.)
• Long weekend? Shut it off.
• Compressor that drives the thermostat system fails? Get a cheap household unit from Harbor Freight. (Thunderously loud, ran nearly 24 hours a day, for a while anyway.)
• One of the two main air fans broken? Don't fix. Air still moves. It's ok, I can put on the hearing protection and work in the server room in the afternoons when my west facing office is uninhabitable.
• The remaining fan breaks on Memorial day weekend. Why‽ It was already running 24x7, this is just cruel timing! Now the building gets way too hot and starts transferring heat through the uninsulated walls into my machine room overloading my cooling capacity.
• 4:00pm? Shut it off. Leave the blowers going at least. (Had negotiate life support systems until 6pm in the next lease.)
• Long weekend? Shut it off.
• Compressor that drives the thermostat system fails? Get a cheap household unit from Harbor Freight. (Thunderously loud, ran nearly 24 hours a day, for a while anyway.)
• One of the two main air fans broken? Don't fix. Air still moves. It's ok, I can put on the hearing protection and work in the server room in the afternoons when my west facing office is uninhabitable.
• The remaining fan breaks on Memorial day weekend. Why‽ It was already running 24x7, this is just cruel timing! Now the building gets way too hot and starts transferring heat through the uninsulated walls into my machine room overloading my cooling capacity.
It has certainly done that in at least one building in Boston I worked in.
That's how it works in my office building. It also kicks off at 7PM on weekdays. It may not be completely off, but you can hear it blowing all day and at 7PM there's a loud THUMP in the vents and no sound after that.
Not a problem, our server room has independent cooling that takes the heat to a unit on the roof. It might have to work a little harder on long weekends but we keep track of our heat budget and this isn't a problem.
Not a problem, our server room has independent cooling that takes the heat to a unit on the roof. It might have to work a little harder on long weekends but we keep track of our heat budget and this isn't a problem.
Eventually the summer came around and one morning the server was down. Fortunately there wasn't anything time-critical on the server, everything was backed up both on and off site and we got it up and running again a couple of hours.
It turned out that all we needed to do was cut a vent in the door and mount a box-fan to it.