Racking Mac Pros(photos.imgix.com)
photos.imgix.com
Racking Mac Pros
http://photos.imgix.com/racking-mac-pros
316 comments
I have two conflicting responses to what I am seeing here ...
First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool.
Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't build a hackintosh 1U with dual cpus and multiple GPUs and not come out big money ahead. Whatever management overhead gets added by playing the hackintosh cat and mouse is certainly less than building new things out of sheet metal.
Let me say one other thing: right around mid 2000 was when certain companies started selling fancy third-party rack chassis gizmos for the Sun e4500, which was the cadillac of datacenter servers at the time. Huge specs on paper, way underpowered for the money they cost ($250k+) and the epitome of Suns brand-value. And there were suddenly new and fancy ways to rack and roll them.
This reminds me a lot of that time, and that time didn't last long...
[1] Our esteemed competitor, tarsnap.
First, this is awesome. Just like I want to live in a world where people are paying picodollars for cloud storage[1], I also want to live in a world where a bunch of mac pro cylinders are racked up in a datacenter. Very cool.
Second, this is complete silliness. I'm not going to go down the rabbithole of flops per dollar, but there is no way that you can't build a hackintosh 1U with dual cpus and multiple GPUs and not come out big money ahead. Whatever management overhead gets added by playing the hackintosh cat and mouse is certainly less than building new things out of sheet metal.
Let me say one other thing: right around mid 2000 was when certain companies started selling fancy third-party rack chassis gizmos for the Sun e4500, which was the cadillac of datacenter servers at the time. Huge specs on paper, way underpowered for the money they cost ($250k+) and the epitome of Suns brand-value. And there were suddenly new and fancy ways to rack and roll them.
This reminds me a lot of that time, and that time didn't last long...
[1] Our esteemed competitor, tarsnap.
Three possible reasons I can think of for doing this over using PCs or Linux servers:
1. Using the same operating system as the developers of the software, plus access to Apple's fantastic imaging libraries.
2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution.
3. The form factor. Even if you could create PCs that are cheaper with the same spec, they'll use more power, possibly require more cooling (Mac Pro has a great cooling architecture) and will take up a lot more space.
I'd be very interested in hearing how they manage updates and provisioning, however. I can't imagine that'd be much fun on OS X but perhaps there's a way of doing it with OS X Server.
1. Using the same operating system as the developers of the software, plus access to Apple's fantastic imaging libraries.
2. The Mac Pro, whilst expensive, is good value for money. The dual graphics cards inside it are not cheap at all. As servers with GPUs are fairly niche, this might actually be a cheaper solution.
3. The form factor. Even if you could create PCs that are cheaper with the same spec, they'll use more power, possibly require more cooling (Mac Pro has a great cooling architecture) and will take up a lot more space.
I'd be very interested in hearing how they manage updates and provisioning, however. I can't imagine that'd be much fun on OS X but perhaps there's a way of doing it with OS X Server.
Given all of the effort spent to use Quartz's graphics operations, I was curious as to how they actually performed. I opened an account and tried out the upsampling, and was a bit disappointed.
http://chen.imgix.net/rose.png?w=560
What other upsamplers look like: https://github.com/haasn/mpvhq-upscalers/blob/master/Rose.md
Looking at the other operations available, I fail to see what is done better by Quartz than just by imagemagick.
http://chen.imgix.net/rose.png?w=560
What other upsamplers look like: https://github.com/haasn/mpvhq-upscalers/blob/master/Rose.md
Looking at the other operations available, I fail to see what is done better by Quartz than just by imagemagick.
Building on OSX seems like it must add a ton of complexity to your workflow, despite getting access to some of Apple's GPU-optimized image code.
Then again, it's often cheaper to throw silicon at problems than people. If you have in-house expertise in Apple's graphics libraries, that might be cheaper than hiring someone who could write the whole thing to run under a lower-cost Linux solution.
Alternatively, OS X might give you automatic access to patent licenses for some of the more expensive image formats.
Have they ever blogged about why they've gone down this path?
Then again, it's often cheaper to throw silicon at problems than people. If you have in-house expertise in Apple's graphics libraries, that might be cheaper than hiring someone who could write the whole thing to run under a lower-cost Linux solution.
Alternatively, OS X might give you automatic access to patent licenses for some of the more expensive image formats.
Have they ever blogged about why they've gone down this path?
It's really kind of mind-boggling that Apple makes and sells the Pro, which can be upgraded to a really nice high performance GPU workstation, but then doesn't sell the same hardware in rack mountable forms for clusterable computing.
I'm sure they've performed some kind of market analysis for this, but there's enough differences between OSX and Linux solutions that for people who use HPC solutions (a growing market) a cleaner path from OSX to HPC would be very helpful.
I'm sure they've performed some kind of market analysis for this, but there's enough differences between OSX and Linux solutions that for people who use HPC solutions (a growing market) a cleaner path from OSX to HPC would be very helpful.
The mechanics of this are pretty neat. But the photography in the article is incredibly distracting. Does every shot have to be at an off-kilter angle? If this is a story about engineering, how about some head-on shots of the engineered thing.
I get that the Mac Pro is a beautiful object, but this isn't about the mac. It's about the rack, and none of these photos let me understand it in one shot.
I get that the Mac Pro is a beautiful object, but this isn't about the mac. It's about the rack, and none of these photos let me understand it in one shot.
Apparently you can fit a round peg in a square hole and achieve high efficiency density while you're at it.
Is there a summary somewhere that explains what makes "OS X’s graphics frameworks" worth going to all this trouble?
This seems really crazy to me. I get it, when you're a startup sometimes you end up with bubblegum and scotchtape solutions like this and sometimes that really makes the most sense on many levels.
But usually you keep that to yourself! To me, this reads sorta like: "Well, it was really hard to find someone who knew how to build a replacement bridge across the creek. We were pressed for time, and Bob didn't know anything about bridges, but luckily, he used to be in the Air Force and we have a bunch of venture capital. ... So we bought a helicopter instead. We only cross a few times a year, so for now we're coming out ahead and it works out for us. Plus the pictures are nice..."
But usually you keep that to yourself! To me, this reads sorta like: "Well, it was really hard to find someone who knew how to build a replacement bridge across the creek. We were pressed for time, and Bob didn't know anything about bridges, but luckily, he used to be in the Air Force and we have a bunch of venture capital. ... So we bought a helicopter instead. We only cross a few times a year, so for now we're coming out ahead and it works out for us. Plus the pictures are nice..."
Wow, learn something new everyday. I thought everyone who did image processing and cared about performance used NVidia cards for the CUDA libraries. I never knew apple [GPU image libraries](https://developer.apple.com/library/mac/documentation/Graphi...) made AMD a competitive choice.
It is much more expensive, though a lot less engineering work, than buying some used Tesla's on ebay: http://www.ebay.com/sch/i.html?_from=R40&_trksid=p2050601.m5...
or even brand new
It is much more expensive, though a lot less engineering work, than buying some used Tesla's on ebay: http://www.ebay.com/sch/i.html?_from=R40&_trksid=p2050601.m5...
or even brand new
I'm really impressed by the quality of engineers on this forum. It's amazing, it seems that just about everyone here knows how to do skuhn's job better than he does!
Some previous discussion here about using OS X, mounting Mac Minis, etc: https://news.ycombinator.com/item?id=8138791
Seems like you could have gotten higher density going vertical instead of horizontal. It would have been 50% taller (6U instead of 4U) but it could have held 100% more Mac Pros.
A Mac Pro is 9.9 inches tall and 6.6 inches in diameter. 9.9 / 1.75 = 5.65 and 6.6 / 1.75 = 3.77 https://www.apple.com/mac-pro/specs/
A Mac Pro is 9.9 inches tall and 6.6 inches in diameter. 9.9 / 1.75 = 5.65 and 6.6 / 1.75 = 3.77 https://www.apple.com/mac-pro/specs/
Would it really have killed Apple to keep on making rack mountable OS X servers? I bought and configured quite a few of them back in the day and was quite fond of them.
I realize it's not the Apple Way™ but considering just how bizarre and niche the current trash-can Mac Pro line is, it hardly seems more niche than that.
I realize it's not the Apple Way™ but considering just how bizarre and niche the current trash-can Mac Pro line is, it hardly seems more niche than that.
At this point, Apple should just license OS X for VMWare installations on non-Apple hardware so we can skip all this foolishness.
It's deep in the comments and I really like the sentence:
x0054: "You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes."
And place them horizontally. And without additional fans!
And surprisingly, if you read skuhn's answers here, for them it all still has sense, financially.
And also surprisingly, Apple says it's OK to use the Mac Pros horizontally:
https://support.apple.com/en-us/HT201379
Fascinating.
x0054: "You are fitting triangular shaped computers, wrapped into round cases, into square shaped boxes."
And place them horizontally. And without additional fans!
And surprisingly, if you read skuhn's answers here, for them it all still has sense, financially.
And also surprisingly, Apple says it's OK to use the Mac Pros horizontally:
https://support.apple.com/en-us/HT201379
Fascinating.
It is hard not to think that a great deal of time an money would have been saved by removing those "Parts of our technology are built using OS X’s graphics frameworks, which offer high quality output and excellent performance."
IF that's what you want, that's a fairly ingenious solution. If you look at https://macstadium.com/mac-pro you'd think you can just build the racks with the Mac Pro opening back and front. But it's 6.6" inches wide so you can only have two, three would be more than 19" (and you can't put 19" of equipment inside a rack). But this way you can squeeze four in the same space and since the cylinder is 9.9" high if you'd need you could squeeze in quite a few external hard disks as well although that would require using some fans to help moving air. You perhaps have 6-8 inches free space, one 3.5" HDD is 5.75 inch high so you could stand it on the shorter 4" edge and put in 6 taking 1" from the 19" and probably have two banks of this to arrive to 3 disks per Mac Pro and leave 4-6 inches still for moving air. It might not be impossible to squeeze in 6 disks per Mac Pro but cooling would need to be very impressive for that.
This is interesting -- they actually manage to get greater density out of this setup than many traditional rack mount systems offer.
And to those questioning "Why would you use such expensive systems when commodity hardware is just as fast at half the price?" I would reply that the Mac Pro isn't all that expensive compared to most rack mount servers. If you're talking about a difference of $2000 per server, even across a full rack you're talking less than $100k depreciated over 5 years.
Though Apple is sorely lacking a datacenter-capable rack mount solution. I've always felt they should just partner with system builders like HP or SuperMicro to build a "supported" OS X (e.g. certified hardware / drivers, management backplane, etc.) configuration for the datacenter market. It's kind of against the Apple way, but if this is a market they remotely care about, channel sales is the way to go.
And to those questioning "Why would you use such expensive systems when commodity hardware is just as fast at half the price?" I would reply that the Mac Pro isn't all that expensive compared to most rack mount servers. If you're talking about a difference of $2000 per server, even across a full rack you're talking less than $100k depreciated over 5 years.
Though Apple is sorely lacking a datacenter-capable rack mount solution. I've always felt they should just partner with system builders like HP or SuperMicro to build a "supported" OS X (e.g. certified hardware / drivers, management backplane, etc.) configuration for the datacenter market. It's kind of against the Apple way, but if this is a market they remotely care about, channel sales is the way to go.
Just last night I was asking why with so many mobile app companies no one is building their server side in Objective C. Wouldn't that have the same personnel advantages as Node.js (supposedly) offers the web world? I haven't looked to see if there is a decent Objective C web framework, but if it's just an API I guess you don't need too much.
I mean I can think of lots of reasons to stick with Rails/whatever (and that's what I'd do), but I'm surprised it is quite so unheard of. You'd get much better performance. Skipping garbage collection with ARC would be awesome. Coding time is still pretty fast, and it's not as unsafe as C/C++.
Just a crazy idea for anyone about to start a mobile app company. :-)
I mean I can think of lots of reasons to stick with Rails/whatever (and that's what I'd do), but I'm surprised it is quite so unheard of. You'd get much better performance. Skipping garbage collection with ARC would be awesome. Coding time is still pretty fast, and it's not as unsafe as C/C++.
Just a crazy idea for anyone about to start a mobile app company. :-)
This seems like a company destined to fail:
1) Massive premium for compute
2) They're at the mercy of Apple, a single completely unpredictable vendor.
3) Apple changes it's form-factors to the latest "design" way to frequently
4) Apple sucks to manage in mass
1) Massive premium for compute
2) They're at the mercy of Apple, a single completely unpredictable vendor.
3) Apple changes it's form-factors to the latest "design" way to frequently
4) Apple sucks to manage in mass
Isnt this ridiculously expensive? Couldn't you achieve the same thing using cheaper PCs?
This reminds me of the scene from the matrix where all those humans are suspended in cords as their energy is being drained off by a dark power.
How do you remote provision OS X? I mean, how do you get to the boot menu to choose network boof? With the crashcart?
Really interesting post; the Mac Mini rack looks insanely cool.
This has myself and a colleague wondering what Apple run in their data centres. Can anybody hazard a guess? Is it Apple hardware with OSX? Is it custom/third-party hardware running *nix? I seem to remember somebody mentioning Azure not too long ago.
This has myself and a colleague wondering what Apple run in their data centres. Can anybody hazard a guess? Is it Apple hardware with OSX? Is it custom/third-party hardware running *nix? I seem to remember somebody mentioning Azure not too long ago.
A beautiful demonstration by people who like macs and don't understand geometry or economics.
Reminds me of a tube amp.
This seems like an odd choice to me.
I think OS X has been the best all-round Desktop OS for many years now, but what does it give you as a server that a linux-based system can't, and that's worth the trouble of custom racks, vendor lock-in and high costs?
In fact, if you're working with OpenGL, OS X can be frustrating since it only ever supports an OpenGL version a few years behind the latest release - IMO one of the platform's biggest drawbacks.
Then again, I've seen some pretty strange errors on server machines doing GPU-heavy work on linux machines with Nvidia cards, and it's probably easier to get support on a standardised SW/HW system such as the Mac Pro...
I think OS X has been the best all-round Desktop OS for many years now, but what does it give you as a server that a linux-based system can't, and that's worth the trouble of custom racks, vendor lock-in and high costs?
In fact, if you're working with OpenGL, OS X can be frustrating since it only ever supports an OpenGL version a few years behind the latest release - IMO one of the platform's biggest drawbacks.
Then again, I've seen some pretty strange errors on server machines doing GPU-heavy work on linux machines with Nvidia cards, and it's probably easier to get support on a standardised SW/HW system such as the Mac Pro...
It would be interesting, once these have been in use for a while, to see some stats on the relative temperatures (+ fan speeds) of each machine within the enclosure.
I can imagine the dynamics of 4 machines scavenging air from a single chamber, with an opening on one end, will result in the machines nearer the warm aisle having to work harder to keep cool...
I also wonder what kind of ducting could be implemented to minimize this effect.
Anyway, a very cool project ending in what looks to be a fantastic end product. I wish I had the chance to work on something like this!
I can imagine the dynamics of 4 machines scavenging air from a single chamber, with an opening on one end, will result in the machines nearer the warm aisle having to work harder to keep cool...
I also wonder what kind of ducting could be implemented to minimize this effect.
Anyway, a very cool project ending in what looks to be a fantastic end product. I wish I had the chance to work on something like this!
What if Apple decides to change the Mac Pro form factor for the next iteration? Then you have to retool and are left with a bunch of incompatible chassis. What if Apple stagnates with hardware upgrades? You'd be stuck running obsolete hardware. What if Apple discontinues the entire Mac Pro line? Not to mention the price premium of Apple hardware itself, then the time and expense incurred to design and fabricate this.
The fact that their software depends on Apple's graphics libraries doesn't seem like a good justification for doing this. What it says is they are willing to throw a ton of money and effort towards (very cool) custom hardware, but are unwilling to hire a person to write optimized OpenGL shaders for Linux, which would work on pretty much any other server they choose to build/buy/lease/cloudify. Certainly there will be other "debt" to overcome, especially if much of your codebase is non-portable Objective-C or GCD, but that has to be weighed against the possibility of your only vendor pulling the rug out from under you. And looking at Apple's history, that is a very real possibility...
Owning your hardware like this makes complete sense if your core business is directly tied to the platform itself, eg an iOS/OSX testing service. But as far as I can tell, imgix does image resizing and filters... their business is the software of image processing, and they're disowning that at the expense of making unrelated parts of the business more complicated. Not a good tradeoff, IMO.