We use open source CF to host > 1000 applications in my company. I have virtually zero complains from an operational point of view and the developers love it.
I wish more people would spend the time to look into it instead of just defaulting to Docker + tooling, just because.
I would love to use a public PaaS, but cost is a bit, shall we say, steep ATM though.
We run around 550 instances of apps, totalling at ~250Gb RAM. So ~450Mb RAM per instance. In the coming moths that will just grow, as more and more of the online applications gets either migrated or written for running in the PaaS.
With Heroku[1], the pricing example stops at 150 dynos, but if the pricing is linear we would spend $19657/month.
With Bluemix[2] $14295/month.
Im sure that with that amount of instances one could negotiate better terms.
Ofc, hypervisors and storage is something you have to factor in the equation. But being a large enterprise we already have heaps of that stuff lying around and being under-utilised.
Also, been a while(~ 1 year) since I did a performance comparison[3] between in house and public, and last time I did, our performance crushed the public offerings, so we would have to run more instances to get the same performance, bringing up the cost.
Another consideration we would have to take into account is the network, our applications are written with ms latency in mind, so to move into a public offering we would have to migrate all our DBs and services around the apps into a public provider near the public offering, which would be insanely expensive, complicated and time consuming. And ofc, rewrite all the applications to live in a world where we dont have <1ms connectivity to everything.
I would say that I currently spend a couple of hours/month actually working on CF fixing bugs, trouble shooting etc. Rest of the time I can work on other stuff, so I would not really factor in my salary to much into the equation.
We get compute and flash storage for "free" from other projects(massive SAP and other back office installations wont go away anytime soon). We get a predictable awesome network across two DCs with <1ms latency for free (massive SAP and other back office installations wont go away anytime soon). We have full control over a free, open source, general purpose PaaS(CF wont go away anytime soon). Im not spending enough time administrating it(really, CF is great in that respect) to factor into the cost calculations. This VS >10k a month is why we don't use a public provider.
I suspect that the situation is similar in a lot of large companies, there is resources, already there for other big projects not being used.
I agree, BOSH is pretty darn cool. Once you have learned the quirks of it.
Currently I think a couple of things that stands in the way of it being more widespread.
* Its tedious to create a new release.
* The documentation is scattered.
* Distributing a release can be a bit tricky. Spiff + deployment templates doesn't really scale that well. For CF we have to pipe to a mod script just because the original templates are not flexible enough, i.e "./generate-manifest infra stub.yml | ./add-extra-releases-and-change-static-logging-levels-and-other-random-stuff > manifest.yml"
* Its sufficiently different even for a seasoned admin that it will take some time to get up and running and understanding how stuff fits together end to end. Currently it feels like the only way to understand BOSH is to understand BOSH.
I wish more people would spend the time to look into it instead of just defaulting to Docker + tooling, just because.