Charm shutting down because of unfixable kernel panics with Rails on Ubuntu(raw.github.com)
raw.github.com
Charm shutting down because of unfixable kernel panics with Rails on Ubuntu
https://raw.github.com/gist/020171fce9ffca2b77d4/0b194a4bc566e2395833f57f8c11453ddbeb43a5/gistfile1.txt
108 comments
Who gave these guys & gals $100k?
Reading this must make founders striving to get investments furious..
Reading this must make founders striving to get investments furious..
Thomas and Amy speak a lot on never accepting outside investment. So this was probably their own money. ("Bootstrapped," if you want)
Given Amy's modus operandi, I believe it's probably mostly their own money.
I'm pretty sure it's their own money.
Yes, the other gentlemen are correct. This was all our own money. Which we earned from scratch from our other products (all doing very well, thank you). And, perhaps more galling, VCs approach us every month to try to invest (but we're against it).
I'm sure shutting down a perfectly good and potentially very profitable product will make lots of wannabes and aspiring entrepreneurs jealous. Luckily I don't make my life decisions based on what other people would do in my situation.
When you run a real business, with real customers, you quickly discover that what sounds good, isn't, and what looks bad, is often the absolutely right thing to do. Luckily, our early access customers for Charm heard & understood & appreciated my email.
I'm sure shutting down a perfectly good and potentially very profitable product will make lots of wannabes and aspiring entrepreneurs jealous. Luckily I don't make my life decisions based on what other people would do in my situation.
When you run a real business, with real customers, you quickly discover that what sounds good, isn't, and what looks bad, is often the absolutely right thing to do. Luckily, our early access customers for Charm heard & understood & appreciated my email.
I'm sorry I jumped to the wrong conclusion. Not very sorry, but a little bit.
FWIW, it deserve respect to shut down something you built with your own money.
In hindsight, I think you would have gotten an entirely different response had you said "We can't run Charm with the team we currently have, there are technical challenges we can't solve". Understandable and believable. But for all the people dealing with problems like the one you blamed on a regular basis, saying what you did just sounds like:
"We spent or life-savings building this great house! But it keeps running out of toilet-paper, so we burned that s"#¤ down!!"
FWIW, it deserve respect to shut down something you built with your own money.
In hindsight, I think you would have gotten an entirely different response had you said "We can't run Charm with the team we currently have, there are technical challenges we can't solve". Understandable and believable. But for all the people dealing with problems like the one you blamed on a regular basis, saying what you did just sounds like:
"We spent or life-savings building this great house! But it keeps running out of toilet-paper, so we burned that s"#¤ down!!"
When running out of toilet paper causes your house to fall down and people to lose money, your metaphor will actually make sense.
It's always nice to hear from people who are informed on the facts of the situation, who analyze things calmly and rationally and consider all the individuals involved. Thank you for participating.
BTW — you got all the facts wrong. Just in case that wasn't clear. And nobody was asking for your sympathy.
BTW — you got all the facts wrong. Just in case that wasn't clear. And nobody was asking for your sympathy.
It's good to see that you guys are going down with grace and ironically, lots of charm, and not flailing around like angry children when people question your motives and competency.
Good to know you couldn't master the most popular Linux distro though! Hope the VCs can read this.
Good to know you couldn't master the most popular Linux distro though! Hope the VCs can read this.
They seem to be over their limit with Github, pasted below as a mirror:
You probably noticed Charm had some nasty downtime a couple weeks ago.
Service quality is very important to us. If we didn't think we could do better, we wouldn't do it at all.
We've spent very generously on sysadmin services and infrastructure (nearly $100k of investment on sysadmin services/infrastructure alone). We hired the best possible, and we splurged on a redundant, powerful, and expensive server configuration from the beginning.
Now we've discovered that there's some kind of base incompatibility with Ubuntu, which is giving us kernel panics which nobody can track down. Charm has been plagued by mystery technical problems from the beginning, when we had to backport from Rails 3.x to 2.x because of massive performance slowdowns which even Rails Core members couldn't identify.
What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger.
There are a lot of things I'm willing to take risks with, but not with your ability to provide support to customers for your business.
And so it is with a very heavy heart that we will cease operating Charm from Dec 15, for the foreseeable future.
You won't be billed again, and we'll refund your last payments.
We will gladly help you migrate your data out of Charm. Please contact us directly ([email protected]) for help.
Thank you so much for taking a chance on us, and sharing our dream for a superior email interface.
I'm truly sorry to disappoint you.
Best wishes,
Amy
You probably noticed Charm had some nasty downtime a couple weeks ago.
Service quality is very important to us. If we didn't think we could do better, we wouldn't do it at all.
We've spent very generously on sysadmin services and infrastructure (nearly $100k of investment on sysadmin services/infrastructure alone). We hired the best possible, and we splurged on a redundant, powerful, and expensive server configuration from the beginning.
Now we've discovered that there's some kind of base incompatibility with Ubuntu, which is giving us kernel panics which nobody can track down. Charm has been plagued by mystery technical problems from the beginning, when we had to backport from Rails 3.x to 2.x because of massive performance slowdowns which even Rails Core members couldn't identify.
What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger.
There are a lot of things I'm willing to take risks with, but not with your ability to provide support to customers for your business.
And so it is with a very heavy heart that we will cease operating Charm from Dec 15, for the foreseeable future.
You won't be billed again, and we'll refund your last payments.
We will gladly help you migrate your data out of Charm. Please contact us directly ([email protected]) for help.
Thank you so much for taking a chance on us, and sharing our dream for a superior email interface.
I'm truly sorry to disappoint you.
Best wishes,
Amy
Paste the link in another tab.
For a 100k, they could have
1) Bought a ton of Heroku dynos with every single addon enabled. That would have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even 2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and three different kinds of asynchronous processing.
2) Gotten fully managed servers at Rackspace with pro support for pretty much any setup they wanted.
How is this even possible? The same application happens to hit bugs in the Ubuntu kernel and Rails core that haven't been fixed all the way to 3.2.9? And they have 100k to spend but couldn't be bothered to try different kernels and distros? Really? Have they seen the AMI launch screen on EC2?
1) Bought a ton of Heroku dynos with every single addon enabled. That would have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even 2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and three different kinds of asynchronous processing.
2) Gotten fully managed servers at Rackspace with pro support for pretty much any setup they wanted.
How is this even possible? The same application happens to hit bugs in the Ubuntu kernel and Rails core that haven't been fixed all the way to 3.2.9? And they have 100k to spend but couldn't be bothered to try different kernels and distros? Really? Have they seen the AMI launch screen on EC2?
We had Rails Core team members spend lots of time trying to fix our Rails 3.x slowdowns. As you probably know, Thomas is a Rails Core Alumnus and we have lots of personal connections in the Rails community.
Nobody could identify it.
Our sysadmin was a sysadmin for one of the first, biggest, and most reliable Rails app ecosystems. He couldn't trace the Ubuntu issue.
It sounds like you don't have a lot of experience running a business. When you do, you will realize that just throwing money at problems doesn't solve anything.
PS - We have servers at Rackspace. 3 of them. Where do you think a lot of that money went? They're $3k a month. And guess what — they won't manage servers with the configuration we required. Fab.
Nobody could identify it.
Our sysadmin was a sysadmin for one of the first, biggest, and most reliable Rails app ecosystems. He couldn't trace the Ubuntu issue.
It sounds like you don't have a lot of experience running a business. When you do, you will realize that just throwing money at problems doesn't solve anything.
PS - We have servers at Rackspace. 3 of them. Where do you think a lot of that money went? They're $3k a month. And guess what — they won't manage servers with the configuration we required. Fab.
Wow, this is unprofessional.
Lots and lots of people run Rails apps on Ubuntu. Many more run them on non-Ubuntu distros. There is not an epidemic of "welp, Rails magically crashes the kernel, time to pack up and go home", and citing that as the reason to close down after spending over $100k on infrastructure smells really, really funny.
"A poor workman blames his tools."
Lots and lots of people run Rails apps on Ubuntu. Many more run them on non-Ubuntu distros. There is not an epidemic of "welp, Rails magically crashes the kernel, time to pack up and go home", and citing that as the reason to close down after spending over $100k on infrastructure smells really, really funny.
"A poor workman blames his tools."
"A poor workman blames his tools."
. . . but even a master carpenter can't build a house out of rotten wood. Of course, to me, rotten wood is OSX and Windows (why anyone would build servers on those instead of one of the BSDs or Linux, I'll never know; to me, development is also incredibly painful on Windows and OSX), but Ubuntu is not without issues. The rallying cry of "usability first!" is all well and good, until things like taking shortcuts by way of binary drivers causes crashes. Still, one has to wonder why they didn't just try another distro. If you think your tools (or your "lumber") are the problem, try different ones.
. . . but even a master carpenter can't build a house out of rotten wood. Of course, to me, rotten wood is OSX and Windows (why anyone would build servers on those instead of one of the BSDs or Linux, I'll never know; to me, development is also incredibly painful on Windows and OSX), but Ubuntu is not without issues. The rallying cry of "usability first!" is all well and good, until things like taking shortcuts by way of binary drivers causes crashes. Still, one has to wonder why they didn't just try another distro. If you think your tools (or your "lumber") are the problem, try different ones.
Lots of people successfully run Rails on Ubuntu in production. I don't think it's fair to cry "rotten wood" here, unless the Rails ecosystem is flush with amazing systems wizards who can work around crippling flaws in the OS to run their software.
I don't think it's fair to cry "rotten wood" here, unless the Rails ecosystem is flush with amazing systems wizards who can work around crippling flaws in the OS to run their software.
Agreed; however, if they were having problems with Ubuntu (or indeed Linux in general), why didn't they try something else? That's the quick and easy way to see if it's really the platform or your application that's the problem (quick tip from my own userland developer experience: it's almost always not the platform; the times I've had Linux crash on me I could always trace to one of three things: 1) flaky hardware, 2) binary drivers, or 3) I was mucking about in kernel space ;)
Agreed; however, if they were having problems with Ubuntu (or indeed Linux in general), why didn't they try something else? That's the quick and easy way to see if it's really the platform or your application that's the problem (quick tip from my own userland developer experience: it's almost always not the platform; the times I've had Linux crash on me I could always trace to one of three things: 1) flaky hardware, 2) binary drivers, or 3) I was mucking about in kernel space ;)
What really blows my mind is that they did a regressive rewrite to Rails 2 rather than learn to use profiling tools.
The whole situation just makes my brain stare blankly and wonder what on earth was happening over there.
The whole situation just makes my brain stare blankly and wonder what on earth was happening over there.
Yes, because it's not like Thomas (my partner/husband) is a performance expert or anything. And it's not like 3-4 Rails Core team members who tried very hard to help us have ever heard of profiling tools.
My mind just exploded, too, due to your willful inability to read the details.
My mind just exploded, too, due to your willful inability to read the details.
If rails causes a kernel panic then this is really a bug in the kernel. No user land app should ever do this. Why not try to scale on *BSD or Windows all of whom have different kernel and scale just fine?
$100,000 and no one said "hey what about running it on <insert other os>"?
$100,000 and no one said "hey what about running it on <insert other os>"?
Or just changing kernel version?
It's quite fascinating really, especially in 2012 to give up because of software failure.
It's quite fascinating really, especially in 2012 to give up because of software failure.
If you read the email I wrote, it's clear that we're not "giving up because of software failure." Look at it again and you'll see it's a different reason altogether.
I'm as bewildered as all of you. But let's give them the benefit of doubt, he probably just forgot to mention that they tried other OSes. I mean, who wouldn't?
Did they try other kernel versions? Other Ruby VMs? This is a terribly weak excuse. It's like quitting your job because you can't get the printer driver working.
[deleted]
This is normal for Ubuntu kernels from experience. They are shitty. Various server releases have given us hell. They're fudging them somehow.
We moved to debian after numerous problems and have had no issues since.
We moved to debian after numerous problems and have had no issues since.
Yeah, when I see people running Ubuntu for their production servers I always assume it's developers with minimal system admin experience who just want to use the same distro for their server as their desktop. I've got nothing against Ubuntu, and have even used it to convert a few windows users over, but if you're going to run a production server then install Debian on it.
I have been using Debian since 1996, and became a package maintainer in 1997, a position I held for nearly a decade until I began to not have enough time for it. While continuing to think that Debian is a first-rate effort, I got tired of certain aspects of it, and moved to Ubuntu, lock stock and barrel. I actually like to have my desktop system marching in sync with my server, so that the packages are pretty much exactly the same. And you know what? Ubuntu works pretty well on the server, and comes out with nice regular releases, allowing me to decide, and gauge, time-wise, when to upgrade. Certainly, like everything else, it has its defects, but by and large, I've had good luck with it, as both a desktop and server distribution.
Well it was our operations guys who refused anything but Ubuntu because "they had installed it at home". We now no longer have operations guys and the development team own the infrastructure :)
Do you have links to bugreports to your problems by any chance? Or more detail on what problems you ran into so I could have the Ubuntu kernel team check them out?
They were submitted to launchpad over a year ago. No action was taken by canonical despite extensive debugging information being provided.
I don't have the report ids any more.
I don't have the report ids any more.
[deleted]
[deleted]
Thanks, I'll keep that in mind for the future.
I don't even know what we're running Freckle on, because it's an app that's had zero problems. Alas, Charm is a troubled baby.
I don't even know what we're running Freckle on, because it's an app that's had zero problems. Alas, Charm is a troubled baby.
Exactly. There's loads of Unixen they could have tried. There's also tons of people running Rails. Though I'm not a big fan of Rails I can't deny there's a lot of it running successfully. There's got to be more to this story!
I'm pretty sure a lot of people are running Linux and the world's largest websites...
Just saying.
Just saying.
My business runs on Ubuntu/Rails just fine.
Maybe if they were a bit more familiar with Linux they would have done better, or at least have saved some money figuring out what went wrong:
https://twitter.com/amyhoy/statuses/189072763590418432
Maybe if they were a bit more familiar with Linux they would have done better, or at least have saved some money figuring out what went wrong:
https://twitter.com/amyhoy/statuses/189072763590418432
That link was really insightful. The fact is that there are tens of thousands of servers running Rails on Ubuntu without any issues.
Sounds to me they would rather blame Ubuntu than their inability to fix the issue.
Later edit: to elaborate that last point. I'm not saying Ubuntu is not at fault, but rather that problem solving goes with the territory. Given the wide array of alternatives, this was not the sort of thing you could openly complain about.
Sounds to me they would rather blame Ubuntu than their inability to fix the issue.
Later edit: to elaborate that last point. I'm not saying Ubuntu is not at fault, but rather that problem solving goes with the territory. Given the wide array of alternatives, this was not the sort of thing you could openly complain about.
Wow. Hmm, how can I put this politely? As a desktop Linux user, I don't feel particularly left out in being ineligible for the hiring pool for a company that wouldn't let me use Linux. Maybe they scared away all the competent Linux admins with their negativity towards Linux?
Baseless speculation is one thing, when the original post actually contains information which contradicts you, it's just a sign of laziness.
What do you mean baseless? There were two provided links showing your distaste for Linux. One could be a fluke (had a bad day), but two shows a pattern. This, added with the mention of using Ubuntu (and not trying another distro) shows that Linux is not your forte.
I, for one, would not like to work for you. I'm not trying to be offensive; in fact, I respect the decision to cut bait when you realized you were over your head. I just don't think I, as a Linux user, would fit in your organization. Many of the good sysadmins also have this same viewpoint, so the above certainly is well founded.
I, for one, would not like to work for you. I'm not trying to be offensive; in fact, I respect the decision to cut bait when you realized you were over your head. I just don't think I, as a Linux user, would fit in your organization. Many of the good sysadmins also have this same viewpoint, so the above certainly is well founded.
Does this make sense? They don't mention trying to get support from Engine Yard on hosting their app on Rails without kernel panics, or trying a managed solution such as Heroku, or trying other OSs other than Ubuntu such as CentOS or something from Joyent. Just trying with the best possible syadmin they could hire and "Rails Core members".
Anyway, kudos for not submitting their users to a service they're not confident they would be able to provide with adequate quality and better luck next time!
Anyway, kudos for not submitting their users to a service they're not confident they would be able to provide with adequate quality and better luck next time!
or trying other OSs other than Ubuntu such as CentOS or something from Joyent.
Or even just Debian. I ran into issues (years ago) with Ubuntu stability due to closed source drivers on a laptop; sure, I have the skills to debug that sort of thing, but why waste my time? I had been running Debian just fine on desktops and servers for years, and after switching the laptop to it, the lockups stopped.
I'm more than willing to admit there may be real technical issues they ran into, but it sounds to me like they gave up too easy, and we may never know what the problems are. Would've been nice to at least have seen a bug report (honestly don't know if they submitted any, but I'm willing to bet they didn't).
Or even just Debian. I ran into issues (years ago) with Ubuntu stability due to closed source drivers on a laptop; sure, I have the skills to debug that sort of thing, but why waste my time? I had been running Debian just fine on desktops and servers for years, and after switching the laptop to it, the lockups stopped.
I'm more than willing to admit there may be real technical issues they ran into, but it sounds to me like they gave up too easy, and we may never know what the problems are. Would've been nice to at least have seen a bug report (honestly don't know if they submitted any, but I'm willing to bet they didn't).
Or Arch Linux. Arch packages are pretty vanilla, so it would be a pretty good way to reduce the attack surface.
Also, with Arch, you get less by default, so you'll end up with a cleaner system. I didn't realize how much stuff I didn't need on a default install of Fedora until I set up an Arch system.
Also, with Arch, you get less by default, so you'll end up with a cleaner system. I didn't realize how much stuff I didn't need on a default install of Fedora until I set up an Arch system.
I'm curious what kind of "massive performance slowdowns" they encountered with Rails 3 that could "not be identified".
Me too, because it sounds an awful lot like someone doesn't know how to use ruby-prof.
You realize that in your example, "somebody" would be several Rails Core team members, right?
It's easy to armchair quarterback… especially if you're not hampered by actually reading the very brief bit of content you're speculating about.
It's easy to armchair quarterback… especially if you're not hampered by actually reading the very brief bit of content you're speculating about.
Yes, it's very easy to armchair quarterback. But I also have experience actually doing what I'm talking about - for example, I've contributed patches back to Haml and MongoMapper that have improved their respective speeds by ~25% and ~40% on average respectively. I've profiled and optimized multiple production apps to eliminate these kinds of issues, written tools to aid in memory profiling, tracked down and fixed memory leaks in various gems, and have yet to run into a mysterious "unsolvable and unexplainable" performance problem. If you can't explain it, that means you just don't know where to look.
Software is deterministic. If you can't explain why it's doing something, then it's not leprechauns fiddling with your bits - it's something that you just don't understand yet, and will have to dig a bit to find it. I'm not speaking from armchair quarterbacking here - I'm speaking from experience.
I am exceptionally dubious that you were encountering problems that were just not solvable, let alone explainable. Doing the handwavey "nobody can explain it" has a particuarly bad odor to it.
If you want to do a full writeup on the problems you encountered and what you tried to do to solve them and why you eventually gave up on them, I will be first in line to read it.
Software is deterministic. If you can't explain why it's doing something, then it's not leprechauns fiddling with your bits - it's something that you just don't understand yet, and will have to dig a bit to find it. I'm not speaking from armchair quarterbacking here - I'm speaking from experience.
I am exceptionally dubious that you were encountering problems that were just not solvable, let alone explainable. Doing the handwavey "nobody can explain it" has a particuarly bad odor to it.
If you want to do a full writeup on the problems you encountered and what you tried to do to solve them and why you eventually gave up on them, I will be first in line to read it.
I never said there were leprechauns, nor unsolvable. What I said was: Some of the absolute best & most knowledgeable people couldn't find them.
We did have 3-4 of the Rails Core team members investigating our Rails 3.x problems and none of them could figure out what it was. The problems were so terrible it made it impossible to develop with Rails 3.x any more. Obviously Rails Core members were both A) friends, and B) highly motivated for us to keep using Rails 3.x.
The problem with trusting your abilities is that you can't imagine a scenario where they will let you down. And yet, many such scenarios exist.
We did have 3-4 of the Rails Core team members investigating our Rails 3.x problems and none of them could figure out what it was. The problems were so terrible it made it impossible to develop with Rails 3.x any more. Obviously Rails Core members were both A) friends, and B) highly motivated for us to keep using Rails 3.x.
The problem with trusting your abilities is that you can't imagine a scenario where they will let you down. And yet, many such scenarios exist.
You obviously know more about this situation than I do, so I'm not going to be able to win any rational argument unless you choose to disclose more details.
I get your suggestion of hubris here, but again, I'm speaking from experience. I've done the solo technical founder thing, and that came with a whole host of problems that seemed unsolvable and which I didn't have anyone to appeal to for answers. Those are the problems that taught me most of what I know - most importantly that when you run into something that you don't understand, it's an opportunity to learn about it and solve it, rather than to just give up. In the Rails world, where the entire software stack is open-source down to the kernel it runs on, there is literally nothing to get in the way of understanding what's happening at any point of the application's execution.
I'll reiterate that if you want to do a writeup on why your circumstances were unique, and the problems you faced which were so crippling that they forced a rewrite, I'd love to read it. The suggestion that Rails 3 has landmines so critical in it that it necessitated a full product rewrite in Rails 2 is exceptionally weighty and should not be asserted without a very specific cataloging of what those problems are. That particular assertion - "Rails 3 has problems that cost us $XX,XXX and the Rails Core team doesn't have a clue what they are" - is a big one.
I get your suggestion of hubris here, but again, I'm speaking from experience. I've done the solo technical founder thing, and that came with a whole host of problems that seemed unsolvable and which I didn't have anyone to appeal to for answers. Those are the problems that taught me most of what I know - most importantly that when you run into something that you don't understand, it's an opportunity to learn about it and solve it, rather than to just give up. In the Rails world, where the entire software stack is open-source down to the kernel it runs on, there is literally nothing to get in the way of understanding what's happening at any point of the application's execution.
I'll reiterate that if you want to do a writeup on why your circumstances were unique, and the problems you faced which were so crippling that they forced a rewrite, I'd love to read it. The suggestion that Rails 3 has landmines so critical in it that it necessitated a full product rewrite in Rails 2 is exceptionally weighty and should not be asserted without a very specific cataloging of what those problems are. That particular assertion - "Rails 3 has problems that cost us $XX,XXX and the Rails Core team doesn't have a clue what they are" - is a big one.
If you'd like to prove your "experience" -- an exceptionally weighty assertion, given I don't know you from Adam, and you know nothing about my product -- I'd be happy to read that writeup.
[deleted]
What on earth?
Blaming this on Linux or Rails is ridiculous. And as for 'mystery technical problems'...
I think the reality is more like they got in over their head, decided it was all a bit too difficult, and gave up.
Blaming this on Linux or Rails is ridiculous. And as for 'mystery technical problems'...
I think the reality is more like they got in over their head, decided it was all a bit too difficult, and gave up.
We run a lot of webservers, our own hardware, on Ubuntu and we do not have this issue at all. Rails doesn't cause Kernel panics. It sounds like they have a bad install or honestly, just don't know what their are doing.
Probably a bad idea to advertise you can't keep a simple webserver up. Probably won't get funded ever.
Probably a bad idea to advertise you can't keep a simple webserver up. Probably won't get funded ever.
Exactly. I'd sooner attribute kernel panics to faulty hardware or a corrupt install than a problem with Rails. (Sure, running a web-server might _exacerbate_ the problem, simply by exercising the hardware -- but it's unfair to attribute that to a problem with Rails/Ubuntu/"mystery.")
The fact that they could not achieve break-even/speed-gains on Rails 3 tells me the application was likely a less-than-stellar codebase to begin with.
I'd agree, if you can't get a Rails application running after investing $100,000 [minimum] in it, it's probably wise not to advertise that fact.
The fact that they could not achieve break-even/speed-gains on Rails 3 tells me the application was likely a less-than-stellar codebase to begin with.
I'd agree, if you can't get a Rails application running after investing $100,000 [minimum] in it, it's probably wise not to advertise that fact.
>Probably won't get funded ever.
Sure. But it also means a lot of people who are looking at her future projects will pause and think twice.
Think twice, like what? "Gee, there's a woman with integrity who shuts down a service while it still works 99% of the time, and refunds my money, instead of growing it like a nutjob until it has weekly downtime like oh I don't know Desk.com"?
I do so hope you're right.
I do so hope you're right.
A service with a weekly downtime is still more useful than no service at all.
Except 37Signals took VC money from Jeff Bezos.
Not in the sense that most HN readers view VC money, as a necessity to launch or continue operations. 37signals was 2 and a half years into existence with multiple products and a few hundred thousand paying users. They didn't give up control and don't have to sweat out how to exit so their investor gets an acceptable ROI. 37signals would have been able to continue operations without the investment.
The investment looks like a way to formalize a long term relationship both sides desired. It doesn't resemble anything like 95% of VC deals that are associated with SV or the types of starts ups associated with HN's readership.
http://37signals.com/svn/archives2/bezos_expeditions_invests...
The investment looks like a way to formalize a long term relationship both sides desired. It doesn't resemble anything like 95% of VC deals that are associated with SV or the types of starts ups associated with HN's readership.
http://37signals.com/svn/archives2/bezos_expeditions_invests...
>Probably won't get funded ever.
that's a product from Amy Hoy[1]. I doubt they'll ever would want to get funded.
http://unicornfree.com/about/
http://unicornfree.com/about/
If you ask me, it's bad to advertise that you publicly speculate, with resounding authority, on things you know absolutely nothing about.
Haha, won't get funded ever? I have VCs knocking on my inbox doors on a monthly basis. We don't take funding. We have, however, brought in >$1 million in revenue on products since we started doing products, including http://letsfreckle.com.
The sysadmin I mentioned is one of the most respected experts in the community. He wasn't cheap. He had a fantastic legacy of very good, very public work. And yet, here we are.
When you run a real business, you will be faced with tough decisions. Hopefully you will be able to make wise choices as well.
Haha, won't get funded ever? I have VCs knocking on my inbox doors on a monthly basis. We don't take funding. We have, however, brought in >$1 million in revenue on products since we started doing products, including http://letsfreckle.com.
The sysadmin I mentioned is one of the most respected experts in the community. He wasn't cheap. He had a fantastic legacy of very good, very public work. And yet, here we are.
When you run a real business, you will be faced with tough decisions. Hopefully you will be able to make wise choices as well.
Let me see
1. Open Source OS
2. Open Source web server
3. Open Source programming language
4. Open Source web framework
5. Some (relatively) simple customer support ticketing app [not likely to be a system pounding Gorilla]
And they claim that they've not been able to root-cause a Kernel panic and a framework slowdown?
And they've spent $100K on that with no results to show for it?
That sounds really strange to me.
[I've debugged Linux kernel panics on custom stacks, with no disk/logs, only console, 32MB of RAM, PPC cross-compiled - and never run into a dead-end like this]
1. Open Source OS
2. Open Source web server
3. Open Source programming language
4. Open Source web framework
5. Some (relatively) simple customer support ticketing app [not likely to be a system pounding Gorilla]
And they claim that they've not been able to root-cause a Kernel panic and a framework slowdown?
And they've spent $100K on that with no results to show for it?
That sounds really strange to me.
[I've debugged Linux kernel panics on custom stacks, with no disk/logs, only console, 32MB of RAM, PPC cross-compiled - and never run into a dead-end like this]
"When Qurve got involved with the Charm project the site was being alpha tested by a handful of early adopters, but a months away from being ready for public consumption. Qurve was brought on in a Ruby on Rails development role to help fix bugs, complete the necessary functionality and ensure the necessary code quality for a public launch.
Over the next few months our role expanded to include a complete rewrite of the user interface, a reimplementation of the credit card billing processor and a few more major changes."
http://www.qurve.com/clients/charm/
Over the next few months our role expanded to include a complete rewrite of the user interface, a reimplementation of the credit card billing processor and a few more major changes."
http://www.qurve.com/clients/charm/
Yes, Daniel helped us backport to Rails 2.x and make a lot of improvements while we were at it, after everybody had given up on finding our Rails 3.x performance problems.
This seems like a bit of a cop out?
Why not try a new kernel or separate stack like Fedora for example?
This is a bogus excuse. There must be some other reason they are shutting down as this appears to be a rectifiable problem.
What was Charm? What did it do?
100k for "infrastructure alone" seems like a massive investment for a startup. How far along were they?
100k for "infrastructure alone" seems like a massive investment for a startup. How far along were they?
It was a customer support / issue tracking service.
This type of service has been written many times before without kernel panics.
I didn't get to see Charm before they shut down, so I have no idea what they were doing.
Could anyone speak towards how their experience was unique enough to cause kernel panics?
I didn't get to see Charm before they shut down, so I have no idea what they were doing.
Could anyone speak towards how their experience was unique enough to cause kernel panics?
Canonical provide commercial support for Ubuntu, don't they? For "$100k of investment on sysadmin services/infrastructure alone", did they ask Canonical for support?
Actually they spent like $20k in what I assume was the salary of their sysadmin:
https://twitter.com/amyhoy/status/271128780058267649
The rest was infrastructure costs ($40k) and porting from Rails 3 to 2.
https://twitter.com/amyhoy/status/271128780058267649
The rest was infrastructure costs ($40k) and porting from Rails 3 to 2.
Totally. I know people who use Ubuntu Server only to be able to use paid commercial support in case of urging/unsolvable issues.
(instead of Debian I mean)
Any links to bug reports or mailing list discussions where Charm reported their problems?
Googling for "rails charmhq" doesn't reveal much, just that one of the Rails contractors was a company called Queve: http://www.qurve.com/clients/charm/
Googling for "rails charmhq" doesn't reveal much, just that one of the Rails contractors was a company called Queve: http://www.qurve.com/clients/charm/
Yes, kernel bugs exists
You can open a bug with distros and try to work around it. It is usually doable
"unfixable kernel panics" don't seem something a person familiar with Linux would say. Also, there are several distros, kernel versions, and Ubuntu isn't my first choice for a server.
And the gist is over the rate already.
You can open a bug with distros and try to work around it. It is usually doable
"unfixable kernel panics" don't seem something a person familiar with Linux would say. Also, there are several distros, kernel versions, and Ubuntu isn't my first choice for a server.
And the gist is over the rate already.
If you trust a random gist link as little as I do, here's a more reputable confirmation:
https://twitter.com/amyhoy/status/271012712253911041
https://twitter.com/amyhoy/status/271012712253911041
Sounds a bit strange.
I'd love to see an Ask HN with _proper_ technical details. Who knows, maybe they'll get better ways to solve this rather than close shop.
I'd love to see an Ask HN with _proper_ technical details. Who knows, maybe they'll get better ways to solve this rather than close shop.
Does anyone have any links to information on charm, or demos? All I can find is their website, which has no information: http://charmhq.com/
Found this so far: http://charmhq.com/Charm_Bootcamp.pdf
Found this so far: http://charmhq.com/Charm_Bootcamp.pdf
Does rails not run on Centos, Debian, or SuSE?
I can _personally_ attest that it runs on SuSE (SLES), Debian, Arch, Fedora, [and Ubuntu].
I have no idea what this team was trying to do, but I imagine there _had_ to be more than what they wrote in this gist. That, or they simply ran out of money to troubleshoot the issue further.
There are also many different ways you can run Rails apps. They could've tried to run them on varoius Rack compatible application servers (Passenger on Apache or Nginx, Thin or Unicorn, et al.) -- If they migrated backwards to Rails 2, they could've used an [obsolete] version of Passenger with Ruby 1.8.7-Enterprise, which had many speed and stability improvements. (Assuming their app wasn't 1.9.x compatible.)
tl;dr: There has to be more to the story than this, they had so many options. For a $100,000 investment in administration, I'd have trouble believing they _didn't_ explore other options.
I have no idea what this team was trying to do, but I imagine there _had_ to be more than what they wrote in this gist. That, or they simply ran out of money to troubleshoot the issue further.
There are also many different ways you can run Rails apps. They could've tried to run them on varoius Rack compatible application servers (Passenger on Apache or Nginx, Thin or Unicorn, et al.) -- If they migrated backwards to Rails 2, they could've used an [obsolete] version of Passenger with Ruby 1.8.7-Enterprise, which had many speed and stability improvements. (Assuming their app wasn't 1.9.x compatible.)
tl;dr: There has to be more to the story than this, they had so many options. For a $100,000 investment in administration, I'd have trouble believing they _didn't_ explore other options.
Or BSD, or OS X or if your business depended on it, maybe even Windows!
.. FreeBSD, Solaris, NetBSD, Windows ..
I wonder how many users Charm had and I wonder how many would have been needed to make porting the code from Rails to something like Java worth it.
The reason that everyone is so incredulous at the shut down is because so few details were given about what they tried. I'd be really interested to know what they tried e.g. different kernels, different OS's.
The message makes it sound like it's all over, but this alludes to a reboot in 2013:
https://twitter.com/amyhoy/status/271279292741201920
https://twitter.com/amyhoy/status/271279292741201920
charmhq.com's current landing page does not render correctly in Firefix. The Charm logo is positioned off screen in Firefox, but centered on the screen in Chrome.
The Firefox layout issue is because Firefox's HTML "quirks mode" is less quirky than WebKit's. With a proper DOCTYPE to disable quirks mode, Chrome will render the page like Firefox.
Anyone have a link to their website or anything? Talk about a horrible name for a startup, simply cannot even find it.
You can find (previous) references to it on Amy Hoy's blog unicornfree.com.
What I find interesting about the reaction to this email, from you bench spectators, is that you completely missed the point of the email.
The email isn't "Wah we hit a server problem, bye bye."
The point is:
"What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger."
If you've never run a serious product, or a real business, or tried to hire for technical positions, I can understand why you'd zero in on the "facts" about the technical situation and ignore all the "foofy personal window dressing," and write things like "I call BS! It's a smoke screen!" or "Why didn't they just try BSD?"
And yet I addressed the actual problem in very clear terms in a paragraph you can't possibly miss.
Next time somebody makes a hard business decision and you hear about it on HN and come out, irony guns blazing, may I humbly suggest you read more than the subject line written by the unrelated HN submitter?
As for any poor silent, lurkers who wonder if this is how they will be treated if they -- gasp! -- ever find themselves in over their heads, or in a business they realize they don't actually want to be in… our customer reactions have been uniformly:
"Aw, I'm so sorry... Charm is such a nice piece of software... your email was so touching."
Why? Because we've always shown our customers respect by creating great software, and we're showing them even more respect by ensuring we do not make promises we can't keep.
Our friends and technical acquaintances have been full of nothing but sympathy, understanding, and for those closest to the situation, praise for making the right, hard decision.
Yep, it sucked. Yep, we poured something like $200k into development, redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece of software and the best thing I've ever designed.
But is it worth the constant heartache of the impossible task of finding people equipped to work on it? Of having it stuck in some kind of product half-life because of that? Of feeling responsible for, but incapable of, being "on call" in the middle of the night?
Of feeling guilty because, unless we can somehow suddenly be great at those things, we're taking money for a service which might let our customers down?
Nope. It's not worth it.
And boy do I feel lucky and privileged that because we spent nothing but our own money on it, we are free to decide to do whatever we think is right.
See also my principles 10 and 11: http://unicornfree.com/2011/lessons-learned-from-16-years-of...
The email isn't "Wah we hit a server problem, bye bye."
The point is:
"What this has really shown us is that, if we open Charm to the general public, we won't be able to provide you with the kind of service you deserve. We are a tiny team, and so far, we've had zero luck in our attempts to grow by hiring developers. Problems which are small now will only get bigger."
If you've never run a serious product, or a real business, or tried to hire for technical positions, I can understand why you'd zero in on the "facts" about the technical situation and ignore all the "foofy personal window dressing," and write things like "I call BS! It's a smoke screen!" or "Why didn't they just try BSD?"
And yet I addressed the actual problem in very clear terms in a paragraph you can't possibly miss.
Next time somebody makes a hard business decision and you hear about it on HN and come out, irony guns blazing, may I humbly suggest you read more than the subject line written by the unrelated HN submitter?
As for any poor silent, lurkers who wonder if this is how they will be treated if they -- gasp! -- ever find themselves in over their heads, or in a business they realize they don't actually want to be in… our customer reactions have been uniformly:
"Aw, I'm so sorry... Charm is such a nice piece of software... your email was so touching."
Why? Because we've always shown our customers respect by creating great software, and we're showing them even more respect by ensuring we do not make promises we can't keep.
Our friends and technical acquaintances have been full of nothing but sympathy, understanding, and for those closest to the situation, praise for making the right, hard decision.
Yep, it sucked. Yep, we poured something like $200k into development, redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece of software and the best thing I've ever designed.
But is it worth the constant heartache of the impossible task of finding people equipped to work on it? Of having it stuck in some kind of product half-life because of that? Of feeling responsible for, but incapable of, being "on call" in the middle of the night?
Of feeling guilty because, unless we can somehow suddenly be great at those things, we're taking money for a service which might let our customers down?
Nope. It's not worth it.
And boy do I feel lucky and privileged that because we spent nothing but our own money on it, we are free to decide to do whatever we think is right.
See also my principles 10 and 11: http://unicornfree.com/2011/lessons-learned-from-16-years-of...
I call BS. It's either a smoke screen for some other issue, or their app was built back asswards. Either way, this post and the circumstances are indicative that Amy and/or the Charm leadership has no idea what they are doing.
And they were not even launched? Beta or something? The comment about opening to the general public implies that they weren't getting that much usage. These days it's hard to build something with a modern framework and decent hardware that does not at least work for a closed-beta period. Add 100k to the mix and honestly what the hell.
Amy/Tom run Freckle and do some other cool shit so why are they dropping the ball on this? It's like getting a paper cut on your finger and going home for the day because it's just too much to bear. Cry me a fucking river.
Why does this keep happening?! People are throwing cash into the dumbest shit these days.