Heroku Postgres Official Maintenance(status.heroku.com)
status.heroku.com
Heroku Postgres Official Maintenance
https://status.heroku.com/incidents/510
9 comments
I'm yet to see an update/fix on Heroku's recent PR nightmare on the random routing issue. Anyone?
They've put together a FAQ about routing: https://blog.heroku.com/archives/2013/4/2/routing_and_web_pe...
They are about to push an update that lets you run larger dynos, which reduces the risk of having all the requests to a dyno stuck doing something long-lived.
I believe the larger dynos are mostly to better handle using Unicorn as the webserver. Unicorn allows multiple workers to run per dyno, and has it's own smart routing, so it compensates well for Heroku's routing system. The downside to Unicorn was that the workers share a dyno's memory and often you couldn't have more than 2 per dyno with 512mb memory. This change helps that a lot.
I'm not sure why this point isn't being made, but they're not going to change anything about how the routing works, least of all for Bamboo.
Heroku is past the point of caring about pushing the envelope, stateful routing is an infrastructure risk and overhead that primarily their least informed customers need or want.
It's a high margin operation is a low margin business, they're trying not to fuck it up.
So stop asking.
Heroku is past the point of caring about pushing the envelope, stateful routing is an infrastructure risk and overhead that primarily their least informed customers need or want.
It's a high margin operation is a low margin business, they're trying not to fuck it up.
So stop asking.
As a customer, no matter what they do, whether it's a high margin risk or low margin profits, I'm supposed to be informed of their decision - whether they try to fix it or not, because I was one of the affected. They promised they would do so in their previous blog posts, hence the curiosity.
>So stop asking.
I have every right to ask and I will rightfully so. A disguised internet profile about sunglasses means nothing to me.
>So stop asking.
I have every right to ask and I will rightfully so. A disguised internet profile about sunglasses means nothing to me.
A good reason to take a few minutes and make sure your backups are up-to-date and working, and that your application won't blow up in unexpected ways when the DB is unreachable.
So. Us normal users will get the security fix on Thursday, but Heroku is somewhat special and gets early access? What does one have to do to become special?
I don't like how this is being handled, no matter how serious the issue. I could kind of understand what the postgres team was doing until this point where they show that while everybody is equal, some are more equal than others and get early access to security fixes.
Doing this does nothing but motivate the bad guys even more to find the hole on their own.
The initial announcement got me scared already, but this just made it worse.
I don't like how this is being handled, no matter how serious the issue. I could kind of understand what the postgres team was doing until this point where they show that while everybody is equal, some are more equal than others and get early access to security fixes.
Doing this does nothing but motivate the bad guys even more to find the hole on their own.
The initial announcement got me scared already, but this just made it worse.
Heroku is special, they host a huge number of Postgres setups that are publicly accessible and somewhat easily discovered. It's not a matter of equality, though, it's one of trust/privacy. Heroku is an easy company for the Postgres guys to trust since they have a pre-existing, healthy relationship.
Meanwhile they are holding back a security fix for numerous other companies that also take security extremely seriously. This creates a situation where companies considering posgresql will now have to ask "will I get security fixes as soon as they are ready or will I intentionally be left vulnerable while more privileged users get early access?"
Not a good precedent as far as I'm concerned.
Not a good precedent as far as I'm concerned.
I don't think you'll find many companies with the exposure (in terms of access to vulnerable Postgres installs) of Heroku, regardless of how seriously they take security. And I suspect most companies that use Postgres don't interact much with the authors/maintainers.
I don't think you're being very fair here, honestly. I doubt anyone with any kind of Postgres responsibility is intentionally leaving people exposed, or considers some users more privileged than others.
I don't think you're being very fair here, honestly. I doubt anyone with any kind of Postgres responsibility is intentionally leaving people exposed, or considers some users more privileged than others.
This is SOP for major open source projects. Most security release announcement lists for major projects consist of big companies whom would be most at risk to a 0day exploit. There is usually an embargo period during which said large companies get to patch their systems before the release goes public.
It is open source. If you are so worried about the SLA you get from this open source software, maybe you should a) pay for better support or b) fix it yourself.
That's a cheeky cop-out. The original criticism is still valid, and we don't know how many companies outside of Heroku (if any) are getting the patch ahead of time. We don't all have the resources to a) Pay Oracle grade prices for support or b) the man power to comb through millions of lines of code to fix it ourselves.
If these are the best excuses for lack of full transparency one could could come up with, then why would anyone choose a Linux distro? Or any of the *BSDs for that matter? Transparency and an expectation of due diligence is why a lot of companies (in IT and elsewhere) still stay in business while choosing Open Source software.
Besides, Postgres isn't some run-of-the-mill startup with a dropped letter domain name. It's a venerable software foundation with almost 20 years of work and experience behind it. Which makes this a questionable course of action on their part despite the need for discretionary release.
If these are the best excuses for lack of full transparency one could could come up with, then why would anyone choose a Linux distro? Or any of the *BSDs for that matter? Transparency and an expectation of due diligence is why a lot of companies (in IT and elsewhere) still stay in business while choosing Open Source software.
Besides, Postgres isn't some run-of-the-mill startup with a dropped letter domain name. It's a venerable software foundation with almost 20 years of work and experience behind it. Which makes this a questionable course of action on their part despite the need for discretionary release.
I agree, I don't think this reflects well on the postgres community( in so far that Heroku is a member of that community ).
Heroku has certainly done a lot to demonstrate how effective postgres can be practice, in addition to other contributions ).
All that being said, we don't actually know what the fix looks like, or indeed what their code looks like either right? For all we know this could be being pushed upstream? It could in some other way be unsuitable for general consumption/adoption at this time. It is possible that the fact that Heroku is going to burn it in will end up uncovering an issue or some other fact that will benefit the project, or they may be uniquely "extra" vulnerable because of deviations.
The PostgreSQL license cuts(so to speak) both ways in this case: http://www.postgresql.org/about/licence/
Heroku has things like: https://postgres.heroku.com/fork
they are a database as a service, not a distributor of software.
Heroku has certainly done a lot to demonstrate how effective postgres can be practice, in addition to other contributions ).
All that being said, we don't actually know what the fix looks like, or indeed what their code looks like either right? For all we know this could be being pushed upstream? It could in some other way be unsuitable for general consumption/adoption at this time. It is possible that the fact that Heroku is going to burn it in will end up uncovering an issue or some other fact that will benefit the project, or they may be uniquely "extra" vulnerable because of deviations.
The PostgreSQL license cuts(so to speak) both ways in this case: http://www.postgresql.org/about/licence/
Heroku has things like: https://postgres.heroku.com/fork
they are a database as a service, not a distributor of software.
But that is exactly the point - this patch isn't open source (at the moment). It's closed source and is being given to preferred "customers". If it was really open source then we wouldn't have this problem.
I'm not an expert in this field but it seems like the proper thing to do would be to announce a release date that gives everyone sufficient time to prepare and then give it to everyone.
I'm not an expert in this field but it seems like the proper thing to do would be to announce a release date that gives everyone sufficient time to prepare and then give it to everyone.
The patch isn't closed source - the patch is in production, and will soon be open sourced.
Calling it closed source is just rhetoric. You just don't think they are moving fast enough or that there should be pragmatic reasons to give favorable treatment to any users.
And so I think it's valid to say Postgres is open source, so fix it yourself if you don't want to wait for the people who are fixing it for you for free.
Calling it closed source is just rhetoric. You just don't think they are moving fast enough or that there should be pragmatic reasons to give favorable treatment to any users.
And so I think it's valid to say Postgres is open source, so fix it yourself if you don't want to wait for the people who are fixing it for you for free.
And so I think it's valid to say Postgres is open source, so fix it yourself if you don't want to wait for the people who are fixing it for you for free.
This is also rhetoric, not to mention damaging and insulting. You know damn well that nobody can "fix it themselves." It's not known what the problem is, let alone how to fix it. For any amount of money.
It's not even about the wait time. The entire point of the embargo is that no one releases early. I initially thought this was an early release, though given that heroku uses a service model it probably isn't. The postgres team obviously cares deeply about handling this in the right way. A great many people value that, myself included.
This is also rhetoric, not to mention damaging and insulting. You know damn well that nobody can "fix it themselves." It's not known what the problem is, let alone how to fix it. For any amount of money.
It's not even about the wait time. The entire point of the embargo is that no one releases early. I initially thought this was an early release, though given that heroku uses a service model it probably isn't. The postgres team obviously cares deeply about handling this in the right way. A great many people value that, myself included.
If it "will soon be open sourced" what is it if it's not "closed source"?
Whilst it's not closed source in the traditional sense, it very much exists (there's no doubt about this) and isn't available to the general population so therefore, IMHO, is closed source.
Don't get me wrong - I understand the need to stage the release however I don't agree with the priority/privileged access.
Whilst it's not closed source in the traditional sense, it very much exists (there's no doubt about this) and isn't available to the general population so therefore, IMHO, is closed source.
Don't get me wrong - I understand the need to stage the release however I don't agree with the priority/privileged access.
As far as I know, postgres has not had a history of "You only get up-to-date patches if you pay us." I've not even seen that you can pay for early access. If they want to use delayed patches as a source of revenue, they should be up front about it so that people can be ready for cases like this.
I would be more concerned about someone being able to reverse engineer the patch somehow and this leaking to those other users that way(seems unlikely).
I will be curious to know who identified the problem initially.
I will be curious to know who identified the problem initially.
iirc Heroku has several full time employees who have commit access to PG. ie they're paying for some of postgres's development.
False (I work there), although some of my colleagues have submitted patches.
Commit access is handed out very selectively at postgresql.org, and committing patches is not the bottleneck in making features (reviewing and writing the patches is).
Commit access is handed out very selectively at postgresql.org, and committing patches is not the bottleneck in making features (reviewing and writing the patches is).
There are similar problems with XEN. aws and linode getting early access etc.
this is happens with other opensource projects as well. you just don't hear about it. if you have contacts in the project then you have a good chance of getting early patches.
This is true for most software vendors. In particular companies which make IDS software, control large number of deployments or are in highly-targeted sectors (defence, finance, etc.) will tend to have arrangements with software vendors to get early access to patches/vulnerability information.
Typically to get that type of relationship you need to show the vendor that you have a real need for early access and that you have strong internal controls that would prevent any information gained from early access from leaking.
Typically to get that type of relationship you need to show the vendor that you have a real need for early access and that you have strong internal controls that would prevent any information gained from early access from leaking.
This is not an uncommon practice at all. There are other projects that provide early notifications to large vendors. Part of the point of that early notice is so binary packages can be available. Packages can't get built, tested and released alongside security announcements without someone getting early notification. Users are in a much better position if the binary packages are available when the announcement hits than if the announcement hits and the binaries are available a week later.
If the issue leaks out early or someone independently finds it they're just in that much better of a position to release the fix. Vendors will be closer to having packages ready.
Yes, it's probably not entirely fair. But there's no way to tell everyone without telling the bad guys too. The risk of telling select people to get the ball rolling is worth it if the entire ecosystem is ready when the announcement comes out.
Dealing with these issues is always going to be about managing the risk. You can't draw a line in the sand.
If the issue leaks out early or someone independently finds it they're just in that much better of a position to release the fix. Vendors will be closer to having packages ready.
Yes, it's probably not entirely fair. But there's no way to tell everyone without telling the bad guys too. The risk of telling select people to get the ball rolling is worth it if the entire ecosystem is ready when the announcement comes out.
Dealing with these issues is always going to be about managing the risk. You can't draw a line in the sand.
I thought the point of the delay was so that people could not reverse engineer the patch through the source or binaries? I suppose with Heroku being a service it might be the case that people don't have access to the new patched binaries. If that's the case, then there is no issue.
I can't speak for the specifics of Postgresql or Heroku. But in my experience the expectation is that you're supposed to control access to the information. If you produce binaries you're not supposed to make them accessible prior to the embargo date. If you can upgrade without revealing the binaries I don't really see a problem with doing so.
Responsible versus full disclosure is a debate rooted in problems far older than HN. At least we get to see security issues in the light these days!
I'm not sure this is a case of responsible vs full disclosure. Obviously, we don't have full disclosure of the issue yet. However, that doesn't mean we won't have it on the 4th when Postgresql said they would be releasing their fix.
To some degree I don't think "full disclosure" and "responsible disclosure" are very useful terms since they have so many different interpretations.
But looking at this particular practice, I'm not sure there's been much debate in the open source world about this practice of pre-notification to vendors/large installations.
In my personal opinion I think it's absolutely necessary. Most open source users depend on binary packages provided by someone else. Unlike a closed source model when everything happens within a single company. Even then I'd be surprised if some of this doesn't happen with large customers with closed source software.
So in practice I don't think it's anything that doesn't happen with just about all heavily used pieces of software. It's just that this case happened to be more obvious this time. Largely because Postgresql closed their public repository in order to package the fix and announced that they were doing so.
To some degree I don't think "full disclosure" and "responsible disclosure" are very useful terms since they have so many different interpretations.
But looking at this particular practice, I'm not sure there's been much debate in the open source world about this practice of pre-notification to vendors/large installations.
In my personal opinion I think it's absolutely necessary. Most open source users depend on binary packages provided by someone else. Unlike a closed source model when everything happens within a single company. Even then I'd be surprised if some of this doesn't happen with large customers with closed source software.
So in practice I don't think it's anything that doesn't happen with just about all heavily used pieces of software. It's just that this case happened to be more obvious this time. Largely because Postgresql closed their public repository in order to package the fix and announced that they were doing so.
You are not entitled to information just because you think you are, or because they have a tradition of publicizing information.
Lose the sense of entitlement.
Lose the sense of entitlement.
> Doing this does nothing but motivate the bad guys even more to find the hole on their own.
Please explain the logic here. How does Heroku getting early access provide more motivation to discover vulnerabilities?
Please explain the logic here. How does Heroku getting early access provide more motivation to discover vulnerabilities?
I was just recently considering whether we should move our postgresql over to Heroku.
On one hand, it sounds promising that they deal with important security updates, and can do it even before the fix is officially announced. Very impressive.
On the other hand, not being able to schedule the downtime to fit with your app/userbase is quite annoying. I understand this could be a potentially big security vulnerability that needs urgent fixing. But each organization is different and their risks are therefore different. Some people might prefer to take the risk of a few hours delay to apply the fix (being exposed) in exchange for not having to experience unscheduled downtime of this nature... It seems a little awkward to me that Heroku is taking this decision for its entire customer-base in such a way.
On one hand, it sounds promising that they deal with important security updates, and can do it even before the fix is officially announced. Very impressive.
On the other hand, not being able to schedule the downtime to fit with your app/userbase is quite annoying. I understand this could be a potentially big security vulnerability that needs urgent fixing. But each organization is different and their risks are therefore different. Some people might prefer to take the risk of a few hours delay to apply the fix (being exposed) in exchange for not having to experience unscheduled downtime of this nature... It seems a little awkward to me that Heroku is taking this decision for its entire customer-base in such a way.
This is rather disconcerting. They are given us a window of 48 hours where our site will be down for 60 seconds. On top of that we cannot schedule the time with them.
60 seconds may seem like a small inconvenience but it is one still the same. I host a few clients on their servers and I need to relay this information. I sound like a jackass being that I can't give more specifics on the timing of the server being down to my clients. How does Heroku not feel the same?
I am appreciative of the update but that is also what I pay them to do.
60 seconds may seem like a small inconvenience but it is one still the same. I host a few clients on their servers and I need to relay this information. I sound like a jackass being that I can't give more specifics on the timing of the server being down to my clients. How does Heroku not feel the same?
I am appreciative of the update but that is also what I pay them to do.
> How does Heroku not feel the same?
Because they, as a hosting provider, have several orders of magnitude more clients to please than yourself. The only way to be fair to everyone is that everyone gets treated like shit (obviously embellished).
All you should be doing to CYA is telling your clients exactly why this is happening and that its Heroku, and not you, that holds the blame at this point.
The caveat would be those difficult customers (suits, exec managers, etc) that you have to handle with kid gloves. For these people... I don't think there's a graceful way to present the upcoming issue.
Because they, as a hosting provider, have several orders of magnitude more clients to please than yourself. The only way to be fair to everyone is that everyone gets treated like shit (obviously embellished).
All you should be doing to CYA is telling your clients exactly why this is happening and that its Heroku, and not you, that holds the blame at this point.
The caveat would be those difficult customers (suits, exec managers, etc) that you have to handle with kid gloves. For these people... I don't think there's a graceful way to present the upcoming issue.
I doubt customers typically know or care whether you use heroku, Amazon or any other provider. What they want is to see their system up and running.
That said, I wonder if with the heroku postgresql setup, this vulnerability could mean that if even one db is compromised, it could allow access to other databases too. Maybe heroku runs several postgresql db's on the same virtual/physical hosts?. If that's the case, then heroku simply can't afford to let even one database stay exposed because it would risk all others.
Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...
That said, I wonder if with the heroku postgresql setup, this vulnerability could mean that if even one db is compromised, it could allow access to other databases too. Maybe heroku runs several postgresql db's on the same virtual/physical hosts?. If that's the case, then heroku simply can't afford to let even one database stay exposed because it would risk all others.
Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...
> Whether or not this is the case, I don't know, but it kinda feels like running your app on a shared-hosting account...
Remember that you are running your app on a shared hosting account with Heroku.
Now, it has somewhat more isolation than a typical shared hosting account, but less isolation than using physically separate hardware.
Heroku is built on top of AWS, so your machine is running on the same physical hardware, though a different virtual machine, as other AWS customers. Furthermore, Heroku uses LXC to isolate its dynos; so you are running on the same VM as another Heroku customers, albeit separated by a container barrier. And finally, if you're using Postgres, then you're running on a shared database service, which is pretty much exactly like what you'd get with shared hosting.
Remember that you are running your app on a shared hosting account with Heroku.
Now, it has somewhat more isolation than a typical shared hosting account, but less isolation than using physically separate hardware.
Heroku is built on top of AWS, so your machine is running on the same physical hardware, though a different virtual machine, as other AWS customers. Furthermore, Heroku uses LXC to isolate its dynos; so you are running on the same VM as another Heroku customers, albeit separated by a container barrier. And finally, if you're using Postgres, then you're running on a shared database service, which is pretty much exactly like what you'd get with shared hosting.
>What they want is to see their system up and running.
This would be a good chance to take a step back and think about what's more important: using Heroku or controlling your up-time?
This would be a good chance to take a step back and think about what's more important: using Heroku or controlling your up-time?
I understand they have more clients to please than myself. Even a smaller window of time would be helpful or to know it is going to be done in off hours. There's no reason I should be fine with this as a paying customer.
I don't think a company that has clients all around the globe like Heroku has "off hours".
It may be de-rigueur to bash Heroku in the wake of the Rap Genius drama/revelations but this kind of "downtime to apply critical security patch" announcement is precisely why I choose them.
I spotted the announcement of the postgres vulnerability and thought "sounds serious". Now I read that my production databases will be fixed as soon as the patch is available without me having to engage with the details.
I spotted the announcement of the postgres vulnerability and thought "sounds serious". Now I read that my production databases will be fixed as soon as the patch is available without me having to engage with the details.
>Due to the nature of this update, a scheduled time is not possible.
What, it takes place outside of linear time?
This crap is unacceptable. Even my $2.00 shared host tells me when they're going to take down my site without asking.
What, it takes place outside of linear time?
This crap is unacceptable. Even my $2.00 shared host tells me when they're going to take down my site without asking.
More likely it takes around 5min per host if everything goes right. If they have 100 hosts and get delayed on the 10th and 15th for some reason, all other scheduled times would get shifted... And the upside is that they spot failures as they appear, rather than "oops, the last 5 hosts we did at the same time do not start up, there's some big issue".
</speculation>
</speculation>
Just as an aside, how good is Heroku's Postgres performance, speed wise?
Don't ask for anecdotes -- test it yourself with conditions similar to those that you'd be operating under.
Hey, reviews and anecdotes save time. Everyone does it.