How Azure Web Sites Sucked in Production(elekslabs.com)
elekslabs.com
How Azure Web Sites Sucked in Production
http://elekslabs.com/2014/05/how-azure-web-sites-sucked-in-production.html
Some hidden gotchas you may encounter when using Azure Web Sites, especially in Shared mode and/or with LAMP stack.
17 comments
Besides the fact that you are using the free ClearDB plan which is not supposed for production use (and it is not related to Microsoft anyway), Azure website (shared) is a preview product [1], with no SLA, you should not consider them for any serious use anyway, so you've made two mistakes here.
[1] http://azure.microsoft.com/en-us/pricing/details/web-sites
[1] http://azure.microsoft.com/en-us/pricing/details/web-sites
Microsoft is trying to compete in a very crowded space, so if the preview product sucks I doubt developers would be willing to try the production version. They should be making their preview a pain-free process that generates positive PR, rather than cutting off a site when the plan gets overutilized.
Not only is the shared tier a preview product it also releases itself from memory when inactive which results in a slow warm up after this happens and a visitor lands on your site.
Great for a personal blog but for a business where your customers expect things instantly, it is not so good.
Great for a personal blog but for a business where your customers expect things instantly, it is not so good.
> it also releases itself from memory when inactive.
That's the nature of application pools in IIS. If the app pool goes inactive for a time, IIS kills the w3wp process running the app. You can set the timeout in the web.config (not in front of code right now, don't know the exact incantations). Does WAWS respect that?
Though, the shutdown would be a good thing for a small site like what WAWS is intended for, because when the site isn't running, it's not still using resources.
In my company's environment (our own datacenter), our monitoring tools keep the app pool from "timing out" because it hits a status page every 2 minutes. We manually (through a script) recycle app pools at a given time every night.
That's the nature of application pools in IIS. If the app pool goes inactive for a time, IIS kills the w3wp process running the app. You can set the timeout in the web.config (not in front of code right now, don't know the exact incantations). Does WAWS respect that?
Though, the shutdown would be a good thing for a small site like what WAWS is intended for, because when the site isn't running, it's not still using resources.
In my company's environment (our own datacenter), our monitoring tools keep the app pool from "timing out" because it hits a status page every 2 minutes. We manually (through a script) recycle app pools at a given time every night.
While that is absolutely true for IIS and the Shared tier in Azure Websites (although it's fully configurable if you own the IIS box[1]), Dedicated tiers (when you own the whole VM) in Azure have a feature called Always On[2] Where there is no idle Time-out on the app itself, so even if the app remained idle for days, w3wp process won't be killed. Also if the app happen to shutdown for any reason (machine restart, Azure Update, Windows Update, app crash, etc) Azure Websites will make sure to send a request to the app to start it up and warm it, then the app can use Application Initialization[3] to do any more complex warmup
[1] http://serverfault.com/a/595215/219792
[2] http://azure.microsoft.com/en-us/documentation/articles/web-...
[3] http://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-...
[1] http://serverfault.com/a/595215/219792
[2] http://azure.microsoft.com/en-us/documentation/articles/web-...
[3] http://www.iis.net/learn/get-started/whats-new-in-iis-8/iis-...
You don't need to have external tool to keep the AppPool from timing out: right-click AppPool in IIS Manager > Advanced Settings > Idle Time-out: set to zero. Also, you don't need the script to reset it at certain time: right-click AppPool in IIS Manager > Recycling... > Specific time(s).
Here's the take-away from this post
* You treated the system's memory like it was some infinite resource. It's not, you need to make sure your application efficiently handles memory and takes measure to ensure it doesn't grow too large. Would you want it swapping on a regular host?
* You thought you could get away with a free database that is used for development.
You're using an application that isn't built well for shared environments, it expects dedicated hosting.
You'd be better off designing your blog around Azure Web Sites, knowing the advantages and disadvantages of the platform and taking advantage of them. Don't try to shoehorn it in.
* You treated the system's memory like it was some infinite resource. It's not, you need to make sure your application efficiently handles memory and takes measure to ensure it doesn't grow too large. Would you want it swapping on a regular host?
* You thought you could get away with a free database that is used for development.
You're using an application that isn't built well for shared environments, it expects dedicated hosting.
You'd be better off designing your blog around Azure Web Sites, knowing the advantages and disadvantages of the platform and taking advantage of them. Don't try to shoehorn it in.
Take-aways that must came with yours:
* There is no way to calculate memory usage, it's completely opaque. The first thing the author told was how he was controlling memory usage, but that does not reflect on Azure's number. Also, swapping is a bit better than just having your site taken down with no warning.
* Nearly all hosting providers have "free" database providing included in their price. (And until I read that artile, I wouldn't even think on adding that "nearly" at the beggining.)
But yes, you shold know your environment, and not shoehorn it in.
* There is no way to calculate memory usage, it's completely opaque. The first thing the author told was how he was controlling memory usage, but that does not reflect on Azure's number. Also, swapping is a bit better than just having your site taken down with no warning.
* Nearly all hosting providers have "free" database providing included in their price. (And until I read that artile, I wouldn't even think on adding that "nearly" at the beggining.)
But yes, you shold know your environment, and not shoehorn it in.
I'm not sure I agree with nearly all hosting providers having free database services. Azure seems like it intends to be a competitor to AWS and Rackspace, both of which charge for their cloud databases.
AWS RDS Pricing - https://aws.amazon.com/rds/pricing/
Rackspace Cloud Database Pricing - http://www.rackspace.com/cloud/databases/pricing/
AWS RDS Pricing - https://aws.amazon.com/rds/pricing/
Rackspace Cloud Database Pricing - http://www.rackspace.com/cloud/databases/pricing/
Paying for provisioned performance is a pretty good idea, but the general script kiddie audience wants stuff to "just work." Obviously there is a divide between SLA quality services and bulk shared services like PHP on Dreamhost.
It may intend that, and may be a competitor in other services, but at least Amazon do not provide any service equivalent to this one.
I think swapping is just as bad as having the quota cut you off, both will result in your site either accepting users or not (if you're swapping you're turning users away by being slow).
Not sure what reliable hosting providers through in database for free and expect it handle 3600+ connections.
Not sure what reliable hosting providers through in database for free and expect it handle 3600+ connections.
Well, no, he told us how he was controlling WordPress' memory usage, but not the web server's or PHP's. And Azure includes free databases, both SQL Server and MySQL, there's just limits (20mb). Which, y'know, is how it works.
I tried to add php_value memory_limit 128M in .htaccess (it's mentioned in the post), but I wasn't aware that Azure pays no attention to .htaccess and I should've been modifying web.config instead.
But I'm a tech guy. As a customer, I would expect Azure to configure the server and PHP on its own and not bother me with the quotas at all :)
But I'm a tech guy. As a customer, I would expect Azure to configure the server and PHP on its own and not bother me with the quotas at all :)
You're a tech guy, and you expected IIS to use Apache configuration files?
EDIT: And as a customer, you're expecting managed hosting from a service that clearly isn't a managed hosting provider? Yeesh. There really are managed hosting providers in the world, if that's what you want.
EDIT: And as a customer, you're expecting managed hosting from a service that clearly isn't a managed hosting provider? Yeesh. There really are managed hosting providers in the world, if that's what you want.
Next time somebody will post a post titled 'How Nginx Sucked in Production' and he'll complain about .htaccess files.
.htaccess is supported not only by Apache. And I wouldn't be surprised if WAWS supported "at least a subset" of .htaccess configuration keys on Azure, since Microsoft has been trying to make a move into the LAMP world during the latest few years. I would consider it a marketable feature.
If we're discussing terms, then we must have a different understanding of what a managed hosting is. A service that allows you to select an application template, deploys the resources for you, sets up source control software, handles OS patches and host software installation for you, is _clearly not_ a managed hosting? What is then?
EDIT: I do admit that editing .htaccess was my own mistake. Not even sure where it came from actually, since the default WAWS WP template doesn't even contain it.
If we're discussing terms, then we must have a different understanding of what a managed hosting is. A service that allows you to select an application template, deploys the resources for you, sets up source control software, handles OS patches and host software installation for you, is _clearly not_ a managed hosting? What is then?
EDIT: I do admit that editing .htaccess was my own mistake. Not even sure where it came from actually, since the default WAWS WP template doesn't even contain it.
Good points, but I must disagree on some details:
* We did not develop the application from scratch and therefore we cannot control its memory consumption, especially if it's a garbage-collected environment. We used WordPress as an existing application template provided by Azure. The development on our side was only about developing a custom WP theme, which is 90% frontend development. The developers barely touched the server side, and we tried to configure the environment to consume not more than X MB.
* Regarding not being built well for shared environments: WordPress powers about 60 million sites in the world, and I bet the vast majority of them runs on a shared hosting. Currently, this blog post is served from one of them :) During the typical days, our traffic is really too low for a dedicated environment, so it would really be an overkill.
* We did not develop the application from scratch and therefore we cannot control its memory consumption, especially if it's a garbage-collected environment. We used WordPress as an existing application template provided by Azure. The development on our side was only about developing a custom WP theme, which is 90% frontend development. The developers barely touched the server side, and we tried to configure the environment to consume not more than X MB.
* Regarding not being built well for shared environments: WordPress powers about 60 million sites in the world, and I bet the vast majority of them runs on a shared hosting. Currently, this blog post is served from one of them :) During the typical days, our traffic is really too low for a dedicated environment, so it would really be an overkill.
The bigger question, gone unanswered, is why your WordPress blog is configured such that three users can tap out 512MB of RAM. I run a few nontrivial WP installs averaging a couple thousand hits a day on 256MB VMs.
My hunch is that anything that actually needs to tap out 512MB of RAM should be running on a plan that's more than $10/month.
My hunch is that anything that actually needs to tap out 512MB of RAM should be running on a plan that's more than $10/month.
Please see the details in my post, I tried to limit RAM usage via usual WordPress configuration tricks, but Azure didn't seem to respond. Perhaps an even bigger question is: why Azure hasn't configured the site itself, if it imposes quotas on it.
I read the post, thanks. Setting aside that .htaccess is Apache-only, needing to allocate 128M per request in WordPress doesn't ring any alarm bells for you? Every trivial WP site I've ever run worked comfortably with under 64M/request (and then you use use supercache, static caching, etc. to reduce the average individual request to something in the single digits).
Azure doesn't configure it for you because it assumes you're knowledgeable. It reacts more harshly than I'd like for sure if you go over, but expecting you to do your work (for cheap) doesn't seem unreasonable to me.
Azure doesn't configure it for you because it assumes you're knowledgeable. It reacts more harshly than I'd like for sure if you go over, but expecting you to do your work (for cheap) doesn't seem unreasonable to me.
This is a mistake on your part, because IIS ignores the .htaccess file.
Azure web sites uses a .user.ini file to setup php.
See the documentation here: http://azure.microsoft.com/en-us/documentation/articles/web-...
Azure web sites uses a .user.ini file to setup php.
See the documentation here: http://azure.microsoft.com/en-us/documentation/articles/web-...
There are sure a number of things to keep in mind when migrating to Azure Web Sites. For instance, since it runs on IIS, Apache configuration files (.htaccess) won't have any effect. And the free MySQL plan is meant for development, not for production use.
So because these guys are either bad at coding, had a site that was on heavy load for that service, or what else you want to call it, an entire platform isn't ready for service? Get real. From everything he told me, if I had a real site & wanted to make use of a lot of things for cheap I'd go for their free tier in a heartbeat.
HN uses this reasoning all the time. For example, "I don't know how to use language x, so language x sucks for sysadmins" or "I don't know how to use language x, so it is too slow"
While I must admit that the title of the post may seem too flashy, the main point I'm trying to make is not 'omg, the quotas are too low for this price', but rather: 'based on this experiment, Azure has given me too much trouble and hidden gotchas to consider it for small or middle-scale websites in the nearest future.'. Not notifying the user about pulling the plug on the website, or hiding a DB quota so deep in the documentation would not make a customer too happy. And while I've worked on (mostly non-Azure) cloud solutions for a couple of years and I'm prepared to read the fine print, I'm scared to imagine how a cloud newbie would feel about this.
Probably best to add "...for PHP Wordpress sites" to the title.
I think that's very true. Azure websites is overall a great ASP.NET solution (one I'm planning on moving my SaaS product to).
I'm not sure where you expect to get great wordpress hosting for "10 times less" than $69/mo. That's not very much spend for a commercial website. Pagely, which has the scale economics advantage of doing nothing but hosting WP sites, starts at $24/mo. To hit the baseline level of "production readiness" with Heroku is going to run you at least $85/mo (2 dynos, 1 free, 1 @ $35/mo + $50/mo postgres instance).
If Azure really wasn't sending log events that memory quotas were being exceeded then I agree the alerting leaves something to be desired. But the DB quota was clearly sending "database error" log messages. Dunno what else you want them to do there.
If Azure really wasn't sending log events that memory quotas were being exceeded then I agree the alerting leaves something to be desired. But the DB quota was clearly sending "database error" log messages. Dunno what else you want them to do there.
Well, currently we're running the blog on a traditional shared hosting that costs less than $10/mo (you can easily find out which one :)) and it handles the blog amazingly well, even during HN bursts. The blog itself it not our primary line of duty, more of a side project actually. So when there's no new updates, it's quiet enough for a shared hosting to handle.
The free/cheap tier is pretty limiting especially for many modern CMS and unoptimized PHP type things.
If your CMS is caching, big memory footprint and some DB thrashing. If you aren't caching, big DB thrashing and some memory consumption.
I'd like a little more flexibility with the lower tiers since I've been "quotad" by memory limits when caching goes bonkers during CMS updates. Something like short-term overage allowances with warnings.
If your CMS is caching, big memory footprint and some DB thrashing. If you aren't caching, big DB thrashing and some memory consumption.
I'd like a little more flexibility with the lower tiers since I've been "quotad" by memory limits when caching goes bonkers during CMS updates. Something like short-term overage allowances with warnings.
I think this is a case of you get what you pay for. $10/month is extremely cheap for hosting. Cheap hosts are great for sharing simple sites or (in the case of digital ocean) simple services with little foot traffic.
Since we're not a professional Internet media (it's not even a commercial project), when we don't hit HN top, the traffic is very low. Currently we're serving this post from a traditional hosting worth about $7/mo, and it handles the HN burst amazingly well.
golden rule of hosting / paas: the less someone pays, the more difficult they will be to satisfy.
this holds all the way down to $0 (free), in which case, the customer is almost never satisfied. because they're getting almost no value for no money.
this holds all the way down to $0 (free), in which case, the customer is almost never satisfied. because they're getting almost no value for no money.
Very good point. But the main message I was trying to deliver is not about the bang for the buck, but rather the amount of issues/decisions WAWS tries to offload to the users instead of handling them intelligently on its side. It's not hard to send an email message, or automatically configure web.config knowing the predefined quota, but the impact on customer satisfaction could be huge.
Our SaaS is on Azure Websites until day one and it has been a pleasure to use, since they added the Always On feature and the new SQL Database pricing tier that allows you to have more robusts SQL databases everything runs very smoothly and fast. We can deploy several times a day without having any downtimes, load balancing, etc.
Is it actually common to limit users on cpu/memory use this way? I'd expect the limitations to simply be based on the resources made available to your processes...
These limits seem weird to me. CPU is a compressible resource. You can get by with less, and it's OK to temporarily soak up available surplus. Memory is incompressible. Either you can allocate it or you can't. It's not OK to "burst" in terms of space because you can't take memory away from a process. It has to give it up freely.
So these are two completely unalike resources that should be treated very differently. A process routinely overusing CPU can simply be capped to a certain rate (I assume you can do this on Windows) but a process that grows too large should simply crash. It doesn't make any sense to take down an application for an hour because it used to much memory.
I suspect that the reason for this enforcement is that Azure has taken some liberties with the isolation between sites and they can't figure out how to charge resources properly. But that is their problem and it shouldn't leak out through the interface and harm the customer.
So these are two completely unalike resources that should be treated very differently. A process routinely overusing CPU can simply be capped to a certain rate (I assume you can do this on Windows) but a process that grows too large should simply crash. It doesn't make any sense to take down an application for an hour because it used to much memory.
I suspect that the reason for this enforcement is that Azure has taken some liberties with the isolation between sites and they can't figure out how to charge resources properly. But that is their problem and it shouldn't leak out through the interface and harm the customer.
Yes, it's common but handled a little more idiomatically, by presenting your available resources as VM/instance/machine. Of course, you aren't given the ability to exceed those metrics in a traditional cloud environment.
I run a Wordpress site on Websites and it works great.
Its not cheap hosting (I'm running a business so I just want my stuff all in one place) but it has been very solid when used at Standard with a paid MySQL database option (the free tier is for development - not production).
Its not cheap hosting (I'm running a business so I just want my stuff all in one place) but it has been very solid when used at Standard with a paid MySQL database option (the free tier is for development - not production).
Funny, I have all my SharpPLM stuff on Azure (free with Bizspark!). So I was going to set up a blog with Wordpress just like you guys did. Same initial experience, it was super easy and quick to get up and running.
I haven't fully made it 'live' yet, but this gives me some pause whether I should. I think the WAWS platform is good overall, but there is probably too many layers of 'magic' happening under the skirt for this to be a great solution yet.
I haven't fully made it 'live' yet, but this gives me some pause whether I should. I think the WAWS platform is good overall, but there is probably too many layers of 'magic' happening under the skirt for this to be a great solution yet.
If you're using MySQL, make sure to select the appropriate plan tier, since the free tier is very limited and not meant for production. And put your configuration in IIS config files and/or in the Azure management portal. It doesn't run on Apache, so Apache configuration files will be ignored.
After they mentioned they were using the shared instance I just stopped reading.
> Error establishing a database connection
I see what they did there
I see what they did there
tl;dr: Using Apache config files to configure IIS doesn't work.