Ask HN: Hacking concerns: does it make a difference which OS I run my SAAS on?
9 comments
Not much. Use an LTS version of a distribution which you understand well. You're more likely to have appsec problems caused by yourself rather than being compromised by problems in the OS.
Make sure you're able to do firedrills like e.g. upgrading the version of Nginx or OpenSSL which you use in response to a vulnerability in them getting discovered. We've been doing a lot of that these last two years, and it probably won't get any less common.
Make sure you're able to do firedrills like e.g. upgrading the version of Nginx or OpenSSL which you use in response to a vulnerability in them getting discovered. We've been doing a lot of that these last two years, and it probably won't get any less common.
Going to echo patio11's comment. But add a little too. Do not run Wordpress / Wordpress Plugins, or Drupal, on the same system you are doing anything with user data. These continually have security issues and you will almost certainly be owned [1, 2]. Out source blog hosting to one of the many hosting providers that maintains the software install for you. The chances of actually getting owned running stock Apache, or Nginx, are extremely low. You really need to watch what third party apps you install, like Wordpress, Drupal, or an unsecured phpMyAdmin.
ps. Make sure you have source code and database backups happening on a regular basis. If the server dies tonight, are you able to restore it to a point in time? What about getting the application stack installed quickly? Think about these things.
[1] https://wordpress.org/news/category/security/
[2] https://www.drupal.org/security
ps. Make sure you have source code and database backups happening on a regular basis. If the server dies tonight, are you able to restore it to a point in time? What about getting the application stack installed quickly? Think about these things.
[1] https://wordpress.org/news/category/security/
[2] https://www.drupal.org/security
For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates.
The reason WordPress tends to be root city is that there are a great many installs out there, of which many have never had step one done to harden it, and were last given security updates ~ when they were installed.
I would still outsource WordPress, but that's mostly in the vein of "the cost of outsourcing is less than the cost of me having to think about it one hour a month, while 1 hr/mo is a reasonable floor for the time cost function, but the actual value will probably exceed that at least once in a given year."
A WPEngine subscription costs $30 to "I don't care, why are you wasting my time with numbers this small." Developer time costs between $75/hr and "everything is on fire and you can only put out one fire at a time." Ergo it makes economic sense to configure nginx / DNS once, then outsource further complications to paid external support.
The reason WordPress tends to be root city is that there are a great many installs out there, of which many have never had step one done to harden it, and were last given security updates ~ when they were installed.
I would still outsource WordPress, but that's mostly in the vein of "the cost of outsourcing is less than the cost of me having to think about it one hour a month, while 1 hr/mo is a reasonable floor for the time cost function, but the actual value will probably exceed that at least once in a given year."
A WPEngine subscription costs $30 to "I don't care, why are you wasting my time with numbers this small." Developer time costs between $75/hr and "everything is on fire and you can only put out one fire at a time." Ergo it makes economic sense to configure nginx / DNS once, then outsource further complications to paid external support.
>For any X, getting rooted is still often a matter of someone with 1000 botnet nodes to burn running a for loop against an IP range. At which point, your odds of getting rooted are the product of how desirable that target is to write for loops against and how long it has been since you last applied security updates.
If you're looking at all Internet-facing servers in aggregate, yes, but if you know what you're doing then the odds of getting owned by a random botnet are very slim. Assuming you have a good security posture, your risks are dedicated and intelligent attackers and 0-days, not botnets scanning for low hanging fruit.
If you're looking at all Internet-facing servers in aggregate, yes, but if you know what you're doing then the odds of getting owned by a random botnet are very slim. Assuming you have a good security posture, your risks are dedicated and intelligent attackers and 0-days, not botnets scanning for low hanging fruit.
Careful, there - it's easy to forget to patch one site out of some hundred-odd. Big organizations are hard. (But yes, a single competent admin who doesn't take holidays will usually have pretty decent security.)
Oh yes, once you have more than a few employees you become much more at risk of that. I was assuming a scenario of one sys admin managing a small set of servers.
I know that Wordpress has a bit of a bad reputation, but this is the first have heard that Drupal has similar problems.
Is there anywhere you can check how good or bad a particular CMS or framework is regarding security?
Is there anywhere you can check how good or bad a particular CMS or framework is regarding security?
I work for a very large Drupal hosting company. We host important web sites on Drupal for corporations big and small.
Web service software is not secure unless you patch it rapidly. There is no silver bullet. Last year there was a big security hole in Drupal, which we patched. Much like the ones that routinely turn up in Wordpress, which they patched. Much like the ones that happened, what, three or four times to Rails in the last few years? Which are like the ones they found in OpenSSL, which are like the ones they found in bash - bash, which is not supposed to be a web service! But which is nonetheless connected to the web in lo these many ways.
You have to watch for patches and apply the patches. Ubuntu gets six of them every week. There are tools that apply them automatically, or with one line of typing (which rapidly ends up in the autocomplete buffer, let me tell you), and server reboots are only needed once a week or so, but you have to do it. You have to spot the patches and you have to apply them. It is like brushing your teeth, with the occasional excitement of an urgent tooth-brushing emergency. It's mostly boring.
You must also have a plan to recover if a patch doesn't get applied in time, because these patches can be hard to write and can take several tries, and because you won't always be perfectly fast at getting your chores done. So you need to keep your family jewels away from the Apache server process, preferably in an alternate universe.
If you are a company, this stuff graduates from boring to "hard", because how do you know that the employee who swore to do the upgrades every week didn't forget for six critical days? But mostly it is just about patience and routine. Routine turns out to be surprisingly hard.
Web service software is not secure unless you patch it rapidly. There is no silver bullet. Last year there was a big security hole in Drupal, which we patched. Much like the ones that routinely turn up in Wordpress, which they patched. Much like the ones that happened, what, three or four times to Rails in the last few years? Which are like the ones they found in OpenSSL, which are like the ones they found in bash - bash, which is not supposed to be a web service! But which is nonetheless connected to the web in lo these many ways.
You have to watch for patches and apply the patches. Ubuntu gets six of them every week. There are tools that apply them automatically, or with one line of typing (which rapidly ends up in the autocomplete buffer, let me tell you), and server reboots are only needed once a week or so, but you have to do it. You have to spot the patches and you have to apply them. It is like brushing your teeth, with the occasional excitement of an urgent tooth-brushing emergency. It's mostly boring.
You must also have a plan to recover if a patch doesn't get applied in time, because these patches can be hard to write and can take several tries, and because you won't always be perfectly fast at getting your chores done. So you need to keep your family jewels away from the Apache server process, preferably in an alternate universe.
If you are a company, this stuff graduates from boring to "hard", because how do you know that the employee who swore to do the upgrades every week didn't forget for six critical days? But mostly it is just about patience and routine. Routine turns out to be surprisingly hard.
I am aware that there is no silver bullet, but at the same time certain pieces of software have worse reputations than others (Adobe Flash, Java for example). I assume this is for a reason.
I work for a hosting company, and we get to see quite a few hacked CMSs. The most hacked—of the popular CMSs—would be Joomla, closely followed by WordPress, and then Drupal. However the margin between Drupal and WordPress is quite big.
I can never fathom why Joomla is so popular. It's horrendous to work with. Also, the attention given to security by the majority of the plugin development community, and even Joomla's core dev team is minuscule.
I can never fathom why Joomla is so popular. It's horrendous to work with. Also, the attention given to security by the majority of the plugin development community, and even Joomla's core dev team is minuscule.
I’m afraid I have no good links for you. And if you really want to to know, you should probably have a security research firm do a survey for you. But that is probably not within the budget of a startup.
The OP also asked about a difference between distros (DIY) and "something else" - and I do think that there is a large difference in the security + maintenance surface area that you have to contend with on a Platform As A Service like Heroku versus your own box.
What's more I think you hit on exactly the reasons why: I'm much more likely to screw something up by misconfiguration or lack of attention than a team of security and maintenance people at a company whose sole job is to make sure that I don't get burnt by these things.
OP is trying to get a SAAS off the ground and not (presumably) doing something with handling distro security and server maintenance - for the minimal cost difference in getting started it's hard to see how the time+effort difference in being your own sysadmin is worth it.
What's more I think you hit on exactly the reasons why: I'm much more likely to screw something up by misconfiguration or lack of attention than a team of security and maintenance people at a company whose sole job is to make sure that I don't get burnt by these things.
OP is trying to get a SAAS off the ground and not (presumably) doing something with handling distro security and server maintenance - for the minimal cost difference in getting started it's hard to see how the time+effort difference in being your own sysadmin is worth it.
Heroku can be expensive. You're looking at $240/yr just for SSL (vs. $0 - $30/yr for non-wildcard SSL). For a 1GB VPS, a dyno costs ~3x as much as Digital Ocean.
For protection from cyber crime, you might be better off paying an insurance company than Heroku prices.
For protection from cyber crime, you might be better off paying an insurance company than Heroku prices.
Heroku is expensive from a hosting perspective but cheap from a sysadmin cost perspective.
And it's this total cost of running a service (or at the very least the opportunity cost of not working on things essential to your business and the risk of your site getting screwed by somebody malicious) that I'm trying to convey.
For a business, $240/year to get a SSL terminating load balancer maintained by a dedicated security sysadmin team to sit in front of their app looks like a heckuva deal.
There is just so much work that is involved in keeping a system properly up and secure that unless you're valuing your time at near zero dollars an hour, or you have so much traffic and cash coming in that you can get a dedicated person to handle it Heroku makes monetary sense.
And it's this total cost of running a service (or at the very least the opportunity cost of not working on things essential to your business and the risk of your site getting screwed by somebody malicious) that I'm trying to convey.
For a business, $240/year to get a SSL terminating load balancer maintained by a dedicated security sysadmin team to sit in front of their app looks like a heckuva deal.
There is just so much work that is involved in keeping a system properly up and secure that unless you're valuing your time at near zero dollars an hour, or you have so much traffic and cash coming in that you can get a dedicated person to handle it Heroku makes monetary sense.
I guess for me, part of the value of my service is the infrastructure. And I sell based on server instance, so any added cost gets passed to the customer (or deducted from my earnings).
For someone competent with sysadmin work, I think it's maybe worth the time. It's not like for any hour not spent on my systems I'd otherwise be earning $200 automatically.
My message would just be for people to not be intimidated by the concept of something being so much work that it's not worth your time. I saw that argument for years with email as apparently conventional wisdom. I've been hosting my own email for years now with about 3 hours of upfront work and (almost) none thereafter.
For someone competent with sysadmin work, I think it's maybe worth the time. It's not like for any hour not spent on my systems I'd otherwise be earning $200 automatically.
My message would just be for people to not be intimidated by the concept of something being so much work that it's not worth your time. I saw that argument for years with email as apparently conventional wisdom. I've been hosting my own email for years now with about 3 hours of upfront work and (almost) none thereafter.
Hosting your own email works great until you try to actually send any. [1] But if you send enough, you eventually have to deal with the spam infrastructure. Customers will complain that critical transactional emails ended up in user spam folders. One person too many will accidentally mark one of your mails as spam. Or you'll relaunch some cloud infrastructure and suddenly your IP is one that is on all the blacklists because of some earlier history that you could never have known. Or one of your customers will innocently assume that you are Mailchimp and send a whole bunch of marketing email through your server.
The symptom of any such problem is that email starts falling into the garbage, perhaps silently. But if you are watching your stats like a hawk (a costly and very boring hobby ) - or, much more likely, if your customers start screaming about the very-real high costs of lost email - you will figure that out within a few days, so it might not take more than another day to figure out how to get yourself off the blacklist (a manual process) and how to apologize to customers for the lost email (a manual process). But, if you value your time at in any way, your first encounter with that problem will be your last, because email services are cheap and easy. I have yet to pay for mine; in small quantities Mailchimp, Mailgun et al are free. Indeed, given that low cost, why wait to get burned before you stop playing with fire, even if the fire looks cute and innocent and warm right now?
[1] Actually, I'm lying: A much more serious problem which occurs at any volume is that email is another piece of the security perimeter. Even if mail servers had almost no configuration and a history of bug-free operation - which they do not, I am joking, and the joke is not very funny - one of the basic rules of security is to put as much distance as possible between your data and services which connect to outside machines. Mailgun is free in small quantities and runs outside your entire datacenter. The math kind of does itself.
The symptom of any such problem is that email starts falling into the garbage, perhaps silently. But if you are watching your stats like a hawk (a costly and very boring hobby ) - or, much more likely, if your customers start screaming about the very-real high costs of lost email - you will figure that out within a few days, so it might not take more than another day to figure out how to get yourself off the blacklist (a manual process) and how to apologize to customers for the lost email (a manual process). But, if you value your time at in any way, your first encounter with that problem will be your last, because email services are cheap and easy. I have yet to pay for mine; in small quantities Mailchimp, Mailgun et al are free. Indeed, given that low cost, why wait to get burned before you stop playing with fire, even if the fire looks cute and innocent and warm right now?
[1] Actually, I'm lying: A much more serious problem which occurs at any volume is that email is another piece of the security perimeter. Even if mail servers had almost no configuration and a history of bug-free operation - which they do not, I am joking, and the joke is not very funny - one of the basic rules of security is to put as much distance as possible between your data and services which connect to outside machines. Mailgun is free in small quantities and runs outside your entire datacenter. The math kind of does itself.
I was referring to just normal, non-automated emails above. An alternative to say, Google Apps, where I can fit an entire email server on a VPS with less than 400 MB memory, vs $10/user with Google.
I agree with you for transactional email.
For any DIY project there's a limit where you really should just pay someone. But for instance I run wrote my own Stripe-based invoice generation, subscription management, and dunning vs paying for a service. It works for me for now. If one day my little app breaks down if/when I have thousands of customers then I'll outsource it.
I run my own private Git repo vs paying $7 to github. Etc.
Edit: to add, I am selling software I wrote basically from scratch (and there's no way around that). Like a comment way up above, my biggest risk is in that stack. I suppose with the uber-paranoid mindset I should have outsourced my own development work. At the end of the day, someone like me is responsible for something important. It's not life or death with my software so at least I got that.
I agree with you for transactional email.
For any DIY project there's a limit where you really should just pay someone. But for instance I run wrote my own Stripe-based invoice generation, subscription management, and dunning vs paying for a service. It works for me for now. If one day my little app breaks down if/when I have thousands of customers then I'll outsource it.
I run my own private Git repo vs paying $7 to github. Etc.
Edit: to add, I am selling software I wrote basically from scratch (and there's no way around that). Like a comment way up above, my biggest risk is in that stack. I suppose with the uber-paranoid mindset I should have outsourced my own development work. At the end of the day, someone like me is responsible for something important. It's not life or death with my software so at least I got that.
FWIW: this is imminently sensible advice, too.
In addition to what patio11 said with regard to firedrills for updates, make sure you use some type of repeatable automation. chef, salt, puppet, ansible -- just pick one and use it... _religiously_
I suggest you pick an LTS version of an OS that is extremely popular like Ubuntu. That way you will have lots of people paying attention and patching security flaws ASAP. Then make sure you update regularly/automatically. Put it behind a strong firewall, AWS makes this really easy but you can do everything you need to with iptables. Disable password based auth in ssh and force key auth.
I disagree with yeukhon. You don't need to hire someone just because you asked that question. Unless you are storing highly sensitive data, then I'll leave that up to you.
Just remember to constantly update your server.
I disagree with yeukhon. You don't need to hire someone just because you asked that question. Unless you are storing highly sensitive data, then I'll leave that up to you.
Just remember to constantly update your server.
It generally comes down to picking something you understand how to update and manage. Most of the base OS security is a practice in staying up to date and applying patches very regularly.
There are two types of Linux OSes in general: versioned release and rolling release, which I will explain below.
When choosing a versioned release distro you need to pick something that you feel comfortable you can update on a regular basis without breaking your application dependencies AND plan for the eventual migration to the next versioned release of the distro. Any release you choose will be supported for several years with security updates and then fazed out. When that end of life date hits you must move your application to that next version. Examples of these types of distros are CentOS, Ubuntu LTS, etc.
An alternative is building your application with containers and choosing a rolling update distro. The advantage for security is two fold:
1) It makes it easier to roll out new versions of your application on a regular basis because it is possible for two versions of your app to live side-by-side. And for a new web based project most of your security problems will likely be from your web frameworks, custom code, etc not from the underlying OS. So, having a system in place to get those updates out will be critical.
2) The Linux OS can be updated on a much tighter cadence because your application isn't relying on much from the OS besides the container runtime. Rolling release server distros include Arch Linux, CoreOS, Gentoo, etc. CoreOS even fully-automates the OS update process to remove the burden of remembering to audit and apply updates.
Using a rolling release distro without containers is an option too but can be bumpier since you will regularly be going through migrations like /usr/bin/python being 2.7 and then migrating to 3.0 without you noticing.
The final thing you might consider, particularly if operations aren't your forte, is using something like Google App Engine, Heroku, etc. These systems are ideal for building a prototype and focusing on the initial minimal viable product.
disclosure: I work at CoreOS
There are two types of Linux OSes in general: versioned release and rolling release, which I will explain below.
When choosing a versioned release distro you need to pick something that you feel comfortable you can update on a regular basis without breaking your application dependencies AND plan for the eventual migration to the next versioned release of the distro. Any release you choose will be supported for several years with security updates and then fazed out. When that end of life date hits you must move your application to that next version. Examples of these types of distros are CentOS, Ubuntu LTS, etc.
An alternative is building your application with containers and choosing a rolling update distro. The advantage for security is two fold:
1) It makes it easier to roll out new versions of your application on a regular basis because it is possible for two versions of your app to live side-by-side. And for a new web based project most of your security problems will likely be from your web frameworks, custom code, etc not from the underlying OS. So, having a system in place to get those updates out will be critical.
2) The Linux OS can be updated on a much tighter cadence because your application isn't relying on much from the OS besides the container runtime. Rolling release server distros include Arch Linux, CoreOS, Gentoo, etc. CoreOS even fully-automates the OS update process to remove the burden of remembering to audit and apply updates.
Using a rolling release distro without containers is an option too but can be bumpier since you will regularly be going through migrations like /usr/bin/python being 2.7 and then migrating to 3.0 without you noticing.
The final thing you might consider, particularly if operations aren't your forte, is using something like Google App Engine, Heroku, etc. These systems are ideal for building a prototype and focusing on the initial minimal viable product.
disclosure: I work at CoreOS
My 2p:
Use something you're familiar with, and that you have a good source of information flow, by which I mean, subscribe to security updates (including for whatever stack you put on your server), and certainly have a test server simply for playing with things, getting used to functionality.
I have a shell app on my phone, which I love, so I can log in at a moment's notice, wherever I am, just to check things out or apply a change.
The Linode and DO tutorials are pretty nice, and cover the vast amount of eventualities. If you're unsure which to choose, go with the one that has more documentation for your use case. DevOps is not cloak-and-dagger mysticism; having control and understanding of your stack is, imho, far more important than choosing between a distro to run it on. Just make sure your server is locked down and that you regularly apply updates (that you've tested on your test server).
Use something you're familiar with, and that you have a good source of information flow, by which I mean, subscribe to security updates (including for whatever stack you put on your server), and certainly have a test server simply for playing with things, getting used to functionality.
I have a shell app on my phone, which I love, so I can log in at a moment's notice, wherever I am, just to check things out or apply a change.
The Linode and DO tutorials are pretty nice, and cover the vast amount of eventualities. If you're unsure which to choose, go with the one that has more documentation for your use case. DevOps is not cloak-and-dagger mysticism; having control and understanding of your stack is, imho, far more important than choosing between a distro to run it on. Just make sure your server is locked down and that you regularly apply updates (that you've tested on your test server).
I don't mean to offend you, but I guess offense well taken. If you are asking this question, it's really important you hire someone with really good system knowledge and security knowledge before you launch your SaaS production, besides having someone doing a proper security testing. If you are running on AWS then make sure you have a proper IAM policy and proper security group settings. If you are running applications then containers are probably a nice addition. Basic WAF is essential. Even enable fail2ban even if your servers are in private subnet. No password-based SSH. UPDATE UPDATE UPDATE. These will get you enough before someone finds a remote shell exploit take over your server or someone leaks secret key in github repo or before you get XSS pwned.
I disagree, unless your SaaS is storing highly sensitive data.
If you're in the bootstrap phase and don't have a lot of employees, then yes, I agree with you.
But as soon as you can afford it and/or as soon as you get big enough, you should have at least one FTE who focuses purely on security. This applies to any company with an Internet presence.
But as soon as you can afford it and/or as soon as you get big enough, you should have at least one FTE who focuses purely on security. This applies to any company with an Internet presence.
I wouldn't use a SaaS if I wasn't convinced you considered everything about me highly sensitive.
I respect your opinion and clearly the person asking the question is trying to learn how to secure the server. I do not feel like they need to hire someone to handle the security for them. However there most certainly is a difference between storing some web analytics and personal finance information.
Sorry, but if your server is hacked, you lose trust, even if all you do is keeping my five years old cousin's drawing. deb.haskell.org didn't store anything sensitive, but as a server storing the .deb it was hacked and has impact on possibly ten of thousands of users. Just an example.
You are assuming that because the poster asked this question that they can't figure out how to keep their sever secure. The person is clearly making an effort to learn how to secure a server and they came to a good place to start learning. I still do not feel they should hire someone because you came up with an example where trusted binaries were being hosted got hacked. It depends on the data. If you follow best practices that can be learned in a afternoon, you can keep a server secure without hiring someone. You have to weigh what the worst case scenario hack would cost based on your data and decide if it is worth the money saved by doing it yourself. In my opinion, for many new SaaS bootstrap products, it is totally worth doing the security yourself.
At least now the poster has multiple perspectives on the matter.
At least now the poster has multiple perspectives on the matter.
> If you follow best practices that can be learned in a afternoon, you can keep a server secure without hiring someone.
Yes and no. Running a business is different. Sure it may be too early to hire someone right now, but not the worst thing to think of. If OP can secure funding, get someone who has good knowledge with system and security. It's important. I am jack-of-all-trade to many, but as a devops I can't do everything at once.
> storing some web analytics and personal finance information
Fair. My company's finance data would be under compliance regulation, but even as a SaaS to provide easy-access easy-to-understand web analytics means you are going to have access to customer's data, or the customer is passing you data that the customer doesn't wish to share with others. Take an analytic SaaS to see your AWS usage. That requires a separate READ-ONLY access to your AWS account.
Twitter recently dropped employee access to internal data. People suggest this was done to avoid insider trading. I don't know if that was the real intent, but let's take that as a yes, then web analytic is just sensitive as one's financial information.
Security is not easy, just like writing code is not easy. I am suggesting OP to do some security audit before the launch. If OP ever hire someone to join the startup, try to secure a seat for someone good at security. FYI, I did some whitehat for some startup and had found numerous serious security bugs (no I am not here for hire, no time and definitely NOT as good as others, and I am 100% happy with my current job...). Just my genuine advice: don't just focus on features :)
Yes and no. Running a business is different. Sure it may be too early to hire someone right now, but not the worst thing to think of. If OP can secure funding, get someone who has good knowledge with system and security. It's important. I am jack-of-all-trade to many, but as a devops I can't do everything at once.
> storing some web analytics and personal finance information
Fair. My company's finance data would be under compliance regulation, but even as a SaaS to provide easy-access easy-to-understand web analytics means you are going to have access to customer's data, or the customer is passing you data that the customer doesn't wish to share with others. Take an analytic SaaS to see your AWS usage. That requires a separate READ-ONLY access to your AWS account.
Twitter recently dropped employee access to internal data. People suggest this was done to avoid insider trading. I don't know if that was the real intent, but let's take that as a yes, then web analytic is just sensitive as one's financial information.
Security is not easy, just like writing code is not easy. I am suggesting OP to do some security audit before the launch. If OP ever hire someone to join the startup, try to secure a seat for someone good at security. FYI, I did some whitehat for some startup and had found numerous serious security bugs (no I am not here for hire, no time and definitely NOT as good as others, and I am 100% happy with my current job...). Just my genuine advice: don't just focus on features :)
If you are not using someone else's proven platform, then yours will be hacked, possibly several times, at least until you've learned how to secure it and have kept it secure for some time.
With that in mind, you must, must, must do a few things:
1. Get really, really good at rollback and recovery. Make sure you can completely rebuild your operation from a clean system in as short a time as possible, with no data loss. Certain types of compromise will require this, it will be the cleanest, safest way to recover.
2. #1 requires you to be really good at backups. Unless you have experience here, they're harder than you think.
3. Notice that #1 starts with "rollback": You will occasionally be your own worst enemy, at least until your release processes get really, really good. Every now and again you will apply something that breaks something else. Being able to get back to a known good state quickly and easily will save you countless hours and much stress.
4. Find an experienced penetration tester that you trust implicitly. Have them review your operational architecture; they will find flaws (that's just on paper). Have them review your hardware and software configurations; they will find flaws. Listen to their proposals for risk mitigation. Once you have a system that passes their paper attack, have them actually kick at it, from three different places: one, from the outside, from the Internet everyone comes from; two, from just inside, from within your first tier, between outside interfaces to the world and app logic; three, from deep within, behind the app tier (using test data, of course). The three reports should be very different, with increasing levels of access and vulnerability as they go (because you are letting them past the mitigations). This will give you considerable information about how at risk you are.
(I know quite a good one, does this sort of thing professionally for government, private sector, the military, police forces, etc. He'll break your stuff. And tell you how to make it better.)
5. #4 implies, strongly implies, that you have a restricted access test environment that matches your operational environment as closely as possible, with two exceptions: one, no production data; two, it may be a patch or two or a major update ahead of production, since it's where you will soak changes to make sure they work properly and break nothing.
6. Managing #5, #3, and #1 means you must, must, must understand and get good at change management. And risk assessment and risk management.
IT Security is mostly about governance and management. If you focus on technology first, you've already lost.
With that in mind, you must, must, must do a few things:
1. Get really, really good at rollback and recovery. Make sure you can completely rebuild your operation from a clean system in as short a time as possible, with no data loss. Certain types of compromise will require this, it will be the cleanest, safest way to recover.
2. #1 requires you to be really good at backups. Unless you have experience here, they're harder than you think.
3. Notice that #1 starts with "rollback": You will occasionally be your own worst enemy, at least until your release processes get really, really good. Every now and again you will apply something that breaks something else. Being able to get back to a known good state quickly and easily will save you countless hours and much stress.
4. Find an experienced penetration tester that you trust implicitly. Have them review your operational architecture; they will find flaws (that's just on paper). Have them review your hardware and software configurations; they will find flaws. Listen to their proposals for risk mitigation. Once you have a system that passes their paper attack, have them actually kick at it, from three different places: one, from the outside, from the Internet everyone comes from; two, from just inside, from within your first tier, between outside interfaces to the world and app logic; three, from deep within, behind the app tier (using test data, of course). The three reports should be very different, with increasing levels of access and vulnerability as they go (because you are letting them past the mitigations). This will give you considerable information about how at risk you are.
(I know quite a good one, does this sort of thing professionally for government, private sector, the military, police forces, etc. He'll break your stuff. And tell you how to make it better.)
5. #4 implies, strongly implies, that you have a restricted access test environment that matches your operational environment as closely as possible, with two exceptions: one, no production data; two, it may be a patch or two or a major update ahead of production, since it's where you will soak changes to make sure they work properly and break nothing.
6. Managing #5, #3, and #1 means you must, must, must understand and get good at change management. And risk assessment and risk management.
IT Security is mostly about governance and management. If you focus on technology first, you've already lost.
Don't go for analysis paralysis
What is your exit strategy? What will your buyer value? What is best for your product? What do your engineers understand?
Does it make any difference from a security perspective if I use Linux or FreeBSD or something else?