Yes, due to the first OpenSSH remote exploit. With some many servers to manage one of them was forgotten and hacked.
Our recovery was a bit complicated, because we didn't trust any of our boxes after that. So, we shut down the hacked box, mounted somewhere else, removed all the data and reinstalled. We reinstalled all of our other boxes too... Lots of work and lesson learned.
To secure our systems now we take multiple approaches:
-Different operating systems across our network. For example, our web server runs on OpenBSD and our database on Linux (RHEL). We also have Debian and FreeBSD.
-Reduce the functionality of each system and who can access it. Plus, never allow SSH from one critical system to another.
Added monitoring. We can't be 100% safe, so we started using:
I think it depends on how you phrase the question. If the question can be easily answer (or even have an answer) and is properly written, people will be more keen to answer back.
But things that are too subjective or easily find on google, you will probably not get many answers..
Our recovery was a bit complicated, because we didn't trust any of our boxes after that. So, we shut down the hacked box, mounted somewhere else, removed all the data and reinstalled. We reinstalled all of our other boxes too... Lots of work and lesson learned.
To secure our systems now we take multiple approaches:
-Different operating systems across our network. For example, our web server runs on OpenBSD and our database on Linux (RHEL). We also have Debian and FreeBSD.
-Reduce the functionality of each system and who can access it. Plus, never allow SSH from one critical system to another.
Added monitoring. We can't be 100% safe, so we started using:
-http://modsecurity.org to filter web attacks
-http://ossec.net to monitor logs and file changes
-SeLinux enabled on all our Linux systems
-systrace on all our BSD boxes
-http://sucuri.net to monitor our DNS, Whois and site.