New Year's Resolutions for SysAdmins(usenix.org)
usenix.org
New Year's Resolutions for SysAdmins
https://www.usenix.org/blog/new-years-resolutions-sysadmins
7 comments
I should "Finally learn IPv6" but I know that I will procrastinate again and again..
"Check that your backups are working the way you think they are."
I'm a bit horrified to read this here. If you're a sysadmin and don't have both automated and manual testing of backups, it's hard to imagine what else was a more important use of your time.
There aren't many things that could bankrupt a healthy business overnight, but catastrophic data loss is certainly one of them.
An analogous entry for a lawyer might be to pay attention while reading contracts.
I'm a bit horrified to read this here. If you're a sysadmin and don't have both automated and manual testing of backups, it's hard to imagine what else was a more important use of your time.
There aren't many things that could bankrupt a healthy business overnight, but catastrophic data loss is certainly one of them.
An analogous entry for a lawyer might be to pay attention while reading contracts.
> what else was a more important use of your time
Keeping shit running now is always more important than keeping shit running in case of catastrophic failure.
Because a catastrophic failure is only catastrophic if you actually have something of value in the first place.
In the real world it's a balancing act in which there is no room for absolutism. Choosing which compromises to make is the hardest part of any job that comes with a level of responsibility.
Keeping shit running now is always more important than keeping shit running in case of catastrophic failure.
Because a catastrophic failure is only catastrophic if you actually have something of value in the first place.
In the real world it's a balancing act in which there is no room for absolutism. Choosing which compromises to make is the hardest part of any job that comes with a level of responsibility.
I'm talking specifically about dedicated sys admins, since that's what the blog post is about; I'm no stranger to cowboy coding on a project that may or may not ever be worth anything. I've lost data once before to corrupted backups and I don't regret not investing more engineering effort. It was an MVP, and making better products gives me more leverage.
But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.
But by the time you're hiring full time for the position, outside of a few edge cases where maybe you're SnapChat and you are growing 20% a week, it's probably time to settle down a bit and be sensible. At which point, if testing your backups to completion to avoid catastrophic data loss isn't #1 on the todo list, it's #2.
It may be #2 perpetually in many cases because the boss will not listen when you insist it's important. For many it then starts slipping down the list, as what is best for the company is often not best for the employee: For many it becomes a reasonable (for them personally) risk to take to bet that they'll do better from keeping the boss satisfied now rather than spend time on backups to avoid a major disaster after they've left. And yes, that means gambling that the major disaster won't hit while you're still there.
I'm not saying this is how it should be, but it is how it often becomes if the sysadmin or whomever taking on those responsibilities don't report to someone who also see the data integrity as priority #1 for the sysadmin.
I've worked in places where the CEO's e-mail client configuration is the #1 priority for the guy that should have been focusing on server backups, for example, and where prioritising the backups would be a bad career move for the person in question.
(Yes, that is a huge warning sign that it's best to find a different job)
I'm not saying this is how it should be, but it is how it often becomes if the sysadmin or whomever taking on those responsibilities don't report to someone who also see the data integrity as priority #1 for the sysadmin.
I've worked in places where the CEO's e-mail client configuration is the #1 priority for the guy that should have been focusing on server backups, for example, and where prioritising the backups would be a bad career move for the person in question.
(Yes, that is a huge warning sign that it's best to find a different job)
Indeed.
Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions.
I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB.
Took them 2 weeks to get it to me.
Crazy place, quit after 2 months. Really thought I'd asked the right questions too.
Last place I worked with sysadmins, they were fire fighting like crazy, not enough of them, bad choice of suppliers, poor code bases meant things fell over and took far to long to run and a dev team begging to get off the internal network on to their own because of all the restrictions.
I needed a backup of the DB of one of the three systems to test a major setting change that no-one was 100% sure would work because no-one understood the bloody awful contractor code with the even more awfully designed DB.
Took them 2 weeks to get it to me.
Crazy place, quit after 2 months. Really thought I'd asked the right questions too.
Excuse my ignorance, but what's wrong with nslookup and ifconfig?
short answer: `/bin/ip` replaces `ifconfig` -- it's newer and more powerful
more info: http://www.tty1.net/blog/2010/ifconfig-ip-comparison_en.html
more info: http://www.tty1.net/blog/2010/ifconfig-ip-comparison_en.html
Asides from ifconfig not being maintained (which is reason enough not to use it), I always wondered specifically what was broken with it.
Then I worked for an arbitrage desk at an investment bank. They used virtual addresses for different IPs, on top of vlans connected to different exchanges, redundantly (ie bonded).
Not a single IP-having interface appeared using ifconfig.
Then I worked for an arbitrage desk at an investment bank. They used virtual addresses for different IPs, on top of vlans connected to different exchanges, redundantly (ie bonded).
Not a single IP-having interface appeared using ifconfig.
OT but ip is a really unfortunate name that will be hard to google for.
They're a bit dated. Use dig instead of nslookup, and right from the ifconfig man page:
NOTE
This program is obsolete! For replacement check ip addr and ip link. For statistics use ip -s link.
NOTE
This program is obsolete! For replacement check ip addr and ip link. For statistics use ip -s link.
Where does it say that?
http://www.openbsd.org/cgi-bin/man.cgi?query=ifconfig
Pedantry, but keep in mind that there isn't really "the" man page for this.
Ubuntu might want to deprecate ifconfig, but that doesn't mean everyone does.
http://www.openbsd.org/cgi-bin/man.cgi?query=ifconfig
Pedantry, but keep in mind that there isn't really "the" man page for this.
Ubuntu might want to deprecate ifconfig, but that doesn't mean everyone does.
You're right, and I use OpenBSD myself so I should have qualified it. The man page I quoted was CentOS in this case.
[deleted]
On nslookup, in particular, the program has some ambiguous behavior and can be problematic when trying to debug certain DNS issues. Dig was developed to be a reliable replacement that has predictable behavior at all times. There is a companion to dig called "host" that is a simple lookup, including looking up in-addr.arpa entries ("host 8.8.8.8"). In most cases, you will get more consistent results by using dig and host, which is why nslookup is deprecated.
They've been replaced with ipadm/dladm: http://omnios.omniti.com/wiki.php/GeneralAdministration
And dnsip/dnsipq/dnsmx/dnsname/dnstxt: http://cr.yp.to/djbdns/tools.html
And dnsip/dnsipq/dnsmx/dnsname/dnstxt: http://cr.yp.to/djbdns/tools.html
What? Your reply seems purposefully disingenuous and misleading. The ipadm and dladm commands seem to be specific to Solaris and derivatives. And I can not believe that you are seriously proposing that the new standard commands for doing DNS is to use DJB’s tools.
What the original post most likely referred to is to use ip instead of ifconfig, and to use dig instead of nslookup.
What the original post most likely referred to is to use ip instead of ifconfig, and to use dig instead of nslookup.
Calm down a bit. Of course my post was in jest (well intentioned I might add) - I was pointing out the TMTOWTDI nature of this entire debate. The goal here wasn't to instruct or mislead.
The Linux way isn't always the way the rest of Unix is going. In fact, most of our current OS's are a hodgepodge of different great things developed by different teams - for example, rsync from the Samba devs, sudo from OpenBSD, git from Linux, etc.
This is one of the great things about Unix is that it's a big sandbox to play in. The entire "lets replace init" movement from the last 10 years is pretty fascinating in it's own right. Spend some time looking at how the other crowds out there are doing things - you might find something interesting and helpful.
The Linux way isn't always the way the rest of Unix is going. In fact, most of our current OS's are a hodgepodge of different great things developed by different teams - for example, rsync from the Samba devs, sudo from OpenBSD, git from Linux, etc.
This is one of the great things about Unix is that it's a big sandbox to play in. The entire "lets replace init" movement from the last 10 years is pretty fascinating in it's own right. Spend some time looking at how the other crowds out there are doing things - you might find something interesting and helpful.
Nothing.
I'd also recommend any sysadmin to start looking into an automation framework like Chef, Puppet, Ansible, or any other of the myriad options.
I just started digging into chef with the intention of using with AWS opsworks and... holy complexity. There just doesn't seem to be any obvious entry point, as far as I can tell. I've spent two entire days searching and there doesn't seem to be anything in between "hello wordpress" and "read this 300+ page user manual". Can anyone recommend a hands on guide for setting up a multi-node stack with opsworks that isn't just plain vanilla templates? I was really impressed by Richard Seroter's videos on AWS at pluralsight, but they are more of a 30,000 foot overview and gloss over implementation details, for the most part (even at 10+ hours). The project I'm working on is mostly a pretty straightforward node service running on postgres, but it uses some custom services that need to scale horizontally, and host sensitive data that should probably be running in a VPC.
Edit: I'm also having a difficult time trying to figure out how to get up to speed with chef in the specific context of opsworks, since most of the chef-specific courseware I can find assumes I'm using a hosted chef server, and I'm not quite sure how that overlaps with the opsworks flavor. It's starting to feel like I'll be at this for another week before I can get even a simple stack running.
Edit: I'm also having a difficult time trying to figure out how to get up to speed with chef in the specific context of opsworks, since most of the chef-specific courseware I can find assumes I'm using a hosted chef server, and I'm not quite sure how that overlaps with the opsworks flavor. It's starting to feel like I'll be at this for another week before I can get even a simple stack running.
Try with ansible.
Agreed.
As well as the reasonably good ansible documentation, checkout the ansible-examples github repo. ( https://github.com/ansible/ansible-examples )
There's a few initial 'conventions' to figure out (roles, group vars, playbooks), but then it's so incredibly simple to get into.
My dad is playing with a raspberry pi at home, wanting to set up a web kiosk thing. I'd been doing something similar a few months ago for work, so I send him my ansible playbook. He could follow it as if it were a plain text checklist/tutorial, without needing ansible at all. If he wants to automate it, he can.
That, I think, is one of the benefits of ansible. It's not a lot more work than just writing an overview of what you need to do in the first place, but with the benefit of it being repeatable and automated.
As well as the reasonably good ansible documentation, checkout the ansible-examples github repo. ( https://github.com/ansible/ansible-examples )
There's a few initial 'conventions' to figure out (roles, group vars, playbooks), but then it's so incredibly simple to get into.
My dad is playing with a raspberry pi at home, wanting to set up a web kiosk thing. I'd been doing something similar a few months ago for work, so I send him my ansible playbook. He could follow it as if it were a plain text checklist/tutorial, without needing ansible at all. If he wants to automate it, he can.
That, I think, is one of the benefits of ansible. It's not a lot more work than just writing an overview of what you need to do in the first place, but with the benefit of it being repeatable and automated.
Not aimed at exactly this use case but the first two thirds of my book Reliably Deploying Rails Applications https://leanpub.com/deploying_rails_applications is heavily chef focussed.
It's intended for use with Chef Solo but potentially the early sections on writing custom chef cookbooks may be of use.
It's intended for use with Chef Solo but potentially the early sections on writing custom chef cookbooks may be of use.
Chef smells over engineered from the get go. Try salt stack, or ansible. I've settled with Salt, but I believe they are both equally good solutions. Puppet is also OK, even if we moved away from it (towards Salt) because of poor performance (on a deployment of about 100 physical servers plus the VMs running on top of them).
It's what always kept me away from these systems. So many of them are hopelessly over-engineered to the point where it's often easier to write custom solutions.
We just built a simple tool that generates Ansible playbooks for you. Try it out at https://cmify.com
Chef is a bad bet if you don't want unnecessary complexity - it's practically designed to be unnecessarily complex.
> start looking into an automation framework
Cannot stress this enough! There is a sea change happening and sysadmins run the risk of being swept away unless they learn about automation (puppet/chef) software and cloud infrastructure services (openstack/aws). My new years resolution is to release screencasts on these very subjects. I have already released one on Learning Puppet with Vagrant @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag... if anyone is interested.
Cannot stress this enough! There is a sea change happening and sysadmins run the risk of being swept away unless they learn about automation (puppet/chef) software and cloud infrastructure services (openstack/aws). My new years resolution is to release screencasts on these very subjects. I have already released one on Learning Puppet with Vagrant @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag... if anyone is interested.
I don't know if you're aware, but configuration management is not a new concept. CFEngine was started in '93. CFEngine2 which is popular among many sysadmins came out in '98. While configuration management has definitely had some improvements it's not THAT far from what we've been using for decades now.
Good stuff! Thanks!
Automation is a fine thing except for some of this damn "enterprise software" that not only doesn't automate real well[1], but actually cannot be installed on VMWare[2] or even go to a hot swap machine[3]. Sysadmins always love government required software and stuff bought before we got there.
1) luckily backup seem to be reasonably automation friendly
2) "We do not support virtualization of any kind"
3) tied to hardware on one machine - f'n brilliant.
1) luckily backup seem to be reasonably automation friendly
2) "We do not support virtualization of any kind"
3) tied to hardware on one machine - f'n brilliant.
You might find this interesting: http://www.slideshare.net/opscode/chef-vmwarewebinarslidesse...
I personally haven't tried building environments on VMWare as I run pretty much everything in AWS. However, this presentation looks promising.
I personally haven't tried building environments on VMWare as I run pretty much everything in AWS. However, this presentation looks promising.
Since the government vendor's software "cannot run" VMWare, I cannot quite use the results.
Absolutely. Automate relentlessly. As a bonus, the recipes/playbooks you develop serve as documentation for how things are configured.
I learned #2 very early on. Further, the specs I give for a project are the very minimum. If you aren't willing to do things correctly the first time, I'm not willing to be a client.
#2 is the most important in that list.