Ask HN: Must have (Linux) sysadmin skills?
10 comments
* understand the basic directory structure (/, /boot/, /var, /usr, /opt, /etc)
* file management (commands: tail, head, cat, awk, sed)
* process management (commands: ps, top, kill)
* package management (yum, dpkg)
* one configuration manager (puppet, chef, ansible)
* basics of tcp/ip
network management (ifconfig/ip, netstata/ss, ping, trace)
addiontal commands if you want to be more than a junior (strace, lsof, iostat, vmstat)
I'd like to point out that some sysadmins are focused on linux internals while others focus on application in production. So, of course, the list may vary according to the position you're looking to be hired.
I'd like to point out that some sysadmins are focused on linux internals while others focus on application in production. So, of course, the list may vary according to the position you're looking to be hired.
Good list, I'd also say permissions. The number of problems I fix by understanding users/groups and assigning the appropriate permission (not `chmod 777 *`) is huge. It shows up in a variety of "Cannot access file" type messages or "File does not exist" application errors and people sometimes freak out because the file is there.
Also relavent to permissions:
https://people.redhat.com/duffy/selinux/selinux-coloring-boo...
https://people.redhat.com/duffy/selinux/selinux-coloring-boo...
Definitely. If something's not working in a Red Hat based distro, it's usually stopped by SElinux.
And the right answer is never "Disable SELinux"
> understand the basic directory structure...
I agree with this.
Anyone interested in command line hacking can find everything nelsonmarcus recommends (and a bit more) in this very readable book: The Linux Command Line A complete Introduction By Shotts https://www.amazon.com/Linux-Command-Line-Complete-Introduct...
I agree with this.
Anyone interested in command line hacking can find everything nelsonmarcus recommends (and a bit more) in this very readable book: The Linux Command Line A complete Introduction By Shotts https://www.amazon.com/Linux-Command-Line-Complete-Introduct...
Brian Ward's - How Linux Works is also very good https://www.nostarch.com/howlinuxworks2
There is a very big overlap in what good developer and sysadmin should know about Linux. There are a lot of tasks, which in one teams performed by developer and in other — by sysadmin. Random topics which IMHO more specific for sysadmin work:
1. Troubleshooting and performance analysis. There is excellent site about performance: http://www.brendangregg.com/linuxperf.html I don't know a good resource on troubleshooting, but some tools are the same.
2. Problem can be anywhere including network so the next topic is networks. There is excellent book TCP/IP Illustrated — old, but still relevant.
3. Sysadmin often spends a lot of time in a shell so it is good to know it very well (and common shell commands too).
1. Troubleshooting and performance analysis. There is excellent site about performance: http://www.brendangregg.com/linuxperf.html I don't know a good resource on troubleshooting, but some tools are the same.
2. Problem can be anywhere including network so the next topic is networks. There is excellent book TCP/IP Illustrated — old, but still relevant.
3. Sysadmin often spends a lot of time in a shell so it is good to know it very well (and common shell commands too).
> What are the most common things a sysadmin does daily
- reacting to monitoring alerts and critical messages in logs. First you need to understand what given alert/message mean, and then goes troubleshooting if root cause is not obvious - improving monitoring system settings/thresholds/metrics (if there no separate teem for this) - deployment (but it often performed by developers) - upgrading OS and 3rd party software (if 3rd party software not deployed alongside own code) - performance tuning - learning software used in production (both: 3rd and written in the company) - writing numerous config files and ensuring that all settings adequate to given environment/system - if system is not entirely in cloud - replacing broken hardware and parts (HDD), usually using "remote hands" in datacenter. - managing ACLs / credentials for different systems, e.t.c.
- reacting to monitoring alerts and critical messages in logs. First you need to understand what given alert/message mean, and then goes troubleshooting if root cause is not obvious - improving monitoring system settings/thresholds/metrics (if there no separate teem for this) - deployment (but it often performed by developers) - upgrading OS and 3rd party software (if 3rd party software not deployed alongside own code) - performance tuning - learning software used in production (both: 3rd and written in the company) - writing numerous config files and ensuring that all settings adequate to given environment/system - if system is not entirely in cloud - replacing broken hardware and parts (HDD), usually using "remote hands" in datacenter. - managing ACLs / credentials for different systems, e.t.c.
Well this one is not really a skill, but quite a useful guideline or precaution to follow - and consistently:
Make your actions reversible (as far as possible). Had learned it in the field, later also saw it mentioned in a sysadmin book.
I can't count the number of times I've seen colleagues just blindly charge ahead and do, for example, non-trivial surgery (a.k.a. edits) on important system config files (in a haphazard, lets-see-if-this-works manner, without so much as making a backup copy of the file), which sometimes resulted in screwing up the system further, sometimes irrevocably (because, you guessed it, no backups - or no current ones, anyway). And yes, on production systems. Got a few stories about that, may mention them some other time.
Make your actions reversible (as far as possible). Had learned it in the field, later also saw it mentioned in a sysadmin book.
I can't count the number of times I've seen colleagues just blindly charge ahead and do, for example, non-trivial surgery (a.k.a. edits) on important system config files (in a haphazard, lets-see-if-this-works manner, without so much as making a backup copy of the file), which sometimes resulted in screwing up the system further, sometimes irrevocably (because, you guessed it, no backups - or no current ones, anyway). And yes, on production systems. Got a few stories about that, may mention them some other time.
The skills listed on Red Hat's RHCSA [0] page might be a good start.
[0]: https://www.redhat.com/en/services/certification/rhcsa
[0]: https://www.redhat.com/en/services/certification/rhcsa
I love linux even though I am not an expert devops or sysadmin. I get around working with servers for my product and thats about it. I stackoverflow shell scripts whenever needed. Here is my 'must know' list for linux:
- ssh: you need to know what that means and how to use it
- Difference between ftp and sftp
- cd, ls, pwd (directory stuff)
- scp and rsync (move files from server to server)
- find, cat, grep, sed, awk, head, tail (manipulations and discovery)
- chown, chmod (permissions)
- cp, mv, rm, rmdir, mkdir, touch
- Don't do rm -rf unless you know what you are doing
- Setup aliases on your shell. e.g. .profile file for bash etc.
- top, ps (check processes)
- sudo and su (switching users or running commands as root)
- Know various OS differences like CentOS vs Ubunut.
- Know where to find logs for web servers like Apache or nginx (e.g. /var/log/nginx in Ubuntu).
- Star/Stop services as needed
- ssh: you need to know what that means and how to use it
- Difference between ftp and sftp
- cd, ls, pwd (directory stuff)
- scp and rsync (move files from server to server)
- find, cat, grep, sed, awk, head, tail (manipulations and discovery)
- chown, chmod (permissions)
- cp, mv, rm, rmdir, mkdir, touch
- Don't do rm -rf unless you know what you are doing
- Setup aliases on your shell. e.g. .profile file for bash etc.
- top, ps (check processes)
- sudo and su (switching users or running commands as root)
- Know various OS differences like CentOS vs Ubunut.
- Know where to find logs for web servers like Apache or nginx (e.g. /var/log/nginx in Ubuntu).
- Star/Stop services as needed
I want to do a blog post series on this topic.
Systems administration is a huge topic with so many interconnected parts and such vast variety of tools. Even breaking it down to segments can fall apart easily as nearly all topics overlap from user management, to filesystems, etc.
My only advice is this: Get a copy of "UNIX and Linux System Administration Handbook" and read the topics that you work on. It's a reference not something that you can read from cover to cover, but it's I own the 4th edition and it's depth is amazing. On 18 Aug there is the 5th edition coming out. Grab a copy.
My only issue is that I'd love to have *BSD included along Linux, Solaris and AIX (which is hard to find these days...).
Systems administration is a huge topic with so many interconnected parts and such vast variety of tools. Even breaking it down to segments can fall apart easily as nearly all topics overlap from user management, to filesystems, etc.
My only advice is this: Get a copy of "UNIX and Linux System Administration Handbook" and read the topics that you work on. It's a reference not something that you can read from cover to cover, but it's I own the 4th edition and it's depth is amazing. On 18 Aug there is the 5th edition coming out. Grab a copy.
My only issue is that I'd love to have *BSD included along Linux, Solaris and AIX (which is hard to find these days...).
Is that the Evi Nemeth et al. book? I didn't think they were doing another edition after she was lost at sea? I'll happily buy an updated version.
Yes! It's an amazing book for sysadmins. I bought my copy (4th edition) a few months ago. I'm still reading various bits and pieces.
The new book I'm sure will have a new section on virtualisation, automation (ansible, etc.), docker, docker orchestration tools, service discovery, etc.
I'll buy a copy on Christmas.
The new book I'm sure will have a new section on virtualisation, automation (ansible, etc.), docker, docker orchestration tools, service discovery, etc.
I'll buy a copy on Christmas.
While not a fan of most certs, I would highly recommend the RHCSA to get an understanding of common sysadmin tasks as it is a hands-on exam and fairly relevant to the real world, and RHCE as well. In addition you should find some random service in your language of choice on GitHub that requires a database and try to implement in a cloud provider taking into account high availability, deployment types (Blue/Green vs Rolling, for example), durability, backups, security, logging, and monitoring by rolling out the services yourself or through managed services at a given cloud provider using a CM like Ansible and/or Terraform.
[deleted]
Not sure why nobody mentioned it yet but iptables.
What would you list as the most essential skills one must be able to demonstrate to be hireable ? (Coming from a developer). What are the most common things a sysadmin does daily ?
Cheers