Ask HN: What do you need to know to be good at the Unix Command Line?
9 comments
IMO: These days, it's hard to see a reason for learning this stuff by rote memorization when ChatGPT is already so, so good at it. You just give it a sample few lines from your text file and tell it what you need the output as, and it'll break down every command and parameter for you, whether that's a complex regex or a bunch of piped commands.
Man pages in (in its training), results out, all expressed in natural language. Seems more useful than an old-school curriculum, TBH...
Man pages in (in its training), results out, all expressed in natural language. Seems more useful than an old-school curriculum, TBH...
Yup, replace learning the syntax of tools with learning how to ask ChatGPT what a command is in English. I've found sometimes it will take a few tries for it to a working answer. I don't care if its sed or awk or grep or whatever as long as I accomplish the goal of what I want to be done.
i hope you realize that there is no intelligence at work, and all that this is, is just advanced interpolation of whatever is in the model
it doesn't actually understand what you are asking it.
it doesn't actually understand what you are asking it.
Even if it's a glorified autocomplete, it suits this use case perfectly. It's far better than trying to read poorly written man pages directly. It can digest documentation and parse it into natural language.
Most of these pipes / scripts are for one off jobs here and there, and there's no reason to memorize every random parameter across a bunch of them. ChatGpt is like a smart reference that's also really good at understanding English. It should probably just be built into the terminal at this point (but with a dry run for any commands).
Most of these pipes / scripts are for one off jobs here and there, and there's no reason to memorize every random parameter across a bunch of them. ChatGpt is like a smart reference that's also really good at understanding English. It should probably just be built into the terminal at this point (but with a dry run for any commands).
Manipulating text files is probably not enough. This is what I usually teach:
1) learn the parts of the command including long and short versions of options(command, options arguments) 2) learn how to execute commands and troubleshoot (which, ls -al, chmod, etc) 3) learn to chain commands with the pipe (grep, sort, awk, sed, wc and grep -c, learn the common misuse of cat since most commands you’re passing to take a filename argument) 4) learn about history, reverse history search, history shortcuts like !! And !:1, tab completion
The way I do this is to have students set up a raspberry pi and ssh into it. There’s a lot of fiddling and troubleshooting involved and it builds good practice.
You should also probably teach command line tool access in vi for processing huge files visually e.g :%! grep
1) learn the parts of the command including long and short versions of options(command, options arguments) 2) learn how to execute commands and troubleshoot (which, ls -al, chmod, etc) 3) learn to chain commands with the pipe (grep, sort, awk, sed, wc and grep -c, learn the common misuse of cat since most commands you’re passing to take a filename argument) 4) learn about history, reverse history search, history shortcuts like !! And !:1, tab completion
The way I do this is to have students set up a raspberry pi and ssh into it. There’s a lot of fiddling and troubleshooting involved and it builds good practice.
You should also probably teach command line tool access in vi for processing huge files visually e.g :%! grep
>They clone a git repository
I think this is a bad starting point. To learn Unix, first learn git. Sure, it’s one command for them to type that can be given to them, but I know a lot of people who are scared of anything git related and it would create a barrier to entry, especially for those who have no other reason to know much about git (like every non-dev).
This also creates a chicken and egg situation. I need to clone a git repo using the command line to start using the command line. Without already knowing commands like ls, cd, and pwd, I may not clone it to the right place, or be able to find it once it’s cloned. I’ve seen this countless times, even from people working in the tech industry.
> 1. I know a lot of people have already tried to do this. Can you point to any existing learning tools that have really knocked it out of the park?
vimtutor is decent, as a hands-on way to learn vim. Though, few people seem to have the patience to actually go through it. Learning the command line is self-selecting in a way. Those who has the willingness to use a lot of the current materials are the ones who will have the willingness to use it day to day.
Learning how to read the man pages, more than just the command to type, should be an early required part of learning, but it often skipped. There is a lot to be said for becoming good at using the reference material that will always be with you, and not having to hope someone wrote a blog post about the exact issue a person is having. Making something to help users learn that really well, will pay dividends.
I think this is a bad starting point. To learn Unix, first learn git. Sure, it’s one command for them to type that can be given to them, but I know a lot of people who are scared of anything git related and it would create a barrier to entry, especially for those who have no other reason to know much about git (like every non-dev).
This also creates a chicken and egg situation. I need to clone a git repo using the command line to start using the command line. Without already knowing commands like ls, cd, and pwd, I may not clone it to the right place, or be able to find it once it’s cloned. I’ve seen this countless times, even from people working in the tech industry.
> 1. I know a lot of people have already tried to do this. Can you point to any existing learning tools that have really knocked it out of the park?
vimtutor is decent, as a hands-on way to learn vim. Though, few people seem to have the patience to actually go through it. Learning the command line is self-selecting in a way. Those who has the willingness to use a lot of the current materials are the ones who will have the willingness to use it day to day.
Learning how to read the man pages, more than just the command to type, should be an early required part of learning, but it often skipped. There is a lot to be said for becoming good at using the reference material that will always be with you, and not having to hope someone wrote a blog post about the exact issue a person is having. Making something to help users learn that really well, will pay dividends.
> 2. What text processing tasks do you consider absolutely essential to Unix command line proficiency?
Learn regular expressions (REs). Every text processing tool including decent editors uses REs. And, you will never have mastered any of them without having mastered REs first.
Learn regular expressions (REs). Every text processing tool including decent editors uses REs. And, you will never have mastered any of them without having mastered REs first.
http://man.openbsd.org/sh
That's all you need to read to learn the shell. Learning about the commands requires actual hands-on work with a system, because the commands vary between systems and which ones you use depends on what you're doing. A general "how to be good at *nix command line" class will always be boring rote memorization, because there aren't really any concepts or skills to learn, just details.
That's all you need to read to learn the shell. Learning about the commands requires actual hands-on work with a system, because the commands vary between systems and which ones you use depends on what you're doing. A general "how to be good at *nix command line" class will always be boring rote memorization, because there aren't really any concepts or skills to learn, just details.
You need to use it as your primary OS and learn by doing.
Start on the basis that you shouldnt have to download something to do what you need, it all comes pre-installed and go from there.
Need to process some text files? Use awk & sed etc Need to partition and format a drive? use fdisk, parted etc Need to encrypt a drive / folder - use LUKS
Everything you need is on the system itself including the files to run tests on, the rest of it will come with normal usage.
Start on the basis that you shouldnt have to download something to do what you need, it all comes pre-installed and go from there.
Need to process some text files? Use awk & sed etc Need to partition and format a drive? use fdisk, parted etc Need to encrypt a drive / folder - use LUKS
Everything you need is on the system itself including the files to run tests on, the rest of it will come with normal usage.
One extreme is to implement a simple shell with some Unix tools by yourself. I used to do that for ls and thus remembered many of its options -- but of course such knowledge does not retain well.
> Ask HN: What do you need to know to be good at the Unix Command Line?
man.
man.
Unix seems hard to learn because it started as a tool for experts, people who already understood the underlying concepts and had experience with other operating systems. Someone coming at Unix without some understanding of the fundamentals will struggle with it.
Unix doesn't refer (anymore) to a single operating system or collection of tools. You probably mean Linux, which comes in many flavors (distributions), or maybe to MacOS. No one company or organization makes Unix, and while the fundamentals and basic concepts hold across the Unix-derived operating systems the details -- things like command line arguments and directory structure -- can vary a great deal.
You conflate the core operating system with "user space" tools such as awk and sed, abstractions such as regular expressions, and vendor-specific things such as package management. We can lump those together and call them "Unix" but only in the same way we can refer to a generic "car" as a thing with an engine and four wheels.
I have taught several people how to use Unix/Linux/MacOS to some degree, to accomplish some specific tasks. I have seen them struggle with basic concepts such as directories and file organization, permissions, the whole idea of executable scripts, "piping" and redirecting input and output ("everything is a file" doesn't mean much to most people). When it comes to processes, networking, inter-process communication, different kinds of text files (ASCII vs. Unixcode/UTF-8, for example), etc. most people just get lost or lose interest because they don't have any mental categories or real-world analogies to connect to. Anyone approaching the problem of "learning Unix" today probably has experience using computers, but they really have experience using graphical user interfaces that hide the details and application programs such as Word or GMail that offer task-oriented user interfaces with none of the underlying fundamentals exposed. Learning the concepts that underly Unix, or any operating system, requires going down some rabbit holes to understand things like text file, directories, processes, and so on. Few people will have the patience and motivation to do that work, to read technical books on operating system fundamentals and design.
If I showed someone how to use grep to search text files for a string they might get that after a while, but they probably don't really understand text files or regular expressions -- they have just memorized the steps to complete a specific task. If I then tried to explain how to search a CSV or XML file that new information might not connect, and the person has simply learned how to do two or three very specific tasks without gaining much fundamental expertise. That's like someone memorizing a few recipes from a cookbook but not actually learning much about ingredients or cooking.
What you propose seems like trying to learn top-down, by practicing specific tasks such as text processing (largely incidental to the operating system). Some people can learn that way -- working through enough tasks with repetition will let them deduce the abstractions and encourage them to explore on their own. Most people will fail because years of schooling have discouraged that kind of thinking, making them goal-oriented and focused on completing a specific task by memorizing specific steps rather than actually learning. And many people don't have the motivation and desire to learn how regular expressions work, or what a "process" means in the context of Unix.
No amount of instructional design will help everyone learn a complex topic. Analogously, you can't get through medical school by practicing a few procedures over and over. You have to learn some biology and anatomy so the many things you have to study and remember make sense and hang together.
There's no way I know of to even have this discussion without coming across as as elitist. I learned Unix (and C) from the Bell Labs Technical Journal articles published back in the '70s because I had an obsessive desire to do that, and the spare time and energy of a teenage boy bored with high school. I put a lot of time in, sought out mentors, practiced, explored, studied. Everyone I know who has mastered a complex skill or set of tools tells the same story. My experience was more like learning a language by immersion than with flashcards presented in an app. I don't think most people incapable of learning Unix or any other technical skill, but I do think few people have the motivation, time, and persistence to do it, and instead want some easy way to get information spoon-fed to them, a process they already understand as "learning" because schools mostly work that way.
Good luck with your project.
Unix doesn't refer (anymore) to a single operating system or collection of tools. You probably mean Linux, which comes in many flavors (distributions), or maybe to MacOS. No one company or organization makes Unix, and while the fundamentals and basic concepts hold across the Unix-derived operating systems the details -- things like command line arguments and directory structure -- can vary a great deal.
You conflate the core operating system with "user space" tools such as awk and sed, abstractions such as regular expressions, and vendor-specific things such as package management. We can lump those together and call them "Unix" but only in the same way we can refer to a generic "car" as a thing with an engine and four wheels.
I have taught several people how to use Unix/Linux/MacOS to some degree, to accomplish some specific tasks. I have seen them struggle with basic concepts such as directories and file organization, permissions, the whole idea of executable scripts, "piping" and redirecting input and output ("everything is a file" doesn't mean much to most people). When it comes to processes, networking, inter-process communication, different kinds of text files (ASCII vs. Unixcode/UTF-8, for example), etc. most people just get lost or lose interest because they don't have any mental categories or real-world analogies to connect to. Anyone approaching the problem of "learning Unix" today probably has experience using computers, but they really have experience using graphical user interfaces that hide the details and application programs such as Word or GMail that offer task-oriented user interfaces with none of the underlying fundamentals exposed. Learning the concepts that underly Unix, or any operating system, requires going down some rabbit holes to understand things like text file, directories, processes, and so on. Few people will have the patience and motivation to do that work, to read technical books on operating system fundamentals and design.
If I showed someone how to use grep to search text files for a string they might get that after a while, but they probably don't really understand text files or regular expressions -- they have just memorized the steps to complete a specific task. If I then tried to explain how to search a CSV or XML file that new information might not connect, and the person has simply learned how to do two or three very specific tasks without gaining much fundamental expertise. That's like someone memorizing a few recipes from a cookbook but not actually learning much about ingredients or cooking.
What you propose seems like trying to learn top-down, by practicing specific tasks such as text processing (largely incidental to the operating system). Some people can learn that way -- working through enough tasks with repetition will let them deduce the abstractions and encourage them to explore on their own. Most people will fail because years of schooling have discouraged that kind of thinking, making them goal-oriented and focused on completing a specific task by memorizing specific steps rather than actually learning. And many people don't have the motivation and desire to learn how regular expressions work, or what a "process" means in the context of Unix.
No amount of instructional design will help everyone learn a complex topic. Analogously, you can't get through medical school by practicing a few procedures over and over. You have to learn some biology and anatomy so the many things you have to study and remember make sense and hang together.
There's no way I know of to even have this discussion without coming across as as elitist. I learned Unix (and C) from the Bell Labs Technical Journal articles published back in the '70s because I had an obsessive desire to do that, and the spare time and energy of a teenage boy bored with high school. I put a lot of time in, sought out mentors, practiced, explored, studied. Everyone I know who has mastered a complex skill or set of tools tells the same story. My experience was more like learning a language by immersion than with flashcards presented in an app. I don't think most people incapable of learning Unix or any other technical skill, but I do think few people have the motivation, time, and persistence to do it, and instead want some easy way to get information spoon-fed to them, a process they already understand as "learning" because schools mostly work that way.
Good luck with your project.
As a teacher and cognitive scientist, the reason is interesting: Unix documentation, starting with its man pages, is presented in a format that makes it easy for experts to look up details, not for beginners to learn concepts or build skills. O'Reilly, Sam's, APress and others have built large businesses selling second-best solutions to this problem.
This is a teaching anti-pattern that emerges when domain experts -- but teaching novices -- try to write a curriculum from scratch. They don't realize that teaching is a separate skill from the material to be taught, and that a great deal of effort is needed to break a domain down into right-sized, self-sufficient chunks for the reader to consume. Instead, they produce big volumes of unorganized knowledge (man pages) that reflects their expert-level understanding, that the reader tries to sift through and, as often as not, gives up on in frustration. Even when you pay experts to write beginner-friendly books, you get separate books on bash, awk, sed, grep, regexes, files, processes, packages and so on. These put the onus on the reader to remember all the details and put them all together into something useful.
Having dumped all this information on you, I always get the feeling that the gurus are patting you on the head and saying, "Well, learning is hard. You have to work at it. Be sure you practice enough and you'll figure it all out eventually." Learning is not supposed to be easy, and you do have to practice, but too often that attitude serves as an excuse to stop looking for better teaching methods.
So I'd like to propose one here.
The interactive piece takes place on the user's machine. They clone a git repository that contains lots of text files that they can manipulate with complex commands, according to lessons and exercises delivered via a web app. The web app helps the user remember using a technique I call graduated spaced repetition, which extends the basic SuperMemo algorithm to test concepts, not just facts, with progressively more elaborate questions.
All of this should be relatively simple to write. The main differentiator will be good instructional design which -- sorry if I sound like a jerk here -- I have not seen much in evidence in the 20 years I've spent learning this material.
Which brings me to my questions for you:
1. I know a lot of people have already tried to do this. Can you point to any existing learning tools that have really knocked it out of the park?
2. What text processing tasks do you consider absolutely essential to Unix command line proficiency?
3. What kinds of text files (Natural language text, CSV, logfiles, etc) would it be useful to practice on to get there? Where can I find some?
If you want to talk about building the training set or the extensions I've made to the SuperMemo protocol, or if you just want to spitball ideas, you can email me at [email protected]