Before we start, don’t be misled by the title of this chapter. I am not going to teach you about plumbing here. We are going to cover piping and redirection in Linux.
On a more serious note, if you have read and understood all previous chapters, then at this stage, you are no longer a beginner. You should by now be somewhat comfortable around a Linux system.
That being said, you will embark now on the second part of your journey.
From this chapter on, you will finally start to grasp the true power of Linux. You will be able to automate your workflows, manipulate text, extract and process data, and much more. All of this wouldn’t have been possible without the use of Redirection and Piping.
Before we discuss piping and redirections, we first need to understand a bit of theory about something called data streams.
In Linux, a data stream is the flow of data from a source to a destination. This can either be from one process to another, in which case that would be through a pipe; or, from a file to a process (or the other way around) which would be through a redirect.
Another way of looking at data streams is by thinking of them as links that join two ends. On one end, you may have your command output; and on the other end, you can either have the input to another command or to a file.
As a Linux user, you should distinguish between three standard data streams that are connected by default to every Linux program:
- Stdin (Standard Input, identified by the value 0): This is the input that the program receives.
- Stdout (Standard Output, identified by the value 1): This is the output of the program (On a terminal, whatever goes in stdout is by default displayed on the screen).
- Stderr (Standard Error, identified by the value 2): This is where the program sends its error messages.
Here is a simple figure to better illustrate this:
This is all the theory that we need to cover for now. In the rest of this chapter, we will learn how to manipulate these streams which will allow us to change their default terminations.
I promise you, this won’t be as boring as it sounds. Once complete this chapter, you will finally be able to appreciate the power of Linux (That is,… of course, If you haven’t already).
Piping is connecting the standard output (stdout) of a command to the standard input (stdin) of another command. We can do this using the pipe operator (|).
The pipe operator takes the output of the command on its left and sends it as an input to the command on its right.
Let’s say I run the following command :
$ ls -l
This will result in the following output :
No surprise there. I got a detailed list of the files that are in my current directory.
Now let’s apply what we’ve learned in this section. I will add a pipe operator followed by the head command (If you recall from our previous chapters, the head command outputs only the first 10 lines of a file),
$ ls -l | head
Well now, we get a different result. As you can see in the image below, the output includes only the 10 first lines on the terminal.
So what happened here is that the command on the left of the pipe (ls -l) sent its output to the command on the right (head) instead of displaying it directly on the screen. Then, after getting this output and treating it as any other input file, the head command only displays its first 10 lines.
We can take this even one step further and use a series of pipes to chain multiple commands.
Building on our previous example, let’s say that I want to run a command that outputs only the line corresponding to file03.
We know how to display the first n lines using the head command.
$ ls -l | head -n 5
So, if we run the above command, we should get the following result.
Now if we add another pipe, piping this result into the tail command, then we can select the last line using the ‘-n’ flag.
$ ls -l | head -n 5 | tail -n 1
And this, right here, is where all the magic of Linux lies.
Note that we could have done this using grep, which is another powerful Linux command. However, since it wasn’t yet covered in this tutorial, it would be better to ignore it for the time being. But don’t worry, we’ll cover it soon enough in a future chapter.
Writing output to a file
So far, whenever we execute a command, we see its output on the screen. This means that its stdout is connected to the screen terminal. However, this doesn’t have to be always the case.
We can redirect this output to a file by using the arrow pointing to the right (>) symbol, followed by the location of the file where we want to write the output.
Here is an example.
$ ls -l > output.txt
Although running the above command won’t display anything on the screen, this doesn’t mean that nothing happened. In fact, ls -l was executed, and then its output was stored in the file “output.txt“.
If you read the content of “output.txt”, you can see the output of our command.
Note that if the file “output.txt” exists already, then redirecting to it will overwrite its previous content. If you want to keep its previous content and store your output at the end of the file, then you should use two arrows instead of one.
$ ls -l >> output.txt
Reading from a file
You can read from a file and redirect its content to the input of a command using an arrow symbol pointing to the left (<), followed by the path to the file.
To illustrate this, I created a file called input.txt with the following content.
123 420 857 432 122
Now, we will use the sort command to, obviously, sort the above numbers by reading them directly from the above file.
We will cover “sort” in more detail in a future chapter. I brought it up here only as an example of how a command can take its input from a file.
Before we close this chapter, we need to address the last of the three data streams that I introduced in the first section of this chapter.
If you remember, we said that programs have a separate stream where they send their errors. This means that we can process errors separately from the standard output.
This can be useful for certain commands that raise a lot of errors.
For example, if you remember from the last chapter, the “find” command can be used to search for a file in a specified path. However, this command can sometimes display a lot of errors when it tries to search in a directory that it cannot access due to a lack of permissions.
In the image below, I tried to search for the file ‘patchthenet’.
As you can see, the “find” command shows only the error messages. So, even if it did find the file that I was looking for, I wouldn’t be able to spot it among the sea of errors thrown at me.
So, one way to solve this problem is to send the stderr stream to a file so it does not appear on the screen.
The value of 2 refers to the stderr stream. So, here I am sending all the errors to a file called errors.txt, which leaves the terminal clean to display the desired output.
The only problem here, however, is that I ended up with the file ‘errors.txt’ which I would now have to remove.
Thankfully, I don’t have to create a new file to send my errors. Linux has a solution for this. There is an interesting file where you can send any type of output that you don’t need. This special file is located at /dev/null. Everything that you send there vanishes. Think of it like a black hole. Anything that goes in, gets destroyed.
Now, the final command should look like this:
And that’s it. Problem solved.
This wraps up the piping and redirection chapter. I recommend that you take some time to become comfortable with what we have covered so far. Once you feel ready, I’ll meet you in the next chapter where we start discussing how to extract and process data in Linux.