Background

Streams are a fundamental part of Linux command-line experience. The concept is inherited from Unix.

Following is the “The Unix philosophy” by Peter H. Salus:

  • Write programs that do one thing and do it well.
  • Write programs to work together.
  • Write programs to handle text streams, because that is a universal interface.

Let me provide some context.

Back when Unix was invented, computers had a minuscule amount of memory compared to today. A big machine had memory in the order of kilo-bytes. That put some serious limitation on how big a single program could be.

The solution was to have small programs that could easily be combined together for complex task. The method for combining programs is with pipes, hence the title.

Each program can read from an input stream and write to an output stream. You can connect the output stream of one program to the input stream of another with pipes, the | symbol.

The input stream is called stdin, output is stdout. There is also another output stream for errors called stderr. Where std is short for standard.

Here is an old demonstration of pipes where Brian Kernighan (Unix co-inventor) use very simple programs and pipes to build a spellchecker. Most of the commands used in the demo doesn’t exist anymore. However, you can check out this repository to see how the demo can be recreated using commands found on more modern systems.

Examples

In today’s world, pipes are great for creating scripts to automate tasks without creating a lot of intermediate files.

Pipes are also a powerful tool to filter and transform text interactively. As example, let’s look at how you can use pipe when debugging issues related to a faulty USB device.

First thing we could try is to simply list all the hardware in our machine.

lshw

That created a lot of output. Let’s see if we can narrow it down.

lshw | grep -i usb

The grep command is used to filter the output to only lines containing the word “usb”. The -i flag makes it case-insensitive. We use itbecause it might be spelled “usb”, “USB” or maybe “Usb”.

Just looking at lines with “USB” in them doesn’t help us much. We need some lines above an below to provide some context. Let’s try again with a context window of 10 lines.

lshw | grep -C10 -i usb

Look to see if the device shows up and that it has been claimed by a driver.

We could also try to check kernel messages with the command dmesg.

dmesg | grep -i usb

Note: dmesg needs to be run as root.

We might also try to look for the last (tail) 10 lines of output when we plug the device in.

dmesg | tail -n 10

Redirect to files

Being able to redirect streams to files is really handy.

We can redirect stdout to a file with >.

lshw > hardware.txt

We can redirect stderr with 2>.

lshw 2> errors.txt

To redirect both stdout and stderr, use &>.

lshw &> output.txt

Last, we can redirect stdin using <.

wc -l < hardware.txt

wc is short for word count. However, we can count lines instead by setting -l flag.