# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Overview

_This guide assumes you have access to the Bash shell on a GNU/Linux computer. Most of the examples use a Raspberry Pi running Raspbian. If you haven't, you should start with [What is this "Linux", anyhow?](../../../what-is-linux), [What is the Command Line?](../../../what-is-the-command-line/overview), and [An Illustrated Shell Command Primer](../../../an-illustrated-shell-command-primer)._

![](https://cdn-learn.adafruit.com/assets/assets/000/023/269/medium800/raspberry_pi_pngn_magic_hat.png?1424165069)

Now that we've covered some basic shell commands, let's shift focus a little and talk about some of the features that bind those commands together into a useful working environment. In this guide, we'll cover:

- standard input and output
- combining commands with pipes
- redirecting output to and from files
- standard error and exit codes

While these may sound like dry topics, it's here that the command line begins to come into its own as a tool and environment for solving problems. Pull up a terminal and read on!

# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Input & Output

By now, you've probably read about a number of commands available in the shell. For example, `ls` [shows you a list of files](../../../../an-illustrated-shell-command-primer/ls); `cat`, `head`, and `tail` [look inside files](../../../../an-illustrated-shell-command-primer/looking-inside-files-cat-less-head-and-tail), and so on.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/234/medium800/raspberry_pi_machines_3.png?1424110261)

Let's imagine for a second that each of those commands is a kind of little machine in a workshop or on a factory floor. It's actually a pretty good metaphor. Shell commands are generally little self-contained programs, and a program isn't too hard to think of as a machine. Each machine takes some stuff as **input** , chugs along for a while, and produces some stuff as **output**.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/235/medium800/raspberry_pi_io_machine.png?1424110415)

So you have lots of special-purpose machines at your disposal, and they do all kinds of useful stuff. In addition to the ones we already talked about, there are commands to find files, search for text and transform it, look up words, convert units and formats, scan networks, generate pictures - most of the stuff you can think of to do with a computer, really.

If you've got machines to handle a bunch of different tasks, you can get all kinds of work done. And yet... It can get awfully tedious moving things between them.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/236/medium800/raspberry_pi_code_mines.png?1424110501)

Have you ever had to save a file in one program, work on it in another, and import it into a third? Have you ever spent hours copy-and-pasting things between two programs?

I once had a temporary job that consisted of reading a name off of a sheet of paper, looking up a corresponding folder, finding a number in that folder to search for in a database program, and then using the mouse to copy and paste a different number in between two fields _in the same program_.

A lot of work done on computers can feel that way: Like slowly carrying buckets full of stuff in-between machines and programs.

Let's go back to our metaphorical workshop. What if all the machines had standard connectors on them for some sort of magic pipe, and the magic pipe gave you the option to move stuff between any two machines?

# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Standard I/O & Pipes

![](https://cdn-learn.adafruit.com/assets/assets/000/023/238/medium800/raspberry_pi_machines_connected.png?1424113613)

## Standard Streams
Of course, real factories don't have magic pipes that can transport everything, and in the physical world there's no such thing as a universal input/output connector. It turns out, however, that the shell _does_ have those things, or something pretty close to them:

Let's back up a bit. Imagine that a command like `head` or `tail` is a machine that takes input in one side and spits part of it out the other. In many of [the examples we've see so far](../../../../an-illustrated-shell-command-primer/looking-inside-files-cat-less-head-and-tail), you specify the input by giving the command an **argument** naming a file to use, like `tail /usr/share/dict/words`, and the output appears in your terminal. In other cases, a program runs interactively and accepts input directly from the terminal itself - that is, from what you type on your keyboard.

Info: 

![](https://cdn-learn.adafruit.com/assets/assets/000/023/262/medium800thumb/raspberry_pi_screencast-2015-02-16-23_01_29.jpg?1448316320)

What you see in your terminal is like the stuff coming out one end of a machine, or like an open tap spilling water on the ground. You don't have to let it spill out directly into the terminal, though - the machine has a universal output connector of sorts, in the form of **stdout** , and you can connect it to the **stdin** of other machines with a **pipe**.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/263/medium800/raspberry_pi_machine_stdio.png?1424154875)

## Building a Simple Pipeline
Have a look on your keyboard and see if you can find this little critter: `|`. It's often (though not always) located on the same key as the backslash (`\`). Let's try it on something simple.

Suppose you wanted to see the end of the word list, but starting from the last word and working backwards. The `sort` utility will be helpful here, especially since it optionally takes the `-r` argument to **r** everse its output.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/242/medium800thumb/raspberry_pi_screencast-2015-02-16-13_25_48.jpg?1448316201)

This is called a **pipeline** , and it's one of the fundamental building blocks of the modern shell. Most traditional shell utilities will accept standard input and produce standard output.

Pipelines are often simple affairs joining a few basic commands, but they can also be used to express complicated tasks combining many different operations. Here's a slightly more complicated pipeline to have an ASCII-art cow say the usernames of people who have logged into a server I run, starting with most recent login:

```
lastlog | egrep -v 'Never|Username|root' | awk '{ print $1; }' | xargs cowsay
```

![](https://cdn-learn.adafruit.com/assets/assets/000/023/243/medium800thumb/raspberry_pi_screencast-2015-02-16-13_59_59.jpg?1448316213)

Next, let's talk about what to do if you want to stash standard output somewhere.

# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Redirection To & From Files

![](https://cdn-learn.adafruit.com/assets/assets/000/023/244/medium800/raspberry_pi_echo_hi_to_file.png?1424122103)

Now that you know how to use pipes in between commands, you can do a lot without worrying about storing data in files. But what if you _want_ to keep something in a file? Let's say you want to:

- share a file with other people
- keep data for future reference
- build up a log file over time
- do further work on a file in a text editor like Nano or a spreadsheet application
- use data with some other tool which isn't very good at accepting standard input
- pull data _out_ of a file and send it to the standard input of a command which doesn't know how to deal with files on its own

This is where `>`, `>>`, and their sibling `<` come into play.

## Writing to a File
If you have output you want to store in `filename`, just tack `> filename` onto the end of your command line:

![](https://cdn-learn.adafruit.com/assets/assets/000/023/245/medium800thumb/raspberry_pi_screencast-2015-02-16-14_34_59.jpg?1448316219)

As long as you have permission to write to the specified filename, you'll wind up with a file containing the pipeline's output. I like to remember that the `>` character looks sort of like a little funnel.

Info: 

## Appending to the End of a File
Sometimes, you want to stash your output at the end of an existing file. For that, just use `>> filename`:

![](https://cdn-learn.adafruit.com/assets/assets/000/023/246/medium800thumb/raspberry_pi_screencast-2015-02-16-14_37_05.jpg?1448316228)

This is often handy for accumulating log files over time, or for building up a collection of data with multiple pipelines.

## Reading from a File
Lastly, suppose you want to take the contents of a file and send them to the standard input of some command? Just use `command < filename`.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/258/medium800thumb/raspberry_pi_screencast-2015-02-16-15_36_32.jpg?1448316292)

The little funnel is going the other direction now, from the file back into the command.

This one can be harder to remember, because it doesn't come up nearly as often as the first two. Most utilities you'll encounter are written so that they can read files on their own. Still, it can be a powerful trick that saves you a bunch of contortions, especially once you get into writing your own scripts and utilities.

## A Common Gotcha: Changing a File In-Place
Before long, you might find it natural to try and overwrite the contents of some file by running a command on it and redirecting the output of that command to the same file.

For example, `nl` is a command to **n** umber the **l** ines of its input. Suppose I decided that the version of a file I've already saved would be better with line numbers on it? I might be tempted to try something like the following:

![](https://cdn-learn.adafruit.com/assets/assets/000/023/248/medium800thumb/raspberry_pi_screencast-2015-02-16-15_01_04.jpg?1448316266)

Ouch. The whole file seems to be gone. What's happening here? It turns out that the shell opens `file.txt` for writing _before_ it ever executes the command pipeline! This makes a kind of sense once you know about it, but it's not especially obvious at first.

Fortunately, there's an easy workaround in this case - use `tee`:

![](https://cdn-learn.adafruit.com/assets/assets/000/023/261/medium800thumb/raspberry_pi_screencast-2015-02-16-19_36_14.jpg?1448316305)

This does two things:

1. print its stdin back to stdout
2. write its stdin to a given filename

Since `tee file.txt` isn't executed until after `nl file.txt` has already run, you wind up with the desired outcome.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/260/medium800/raspberry_pi_tee.png?1424135482)

# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Standard Error & Exit Codes

![](https://cdn-learn.adafruit.com/assets/assets/000/023/264/medium800/raspberry_pi_machin_stderr.png?1424156108)

## The Discontents of Error Messaging
I mentioned **stderr** earlier, but I glossed over its actual function.

There are some problems with pipelines. One is that if you stream all the output of a command down a pipe to other commands, and part of that output is an error message, you may never discover that a command reported an error at all.

Another is that [lots of things that might&nbsp;_look_ like an error message could be perfectly valid output](https://en.wikipedia.org/wiki/Semipredicate_problem). Consider, for example, working with a log file containing the string "ERROR" on some line.

Unix systems have, for a long time, dealt with these problems by exposing a third standard stream. Programs print error messages and diagnostics to standard error, which (usually) shows up in your terminal even if you've redirected standard output elsewhere.

You can redirect stderr like you can stdout (and knowing this will come in handy sometimes), but the syntax is kind of fiddly and weird, and you shouldn't feel guilty if you have to look it up every time (or at least I don't):

For a concise and well-researched breakdown of the problem, check out Jesse Storimer's [When to use STDERR instead of STDOUT](http://www.jstorimer.com/blogs/workingwithcode/7766119-when-to-use-stderr-instead-of-stdout).

## Exit Status
As if multiple output streams didn't make for enough extra complexity, there's another way commands signal error conditions: Every command actually has a numeric **exit status** , ranging from `0` to `255`, where anything _other_ than `0` signals a failure.

In Bash, you can check the most recent exit status by inspecting the special variable `$?`.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/268/medium800thumb/raspberry_pi_screencast-2015-02-17-01_03_00.jpg?1448316336)

Exit statuses don't come up directly all _that_ often in interactive use of the shell, but they become interesting once you start writing shell _scrip__ts_. We won't get into scripting just now, but if you want to jump right in at the deep end, have a look at Mendel Cooper's&nbsp;_[Advanced Bash-Scripting Guide](http://tldp.org/LDP/abs/html/)._

# An Illustrated Guide to Shell Magic: Standard I/O & Redirection

## Concluding Remarks: Composability

The classic Unix utilities are, traditionally, designed to do one or two things well, and fit together with other utilities to solve problems. This leads to a property known as **composability** , which is another way of saying that the utilities are small pieces which can be put together and rearranged in many different ways.

The present-day GNU versions of the utilities (and lots of the other programs that have accumulated in the era of Linux) relax the _only do one thing_ idea, out of pragmatism and simple [feature creep](https://en.wikipedia.org/wiki/Feature_creep), but the basic idea holds.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/276/medium800/raspberry_pi_machines_3.png?1424197814)

Along with text files and the filesystem, standard streams, pipes, and redirection provide the necessary plumbing of a composable system. Pipelines can be written to explore and solve a huge range of problems by connecting simple operations together, and most tools can share data with other tools.

![](https://cdn-learn.adafruit.com/assets/assets/000/023/277/medium800/raspberry_pi_machines_connected.png?1424197834)

There's other shell magic to be learned, but the most important pieces are in place.

Next up, we look at aliases, wildcards, and the basics of treating the shell as a full-featured scripting language: [An Illustrated Guide to Shell Magic: Typing Less & Doing More](../../../../an-illustrated-guide-to-shell-magic-typing-less-and-doing-more/overview).


## Featured Products

### Raspberry Pi 2 Model B Starter Pack - Includes a Raspberry Pi 2

[Raspberry Pi 2 Model B Starter Pack - Includes a Raspberry Pi 2](https://www.adafruit.com/product/2380)
Why not trick out your fresh new board with some&nbsp;accessories? The Pi 2 is a big deal - a big, big deal. &nbsp;It has an upgraded ARMv7 multicore procssor and a full Gigabyte of RAM - meaning you're going to see ~2x the performance on processor-upgrade only and 4x on average for...

No Longer Stocked
[Buy Now](https://www.adafruit.com/product/2380)
[Related Guides to the Product](https://learn.adafruit.com/products/2380/guides)
### 4GB SD card for Raspberry Pi preinstalled with Raspbian Wheezy

[4GB SD card for Raspberry Pi preinstalled with Raspbian Wheezy](https://www.adafruit.com/product/1121)
![Raspbian](../images/rasp.png)  
This is a 4GB microSDHC card (SDC44GB) that comes with a SD adapter and is pre-programmed to contain the latest Raspberry Pi Linux distribution "Raspbian Wheezy" which was released on 2015-01-31...

No Longer Stocked
[Buy Now](https://www.adafruit.com/product/1121)
[Related Guides to the Product](https://learn.adafruit.com/products/1121/guides)
### BeagleBone Black Rev C - 4GB Flash - Pre-installed Debian

[BeagleBone Black Rev C - 4GB Flash - Pre-installed Debian](https://www.adafruit.com/product/1876)
Note: As of May 12, 2014 Adafruit is shipping Rev C. We have discontinued selling Rev B. There are no exchanges or "upgrades" for Rev B to Rev C.  
  
If you liked the BeagleBone Black Rev B, you will love the Rev C! The Rev C still has a blistering 1GHz processor and 512MB...

No Longer Stocked
[Buy Now](https://www.adafruit.com/product/1876)
[Related Guides to the Product](https://learn.adafruit.com/products/1876/guides)
### Linux "Tux" Penguin - Skill badge, iron-on patch

[Linux "Tux" Penguin - Skill badge, iron-on patch](https://www.adafruit.com/product/553)
You hacked, modded or made something with Linux! Adafruit offers a fun and exciting "badges" of achievement for electronics, science and engineering. We believe everyone should be able to be rewarded for learning a useful skill, a badge is just one of the many ways to show and...

No Longer Stocked
[Buy Now](https://www.adafruit.com/product/553)
[Related Guides to the Product](https://learn.adafruit.com/products/553/guides)

## Related Guides

- [Raspberry Pi Computer Quick-Start](https://learn.adafruit.com/raspberry-pi-computer-quick-start.md)
- [An Illustrated Shell Command Primer](https://learn.adafruit.com/an-illustrated-shell-command-primer.md)
- [RPi Stock Alert Alarm](https://learn.adafruit.com/rpi-stock-alert-alarm.md)
- [Quickstart - Raspberry Pi RP2040 with BLE and CircuitPython](https://learn.adafruit.com/quickstart-raspberry-pi-rp2040-with-ble-and-circuitpython.md)
- [Raspberry Pi E-Ink Weather Station using Python](https://learn.adafruit.com/raspberry-pi-e-ink-weather-station-using-python.md)
- [Raspberry Pi Video Looper 2](https://learn.adafruit.com/raspberry-pi-video-looper-2.md)
- [Turning your Raspberry Pi Zero into a USB Gadget](https://learn.adafruit.com/turning-your-raspberry-pi-zero-into-a-usb-gadget.md)
- [Portable Macrodata Refinement Terminal](https://learn.adafruit.com/portable-macrodata-refinement-terminal.md)
- [Raspberry Pi Video Synth with Blinka and Processing](https://learn.adafruit.com/raspberry-pi-video-synth-with-blinka-and-processing.md)
- [Running PyPortal Code on Blinka with Displayio](https://learn.adafruit.com/running-pyportal-code-on-blinka-with-displayio.md)
- [Adafruit IO Home: Lights and Temperature ](https://learn.adafruit.com/adafruit-io-house-lights-and-temperature.md)
- [Arcade Bonnet Controller](https://learn.adafruit.com/arcade-bonnet-controller.md)
- [Internet Streaming "On Air" Sign](https://learn.adafruit.com/internet-streaming-on-air-sign.md)
- [Adafruit OLED Displays for Raspberry Pi](https://learn.adafruit.com/adafruit-oled-displays-for-raspberry-pi.md)
- [NFC Raspberry Pi Media Player](https://learn.adafruit.com/nfc-raspberry-pi-media-player.md)