This guide is a followup to An Illustrated Guide to Shell Magic: Standard I/O & Redirection. It's about some of the other ways that Bash and its relatives provide for composable tools and general problem-solving.

It's far from comprehensive. Rather, it's a survey of some techniques and ideas that will let you spend less time typing and more time just doing stuff on a Raspberry Pi or similar machine.

We'll touch on documentation, software installation, persistent terminal sessions, keybindings, pattern matching, process monitoring, aliases, basic scripting, and more.

Unix-like systems have nearly always come with a manual, and Linux is no exception. On a Raspbian machine, there are at least a couple of manuals available at all times.

It's natural to reach for Google as soon as you run into a technical problem, but sometimes the fastest route to a solution is the locally installed manuals. (And sometimes you just don't have an internet connection.)

man

Curious about a given topic? If you know the name of a command, you can use man (short for "manual"). For example, man grep will tell you more than you ever wanted to know about the grep command. You can search by keyword with man -k keyword. You can also search with apropos keyword, which for some reason I have always found easier to remember.

There's an art to reading most man pages. They're not always the friendliest documents, at first glance, but once you have the basics down, they often pack a lot of knowledge into a small space. Here's how the man page for grep usually displays, with a few bits highlighted:

You'll often find other info - the authors of a given program, notes about its history or how it differs from other versions of the same utility, known bugs, pointers to related programs, and so forth.

GNU info

The GNU project, responsible for the versions of the core utilities most commonly used on Linux systems, has its own documentation system, called info. It can be browsed by topic just by invoking info or searched with info -k keyword. If you already know the name of a node, you can just say info node. For example, the documentation on info itself can be reached with info info.

Documentation from Installed Packages

Debian, the distribution Raspbian is based on, installs a lot of documentation in /usr/share/doc. If you look there, you'll find a directory for most installed packages, often containing a brief README and sometimes complete with example configuration files and scripts. These can be a real life-saver.

Ask the Program You're Trying to Use for Help

It's often the case that utilities are written with their own built-in help. For example, grep (about which we'll be talking more in a minute) has a --help option. This isn't always provided, but it's enough of a convention that it's worth trying if you're stumped.

Commands with a lot of options might print quite a bit more help than fits on your screen. There's a generic solution to the problem of text scrolling out of your terminal: Pipe the command to less.

The Raspbian archives can be a little out of date, but they contain a huge collection of software packages.

A package is something like the installers you might download on other operating systems - it contains executable software, documentation, configuration files, and so on, along with instructions for where these files should go on the filesystem.

Packages also contain information about their dependencies, the other packages that they depend on in order to function.

sudo apt-get update

You'll want to remember a handful of commands. The first is apt-get, which updates the list of available packages and installs from it. First, let's make sure we're up-to-date. Because we're updating system-wide files, we'll need to use sudo apt-get update. This will take a while, depending on your network connection, and you'll probably get a few screens full of output.

sudo apt-get upgrade

Next, you can choose to upgrade any packages on the system that have new versions available. This is usually a good idea on freshly installed systems or machines you haven't used in a while. Again, you'll need to use sudo, and you may have to wait a while. If there are a lot of packages to be downloaded, you'll be prompted whether to continue. Hit "y" as long as  you're prepared to wait out the download and installation time.

apt-cache

Next is apt-cache, which works with the cached list of packages. It's handy for figuring out what's available. Maybe your Pi has a real shortage of talking bovines - let's see if anything can help us with that, using apt-cache search cowsay.

In order to inspect the package, you can use apt-cache show cowsay. In particular, look for anything under the Description line:

sudo apt-get install [package]

So cowsay looks like a promising package - let's install with sudo apt-get install cowsay and see what happens.

Success!

dpkg

Finally, if you're working with individual Debian packages (.deb files) rather than downloading them from an apt repository, for example when installing a custom kernel, you can use dpkg.

To see Info on a package:

To install a package:

Get you a tmux

I heard about GNU Screen for years, but aside from using it as a serial terminal to talk to an Arduino a handful of times, I never used it much, and I didn't really get the big idea.

Then one day a friend convinced me to try tmux, which is a lot like screen but more modern and capable in a number of ways. It took a while to sink in, but now I rarely open a shell anywhere that's not inside a tmux session. Why? Well, you should try it out. First, let's install tmux.

And start it up.

Ok, so now you have your regular terminal, but with a little green status bar at the bottom. So what? It kind of seems like you just opened another terminal inside your terminal (yo dawg), but what does this get you?

Persistent Multi-Terminal Sessions & Easy Multitasking

Here's the big idea: Your network connection can go away - you can even just close a terminal window whenever you feel like it - and the next time you connect to the remote machine, you just type:

...and there's your old shell, plugging away, just the way you left it. You can even connect to the same session from a bunch of different machines at once, and see the same thing on all of those screens.

There are other useful features. You can open up multiple shells and cycle between them, and there's a way to search through the scrollback in a given buffer that comes in handy all the time.

But the main thing is that you can leave lots of programs running on the Pi and reconnect to them whenever you feel like it, and even switch between different computers or networks for accessing them.  You could, for example, start things while you have a monitor and keyboard plugged into the Pi, then reattach to them later over SSH. (Or vice versa.)

Keybindings

In order to get much real use out of tmux, you'll want to remember a handful of keybindings.

In order to quit tmux, just type exit or press Ctrl-d in the shell like you normally would to logout. Once all open windows are closed, tmux itself will exit.

There's lots more. The tmux man page and this cheatsheet are both good references.

History

Bash keeps a record of your previous commands - try using the up and down arrow keys (or Ctrl-p / Ctrl-n) to cycle through history.

If you want to see a bunch of history at once, try the history command. This can get long, so try piping it to tail:

Suppose you remember part of a command, and want to reuse it?

Try typing Ctrl-r, followed by the part of the command you remember. Bash will search the history for matches. You can then press Enter to execute the command over again, or Esc to edit it.

Tab Completion

Often, you know the name of a command or file, but it would be nice not to have to type the whole thing. Try typing the first few letters of a command and hit Tab. Bash will attempt to find a matching command and fill it in for you. If there's more than one command that matches, you may need to hit Tab again, and you'll be presented with a list of possible commands.

Similarly, typing part of a file name and pressing Tab will often complete the name. If more than one file matches what you've already typed, hit Tab a couple of times and you'll get a list of files.

Line Editing

If you need to fix a typo or revise an earlier command you've retrieved from history, Bash provides a set of editing commands.

There's more - check out the manual on Readline Interaction.

The people who originally wrote Unix and a lot of the software that runs on it were fond of pattern matching.

You can see this in shells like Bash, in tools like sed (a stream editor used for transforming text) and grep (a finder of text), and in many of the programming languages that emerged in the 80s and 90s.

Pattern matching is an idea that covers a lot of specific tools. For our purposes, it usually means finding a pattern of characters in lines of text, inside files, or in the names of files themselves.

This is usually done by writing patterns in a little language which describes the text to be matched. Here's a really simple version of one of those:

We'll say that *, often said aloud as "star", means "0 or more of any character", and ? means "1 of any character". Any other characters in the pattern will be treated as literal characters, which means they represent themselves and we're literally looking for that character. A b will mean that we want to match on a b.

Consider the pattern R*. It'll match any string of characters that starts with an R, like Ralph or Rob.

How about the pattern R*b? Well, we know Ralph fits some of the criteria. It starts with R, followed by some number of characters, but is there a b at the end?

Not so much. How about for Rob?

Characters like these are often called wildcards, because they're like cards that can stand in for any other card in games like poker.

In the shell, you'll often find yourself working with collections of files. For example, you might have lots of text files, Python scripts, or GIFs.

Right now, I have a collection of many image files in one directory, some of which have .gif extensions and some of which have .png extensions. The former are animated GIFs used for these guides, and the latter are still screenshots saved as PNGs.

That's a lot of files. I've decided I want to put the GIFs in their own folder, but how to move them without typing all of those names?

Well, Bash supports a number of wildcards for working with files, including the basic * and ? just discussed. Let's try the mv command with a wildcard:

To break this down,

1. ls | wc -l pipes the result of ls to wc, which is a command for counting words and lines. The -l option tells it to only return the number of lines. This just tells us the number of files in the current directory.
2. mkdir ../adafruit_guide_gifs makes a new directory one level up from the current working directory (remember that .. is another way of saying "the parent of this directory").
3. In mv *.gif ../adfaruit_guide_gifs, the wildcard *.gif is expanded by Bash to a list of all the files ending in .gif in the current directory, and then the command is executed like normal.
4. We use wc -l again to show that we've moved 63 files to adafruit_guide_gifs.

Since filename expansion of this kind (also known as globbing) is built into Bash, it works with any command that takes a list of filenames.

Using patterns to specify files in the shell is all well and good - it'll save you mountains of keystrokes - but it's also a bit limited. Often, rather than the names of files on the filesystem, you need to work with text produced by other commands, or with the contents of large text files.

It's also often desirable to write patterns that can match more complicated text.

This is where tools like grep come in. Its task is to look at some text (in a file or in the standard output from another command) and print the lines that match a pattern called a regular expression.

Regular expressions (also frequently written regex or regexp) are kind of like wildcards on steroids. Instead of a handful of general-purpose magic characters, they offer a rich vocabulary for expressing what characters and how many of them a pattern should match.

Regexen are actually a deep, complicated topic, and they come in dozens of different flavors. It can take years to master their use, and they frequently confuse even very experienced programmers. Despite all that, it's easy enough to learn the basics, and the basics are enough to get quite a bit done.

As a basic example, let's find some dictionary words. grep takes a pattern and (optionally) a file to search.

The pattern here, ^magi.*$, is passed to grep inside of single quotes so that the shell won't expand characters like *. It demonstrates a bunch of the basics:

Notice how * works subtly different in a regular expression from a shell pattern. Rather than standing in for any character on its own, it imposes a quantity on the previous token.

The ^ and $ serve to anchor the pattern to the beginning and end of the line, respectively. These aren't always necessary, but it can be very useful to say that a string occurs at one end or the other.

By default, the grep command doesn't treat most characters as magical. To get the full range of its abilities, you'll want to invoke it as egrep or grep -E for Extended grep. With that in place, here are more of the basics:

In addition to -E, grep takes a bunch of other options. It's worth reading the man page, but the handful that seem to come up most often in shell pipelines are as follows:

Further Reading

• Mastering Regular Expressions, by Jeffrey Friedl, is one of the more comprehensive available texts.
• The GNU Grep manual

find is a deceptively powerful tool. Invoked without any options, it'll recursively list all of the files under the current directory and any subdirectories.

...that can be useful, but it's also kind of overwhelming. Most often, you'll want to limit the results by file name or type. For example, to find files starting with "squirrel" followed by any extension, you can do something like:

The -name and -iname (the case-insensitive version) options take shell patterns. If you'd rather use more complex regular expressions, there's a -regex option.

find supports literally dozens of other tests on files, along with actions like deleting files (dangerous!), executing commands with files as parameters, and printing file info according to user-supplied formats.

Further Reading

• A find Tutorial and Primer, by Daniel Miessler
• A Unix/Linux “find” Command Tutorial, by Wayne Pollock
• The GNU Findutils manual

Even a simple Linux system is usually doing a lot of things at once.

ps(1)

ps lists processes. If you run it with no options, it'll show you the processes running under your current login.

The PID column lists a process ID. Every running program on the system will have one of these. If a process is stuck otherwise bothersome, you can stop it with kill [process ID].

If you want to see more detail about all of the things, try ps aux:

ps takes a gazillion different options in a confusing array of different formats, but memorizing one or two of them is usually good enough to give you a window into what's happening.

If you're looking for a specific program, you might try grepping the output of ps. Let's say I wanted to know about tmux sessions running on my Raspberry Pi. One approach might be ps aux | grep tmux:

Alternatively, you can use pgrep, which is specifically for finding process IDs:

(The -l is for listing process names along with the PID.)

/proc

Under Linux, ps works by reading the files in /proc, which is a virtual filesystem full of info about running processes and suchlike things. If you know the id of a running process, you can inspect all sorts of info about it like so:

top(1) and htop(1)

ps is useful for checking on the state of a running system or writing a shell pipeline, but often what you want is something more like an interactive monitor. Enter top:

top is old school, and it's probably available on just about every Unix system in use. Unfortunately, it's also kind of cryptic and difficult to read at a glance. My advice is to install htop instead - it's prettier, friendlier, and has a ton of features for sorting and filtering the process list.

Bash is configurable software, and you can tweak a lot of its behavior by editing a file in your home directory called .bashrc. To edit, just:

On a stock Raspbian installation, you should see something like the following:

Have a look around this file. Much of it may seem like gibberish, but the basics are fairly simple. Lines starting with # are comments. Variables are set like FOO=bar.

Once you've made any changes, you can get them to take effect in your current shell with:

Write Aliases For Commands You Use Frequently

An alias is just a shortcut you can type in place of another command. You define one like so:

You can try this at the prompt:

The syntax for adding an alias to .bashrc so that it'll be available on every login is identical - have a look through the stock version of the file and you'll probably find several.

Customize Your Prompt

Your prompt - the bit of text, like pi@raspberrypi ~ $, that appears before your cursor in Bash - is actually defined by a variable called PS1. You can inspect the current value of this variable like so:

Not exactly pretty, is it? It makes use of special escapes like \u for user, \h for hostname, \w for working directory, etc. Sequences like \[\033[01;32m\] are used for special non-printing characters that the terminal recognizes as instructions to  use certain text colors.

Serious prompt customization is outside the scope of this guide, but you can learn plenty by messing around with it. Have a look at ezprompt.net for a quick way to assemble a custom prompt, and see the Bash Reference Manual's Controlling the Prompt section for a breakdown of escape sequences.

Keep More History

The default .bashrc sets history length to 1000. If you're anything like me, your memory needs way more help than that. Look for HISTSIZE (the amount of history kept in memory when Bash is running) and HISTFILESIZE (the amount of history kept in the history file on disk) and adjust them to your liking. Mine look like so:

You can look at the contents of command line history with nano $HISTFILE.

If, metaphorically speaking, a shell pipeline is kind of like an incantation or minor spell that you come up with on the spot to solve some problem, then scripts are a lot like the pages of a spellbook where you keep the incantations that you've found really useful.

If you're familiar with a general-purpose programming language like Python, C, Ruby, or JavaScript, then maybe you've noticed by now that Bash has some familar features. That's because the shell is a programming language.

Just about any sequence of shell commands you'd type at the prompt can be stashed in a simple text file and executed as a program.

What's more, Bash has many of the features you'd expect of a programming language:

Between these constructs and the features provided by the standard utilities, shell scripts can be used to solve all sorts of problems. For example:

• Here's a script adapted from Lady Ada's PiTFT installation guide which does the tedious work of configuring a touchscreen for the Pi.
• Here's one for cross-compiling kernels for the Raspberry Pi.
• raspi-config runs the first time a Pi boots and covers a wide range of configuration options.

A Sample Script

Let's assemble a very simple filter script that turns all the non-space characters in its input into little stars. First, open stars.sh in Nano:

Next, type or paste the following lines and hit Ctrl-x to save the file.

To break this down:

• #!/bin/bash is what's known as a shebang or a hashbang. #! are special characters that tell the kernel "this file should be run by feeding it to the following program". (You'll also commonly see this written as #!/usr/bin/env bash, which is sometimes considered a more portable way to invoke an interpreter.)
• sed is the stream editor. It acts as a filter, taking a stream of text in one side and transforming it by running one or more commands.
• s/\S/★/g is a command that says, more or less, "substitute strings matching this regular expression (\S) with this replacement string (★), globally". The \S will match everything that's not whitespace in its input.
• \s (lowercase) is the metacharacter for space characters; think of the uppercase version as inverting this.
• /dev/stdin is a special file that contains the standard input to our script.

Now make the file executable, and give it a try...

Hit Enter to submit text, and Ctrl-d on a line by itself to end input. You could also, just as easily, pipe the output of some other command to ./stars.sh.

Further Reading

• Advanced Bash-Scripting Guide, by Mendel Cooper.
• Bash Reference Manual

We've established that the shell has lots of the features found in general-purpose programming languages. With that out of the way, I'm going to make the opposite point. Remember that fragment I used as an example of conditional logic in Bash?

This is actually... Well, it's not very much fun dealing with this kind of syntax.

The truth is that, while shell scripting is really useful, it's often not nearly as well suited to writing complicated programs as other languages. If you already know some Python, Perl, Ruby, Node.js, etc., you'll often have a better time using those tools than trying to shoehorn too much into the shell.

As a rough guideline, I usually consider rewriting things in another language once a shell script takes up more than one or two screens in my text editor.

Fortunately, working with other languages than Bash doesn't mean you have to lose the benefits of shell pipelines, standard I/O, and interaction with other utilities.

Following are the equivalents of stars.sh in a handful of popular languages. If you use this basic pattern of reading from standard input and writing to standard out, you can write programs that interact seamlessly with the traditional Unix environment, while benefitting from modern high-level language features and libraries.

Python: stars.py

Perl: stars.pl

Node.js: stars.js

For this one to run, you'll need Node.js. We've got a tutorial on installing Node.js on the Pi, or you can use the Adafruit Pi Finder to install Occidentalis, our growing collection of packages for single-board computers, which includes Node along with other development tools and configuration helpers.

The shell can be a complicated place, but hopefully by now you're starting to get a sense of how the pieces fit together. No one concept defines the Unix environment. Instead, Unix is something like a library of ideas. Many of them are old, dusty ideas, and most have at least a few sharp edges, but they tend to be in the library because they can be combined with each other to solve new and complicated problems.

This series will continue with specific projects utilizing the Raspberry Pi and other single-board Linux computers. In the meanwhile, if you haven't, you might want to check out the huge pile of Adafruit guides already available for the Raspberry Pi and the Beaglebone.