# An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Overview This guide is a followup to [An Illustrated Guide to Shell Magic: Standard I/O & Redirection](../../../basic-shell-magic). It's about some of the other ways that Bash and its relatives provide for composable tools and general problem-solving. It's far from comprehensive. Rather, it's a survey of some techniques and ideas that will let you spend less time typing and more time just _doing stuff_ on a Raspberry Pi or similar machine. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/424/medium800/raspberry_pi_pngn_typing.png?1424726675) We'll touch on documentation, software installation, persistent terminal sessions, keybindings, pattern matching, process monitoring, aliases, basic scripting, and more. # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Read the Manual ![](https://cdn-learn.adafruit.com/assets/assets/000/023/422/medium800/raspberry_pi_manual_shelf.png?1424725706) Unix-like systems have nearly always come with a manual, and Linux is no exception. On a Raspbian machine, there are at least a couple of manuals available at all times. It's natural to reach for Google as soon as you run into a technical problem, but sometimes the fastest route to a solution is the locally installed manuals. (And sometimes you just don't have an internet connection.) ## man Curious about a given topic? If you know the name of a command, you can use `man` (short for "manual"). For example, `man grep` will tell you more than you ever wanted to know about the `grep` command. You can search by keyword with `man -k keyword`. You can also search with `apropos keyword`, which for some reason I have always found easier to remember. There's an art to reading most man pages. They're not always the friendliest documents, at first glance, but once you have the basics down, they often pack a lot of knowledge into a small space. Here's how the man page for `grep` usually displays, with a few bits highlighted: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/336/medium800/raspberry_pi_grep_manual.png?1424467953) You'll often find other info - the authors of a given program, notes about its history or how it differs from other versions of the same utility, known bugs, pointers to related programs, and so forth. ## GNU info The GNU project, responsible for the versions of the core utilities most commonly used on Linux systems, has its own documentation system, called `info`. It can be browsed by topic just by invoking `info` or searched with `info -k keyword`. If you already know the name of a node, you can just say `info node`. For example, the documentation on info itself can be reached with `info info`. ## Documentation from Installed Packages Debian, the distribution Raspbian is based on, installs a lot of documentation in `/usr/share/doc`. If you look there, you'll find a directory for most installed packages, often containing a brief README and sometimes complete with example configuration files and scripts. These can be a real life-saver. ## Ask the Program You're Trying to Use for Help It's often the case that utilities are written with their own built-in help. For example, grep (about which we'll be talking more in a minute) has a `--help` option. This isn't always provided, but it's enough of a convention that it's worth trying if you're stumped. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/337/medium800thumb/raspberry_pi_screencast-2015-02-20-14_48_53.jpg?1448316358) Commands with a lot of options might print quite a bit more help than fits on your screen. There's a generic solution to the problem of text scrolling out of your terminal: Pipe the command to `less`. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/338/medium800thumb/raspberry_pi_screencast-2015-02-20-14_53_38.jpg?1448316372) # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Learn How to Find and Install Packages The Raspbian archives can be a little out of date, but they contain a _huge_ collection of software **packages**. A package is something like the installers you might download on other operating systems - it contains executable software, documentation, configuration files, and so on, along with instructions for where these files should go on the filesystem. Packages also contain information about their **dependencies** , the other packages that they depend on in order to function. ## sudo apt-get update You'll want to remember a handful of commands. The first is `apt-get`, which updates the list of available packages and installs from it. First, let's make sure we're up-to-date. Because we're updating system-wide files, we'll need to use `sudo apt-get update`. This will take a while, depending on your network connection, and you'll probably get a few screens full of output. ```auto pi@raspberrypi ~ $ sudo apt-get update Get:1 http://mirrordirector.raspbian.org wheezy Release.gpg [490 B] Get:2 http://archive.raspberrypi.org wheezy Release.gpg [490 B] Get:3 http://raspberrypi.collabora.com wheezy Release.gpg [836 B] ... Fetched 7,040 kB in 39s (180 kB/s) Reading package lists... Done ``` ## sudo apt-get upgrade Next, you can choose to **upgrade** any packages on the system that have new versions available. This is usually a good idea on freshly installed systems or machines you haven't used in a while. Again, you'll need to use `sudo`, and you may have to wait a while. If there are a lot of packages to be downloaded, you'll be prompted whether to continue. Hit "y" as long as  you're prepared to wait out the download and installation time. ```auto pi@raspberrypi ~ $ sudo apt-get upgrade Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: dbus dbus-x11 e2fslibs e2fsprogs krb5-locales libcomerr2 libdbus-1-3 li libraspberrypi-dev libraspberrypi-doc libraspberrypi0 libss2 libxml2 nt python3-rpi.gpio raspberrypi-bootloader sonic-pi sudo unzip wolfram-eng 31 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 350 MB of archives. After this operation, 6,007 kB disk space will be freed. Do you want to continue [Y/n]? Get:1 http://archive.raspberrypi.org/debian/ wheezy/main wolfram-engine a Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main e2fslibs a Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main e2fsprogs ... ``` ## apt-cache Next is `apt-cache`, which works with the cached list of packages. It's handy for figuring out what's available. Maybe your Pi has a real shortage of talking bovines - let's see if anything can help us with that, using `apt-cache search cowsay`. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/346/medium800thumb/raspberry_pi_screencast-2015-02-20-15_37_17.jpg?1448316463) In order to inspect the package, you can use `apt-cache show cowsay`. In particular, look for anything under the `Description` line: ```auto pi@raspberrypi ~ $ apt-cache show cowsay Package: cowsay Version: 3.03+dfsg1-4 Installed-Size: 89 Maintainer: Francois Marier Architecture: all Depends: perl Suggests: filters Size: 21850 SHA256: db58abec6da06b0114f8798ce77d6ff6ce7e7deb3c8cb8216c86740d1bbc0217 SHA1: 21de074e7e203d283020eb29ceb2840ab459cb46 MD5sum: c6346d681711471184bfa28dfd9754b1 Description: configurable talking cow Homepage: http://www.nog.net/~tony/warez/cowsay.shtml Description-md5: c312f9ae79aed8150f991fcfa3df1a03 Tag: game::toys, implemented-in::perl, interface::commandline, role::program, use::entertaining, works-with::text Section: games Priority: optional Filename: pool/main/c/cowsay/cowsay_3.03+dfsg1-4_all.deb ``` ## sudo apt-get install [package] So cowsay looks like a promising package - let's install with `sudo apt-get install cowsay` and see what happens. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/340/medium800thumb/raspberry_pi_screencast-2015-02-20-15_26_17.jpg?1448316419) ![](https://cdn-learn.adafruit.com/assets/assets/000/023/341/medium800thumb/raspberry_pi_screencast-2015-02-20-15_28_37.jpg?1448316442) Success! ## dpkg Finally, if you're working with individual Debian packages (`.deb` files) rather than downloading them from an apt repository, for example [when installing a custom kernel](../../../../raspberry-pi-kernel-o-matic), you can use `dpkg`. To see **I** nfo on a package: ```auto dpkg -I package.deb ``` To install a package: ```auto sudo dpkg -i package.deb ``` # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Use a Terminal Multiplexer ![](https://cdn-learn.adafruit.com/assets/assets/000/023/430/medium800/raspberry_pi_yo_dawg.png?1424727605) ## Get you a tmux I heard about [GNU Screen](https://www.gnu.org/software/screen/) for years, but aside from using it as a serial terminal to talk to an Arduino a handful of times, I never used it much, and I didn't really get the big idea. Then one day a friend convinced me to try [tmux](http://tmux.sourceforge.net/), which is a lot like screen but more modern and capable in a number of ways. It took a while to sink in, but now I rarely open a shell anywhere that's not inside a tmux session. Why? Well, you should try it out. First, let's install `tmux`. ```auto sudo apt-get update sudo apt-get install tmux ``` And start it up. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/339/medium800thumb/raspberry_pi_screencast-2015-02-20-15_14_08.jpg?1448316387) Ok, so now you have your regular terminal, but with a little green status bar at the bottom. So what? It kind of seems like you just opened another terminal inside your terminal (yo dawg), but what does this get you? ## Persistent Multi-Terminal Sessions & Easy Multitasking **Here's the big idea** : Your network connection can go away - you can even just close a terminal window whenever you feel like it - and the next time you connect to the remote machine, you just type: ```auto tmux attach ``` ...and there's your old shell, plugging away, just the way you left it. You can even connect to the _same session_ from a bunch of different machines at once, and see the same thing on all of those screens. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/349/medium800thumb/raspberry_pi_screencast-2015-02-20-16_05_01.jpg?1448316481) There are other useful features. You can open up multiple shells and cycle between them, and there's a way to search through the scrollback in a given buffer that comes in handy all the time. But the main thing is that you can leave lots of programs running on the Pi and reconnect to them whenever you feel like it, and even switch between different computers or networks for accessing them.  You could, for example, start things while you have a monitor and keyboard plugged into the Pi, then reattach to them later over SSH. (Or vice versa.) ## Keybindings In order to get much real use out of tmux, you'll want to remember a handful of keybindings. In order to quit tmux, just type `exit` or press Ctrl-d in the shell like you normally would to logout. Once all open windows are closed, tmux itself will exit. There's lots more. The [tmux man page](http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/tmux.1?query=tmux&sec=1) and [this cheatsheet](https://gist.github.com/MohamedAlaa/2961058) are both good references. # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Use History, Tab Completion, and Line Editing ## History Bash keeps a record of your previous commands - try using the up and down arrow keys (or **Ctrl-p** / **Ctrl-n** ) to cycle through history. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/366/medium800thumb/raspberry_pi_screencast-2015-02-22-21_38_22.jpg?1448025295) If you want to see a bunch of history at once, try the `history` command. This can get long, so try piping it to `tail`: ```auto pi@raspberrypi ~ $ history | tail 92 help history 93 help history|less 94 cowthink what? 95 cowthink 'what?' 96 history 97 man history 98 help history 99 history | head -1 100 history | head 101 history | tail ``` Suppose you remember _part_ of a command, and want to reuse it? Try typing **Ctrl-r** , followed by the part of the command you remember. Bash will search the history for matches. You can then press **Enter** to execute the command over again, or **Esc** to edit it. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/367/medium800thumb/raspberry_pi_screencast-2015-02-22-21_44_14.jpg?1448316618) ## Tab Completion Often, you know the name of a command or file, but it would be nice not to have to type the whole thing. Try typing the first few letters of a command and hit **T**** ab**. Bash will attempt to find a matching command and fill it in for you. If there's more than one command that matches, you may need to hit Tab again, and you'll be presented with a list of possible commands. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/368/medium800thumb/raspberry_pi_screencast-2015-02-22-21_50_13.jpg?1448316636) Similarly, typing part of a file name and pressing Tab will often complete the name. If more than one file matches what you've already typed, hit Tab a couple of times and you'll get a list of files. ## Line Editing If you need to fix a typo or revise an earlier command you've retrieved from history, Bash provides a set of editing commands. There's more - check out [the manual on Readline Interaction](https://www.gnu.org/software/bash/manual/html_node/Readline-Interaction.html#Readline-Interaction). # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Use Pattern Matching The people who originally wrote Unix and a lot of the software that runs on it were fond of pattern matching. You can see this in shells like Bash, in tools like `sed` (a stream editor used for transforming text) and `grep` (a finder of text), and in many of the programming languages that emerged in the 80s and 90s. Pattern matching is an idea that covers a lot of specific tools. For our purposes, it usually means finding a pattern of characters in lines of text, inside files, or in the names of files themselves. This is usually done by writing patterns in a little language which describes the text to be matched. Here's a really simple version of one of those: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/354/medium800/raspberry_pi_two_wildcards.png?1424662225) We'll say that `*`, often said aloud as "star", means "0 or more of any character", and `?` means "1 of any character". Any other characters in the pattern will be treated as **literal** characters, which means they represent themselves and we're literally looking for that character. A `b` will mean that we want to match on a `b`. Consider the pattern `R*`. It'll match any string of characters that starts with an `R`, like `Ralph` or `Rob`. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/355/medium800/raspberry_pi_r_star_ralph.png?1424648079) ![](https://cdn-learn.adafruit.com/assets/assets/000/023/356/medium800/raspberry_pi_r_star_rob.png?1424648358) How about the pattern `R*b`? Well, we know `Ralph` fits _some_ of the criteria. It starts with `R`, followed by some number of characters, but is there a `b` at the end? ![](https://cdn-learn.adafruit.com/assets/assets/000/023/357/medium800/raspberry_pi_r_star_b_ralph.png?1424648857) Not so much. How about for `Rob`? ![](https://cdn-learn.adafruit.com/assets/assets/000/023/358/medium800/raspberry_pi_r_star_b_rob.png?1424649150) Characters like these are often called **wildcards** , because they're like cards that can stand in for any other card in games like poker. # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Use Wildcards in Bash In the shell, you'll often find yourself working with collections of files. For example, you might have lots of text files, Python scripts, or GIFs. Right now, I have a collection of many image files in one directory, some of which have `.gif` extensions and some of which have `.png` extensions. The former are animated GIFs used for these guides, and the latter are still screenshots saved as PNGs. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/361/medium800thumb/raspberry_pi_screencast-2015-02-22-17_27_24.jpg?1448316589) That's a lot of files. I've decided I want to put the GIFs in their own folder, but how to move them without typing all of those names? Well, Bash supports a number of wildcards for working with files, including the basic `*` and `?` just discussed. Let's try [the mv command](../../../../an-illustrated-shell-command-primer/moving-and-copying-files-mv-and-cp) with a wildcard: ```auto pi@raspberrypi ~/adafruit_guides $ ls | wc -l 160 pi@raspberrypi ~/adafruit_guides $ mkdir ../adafruit_guide_gifs pi@raspberrypi ~/adafruit_guides $ mv *.gif ../adafruit_guide_gifs pi@raspberrypi ~/adafruit_guides $ ls | wc -l 63 pi@raspberrypi ~/adafruit_guides $ ls ../adafruit_guide_gifs/ | wc -l 97 pi@raspberrypi ~/adafruit_guides $ ``` To break this down, 1. `ls | wc -l` [pipes](../../../../basic-shell-magic/standard-io-and-pipes) the result of `ls` to `wc`, which is a command for counting words and lines. The `-l` option tells it to only return the number of lines. This just tells us the number of files in the current directory. 2. `mkdir ../adafruit_guide_gifs` makes a new directory one level up from the current working directory (remember that `..` is another way of saying "the parent of this directory"). 3. In `mv *.gif ../adfaruit_guide_gifs`, the wildcard `*.gif` is expanded by Bash to a list of all the files ending in `.gif` in the current directory, and then the command is executed like normal. 4. We use `wc -l` again to show that we've moved 63 files to `adafruit_guide_gifs`. Since filename expansion of this kind (also known as **globbing** ) is built into Bash, it works with any command that takes a list of filenames. # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Find Text with grep(1) Using patterns to specify files in the shell is all well and good - it'll save you _mountains_ of keystrokes - but it's also a bit limited. Often, rather than the names of files on the filesystem, you need to work with text produced by other commands, or with the _contents_ of large text files. It's also often desirable to write patterns that can match more complicated text. This is where tools like `grep` come in. Its task is to look at some text (in a file or in the standard output from another command) and print the lines that match a pattern called a **regular expression**. Regular expressions (also frequently written **regex** or  **regexp** ) are kind of like wildcards on steroids. Instead of a handful of general-purpose magic characters, they offer a rich vocabulary for expressing _what characters_ and _how many of them_ a pattern should match. Regexen are actually a deep, complicated topic, and they come in dozens of different flavors. It can take years to master their use, and they frequently confuse even very experienced programmers. Despite all that, it's easy enough to learn the basics, and the basics are enough to get quite a bit done. As a basic example, let's find some dictionary words. `grep` takes a pattern and (optionally) a file to search. ```auto pi@raspberrypi ~ $ grep '^magi.*$' /usr/share/dict/words magic magic's magical magically magician magician's magicians magisterial magisterially magistrate magistrate's magistrates ``` The pattern here, `^magi.*$`, is passed to `grep` inside of single quotes so that the shell won't expand characters like `*`. It demonstrates a bunch of the basics: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/369/medium800/raspberry_pi_magi.png?1424672504) Notice how `*` works subtly different in a regular expression from a shell pattern. Rather than standing in for any character on its own, it imposes a quantity on the previous token. The `^` and `$` serve to **anchor** the pattern to the beginning and end of the line, respectively. These aren't always necessary, but it can be very useful to say that a string occurs at one end or the other. By default, the `grep` command doesn't treat most characters as magical. To get the full range of its abilities, you'll want to invoke it as `egrep` or `grep -E` for **E** xtended grep. With that in place, here are more of the basics: In addition to `-E`, `grep` takes a bunch of other options. It's worth reading the man page, but the handful that seem to come up most often in shell pipelines are as follows: ## Further Reading - _[Mastering Regular Expressions](http://www.amazon.com/exec/obidos/ASIN/0596528124)_, by Jeffrey Friedl, is one of the more comprehensive available texts. - [The GNU Grep manual](https://www.gnu.org/software/grep/manual/grep.html) # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Find Files with find(1) `find` is a deceptively powerful tool. Invoked without any options, it'll recursively list all of the files under the current directory and any subdirectories. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/385/medium800thumb/raspberry_pi_screencast-2015-02-23-09_51_28.jpg?1448316642) ...that can be useful, but it's also kind of overwhelming. Most often, you'll want to limit the results by file name or type. For example, to find files starting with "squirrel" followed by any extension, you can do something like: ```auto pi@raspberrypi ~ $ find . -name 'squirrel.*' ./python_games/squirrel.png ./python_games/squirrel.py ``` The `-name` and `-iname` (the case-insensitive version) options take shell patterns. If you'd rather use more complex regular expressions, there's a `-regex` option. `find` supports literally dozens of other tests on files, along with actions like deleting files (dangerous!), executing commands with files as parameters, and printing file info according to user-supplied formats. ## Further Reading - [A find Tutorial and Primer](https://danielmiessler.com/study/find/), by Daniel Miessler - [A Unix/Linux “find” Command Tutorial](http://content.hccfl.edu/pollock/Unix/FindCmd.htm), by Wayne Pollock - [The GNU Findutils manual](https://www.gnu.org/software/findutils/manual/html_mono/find.html) # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Watch Processes Even a simple Linux system is usually doing a lot of things at once. ## ps(1) `ps` lists processes. If you run it with no options, it'll show you the processes running under your current login. ```auto pi@raspberrypi ~ $ ps PID TTY TIME CMD 30054 pts/2 00:00:00 bash 30116 pts/2 00:00:00 ps ``` The `PID` column lists a **process ID**. Every running program on the system will have one of these. If a process is stuck otherwise bothersome, you can stop it with `kill [process ID]`. If you want to see more detail about _all of the things_, try `ps aux`: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/408/medium800thumb/raspberry_pi_screencast-2015-02-23-12_58_20.jpg?1448316648) `ps` takes a gazillion different options in a confusing array of different formats, but memorizing one or two of them is usually good enough to give you a window into what's happening. If you're looking for a specific program, you might try grepping the output of `ps`. Let's say I wanted to know about tmux sessions running on my Raspberry Pi. One approach might be `ps aux | grep tmux`: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/415/medium800thumb/raspberry_pi_screencast-2015-02-23-13_32_01.jpg?1448316653) Alternatively, you can use `pgrep`, which is specifically for finding process IDs: ```auto pi@raspberrypi /proc/16514 $ pgrep -l tmux 16513 tmux 22350 tmux 30247 tmux ``` (The `-l` is for **l** isting process names along with the PID.) ## /proc Under Linux, `ps` works by reading the files in `/proc`, which is a virtual filesystem full of info about running processes and suchlike things. If you know the id of a running process, you can inspect all sorts of info about it like so: ```auto pi@raspberrypi ~ $ ps PID TTY TIME CMD 16514 pts/1 00:00:01 bash 30268 pts/1 00:00:00 ps pi@raspberrypi ~ $ cd /proc/16514 pi@raspberrypi /proc/16514 $ ls autogroup exe mountstats sched auxv fd net smaps cgroup fdinfo ns stack clear_refs io oom_adj stat cmdline limits oom_score statm comm maps oom_score_adj status coredump_filter mem pagemap syscall cwd mountinfo personality task environ mounts root wchan ``` ## top(1) and htop(1) `ps` is useful for checking on the state of a running system or writing a shell pipeline, but often what you want is something more like an interactive monitor. Enter `top`: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/416/medium800thumb/raspberry_pi_screencast-2015-02-23-13_43_21.jpg?1448316654) `top` is old school, and it's probably available on just about every Unix system in use. Unfortunately, it's also kind of cryptic and difficult to read at a glance. My advice is to install `htop` instead - it's prettier, friendlier, and has a ton of features for sorting and filtering the process list. ```auto sudo apt-get install htop ``` ![](https://cdn-learn.adafruit.com/assets/assets/000/023/417/medium800thumb/raspberry_pi_screencast-2015-02-23-13_45_25.jpg?1448316658) # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Customize your .bashrc Bash is configurable software, and you can tweak a lot of its behavior by editing a file in your home directory called `.bashrc`. To edit, just: ```auto nano ~/.bashrc ``` On a stock Raspbian installation, you should see something like the following: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/364/medium800/raspberry_pi_Screenshot-2015-02-22-20_39_00.png?1424662788) Have a look around this file. Much of it may seem like gibberish, but the basics are fairly simple. Lines starting with `#` are comments. Variables are set like `FOO=bar`. Once you've made any changes, you can get them to take effect in your current shell with: ```auto source ~/.bashrc ``` ## Write Aliases For Commands You Use Frequently An alias is just a shortcut you can type in place of another command. You define one like so: ```auto alias moo="cowsay moo" ``` You can try this at the prompt: ![](https://cdn-learn.adafruit.com/assets/assets/000/023/365/medium800thumb/raspberry_pi_screencast-2015-02-22-20_56_04.jpg?1448316609) The syntax for adding an alias to `.bashrc` so that it'll be available on every login is identical - have a look through the stock version of the file and you'll probably find several. ## Customize Your Prompt Your prompt - the bit of text, like `pi@raspberrypi ~ $`, that appears before your cursor in Bash - is actually defined by a variable called `PS1`. You can inspect the current value of this variable like so: ```auto pi@raspberrypi ~ $ echo $PS1 \[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\] \[\033[01;34m\]\w \$\[\033[00m\] ``` Not exactly pretty, is it? It makes use of special **escapes** like `\u` for **u** ser, `\h` for **h** ostname, `\w` for **w** orking directory, etc. Sequences like `\[\033[01;32m\]` are used for special non-printing characters that the terminal recognizes as instructions to  use certain text colors. Serious prompt customization is outside the scope of this guide, but you can learn plenty by messing around with it. Have a look at [ezprompt.net](http://ezprompt.net/) for a quick way to assemble a custom prompt, and see the Bash Reference Manual's [Controlling the Prompt](https://www.gnu.org/software/bash/manual/bashref.html#Controlling-the-Prompt) section for a breakdown of escape sequences. ## Keep More History The default `.bashrc` sets history length to 1000. If you're anything like me, your memory needs _way_ more help than that. Look for `HISTSIZE` (the amount of history kept in memory when Bash is running) and `HISTFILESIZE` (the amount of history kept in the history file on disk) and adjust them to your liking. Mine look like so: ```auto export HISTSIZE=10000 export HISTFILESIZE=120000 ``` You can look at the contents of command line history with `nano $HISTFILE`. # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Write Shell Scripts ![](https://cdn-learn.adafruit.com/assets/assets/000/023/431/medium800/raspberry_pi_spellbook.png?1424743567) If, metaphorically speaking, a shell pipeline is kind of like an incantation or minor spell that you come up with on the spot to solve some problem, then **scripts** are a lot like the pages of a spellbook where you keep the incantations that you've found really useful. If you're familiar with a general-purpose programming language like Python, C, Ruby, or JavaScript, then maybe you've noticed by now that Bash has some familar features. That's because the shell _is_ a programming language. Just about any sequence of shell commands you'd type at the prompt can be stashed in a simple text file and executed as a program. What's more, Bash has many of the features you'd expect of a programming language: Between these constructs and the features provided by the standard utilities, shell scripts can be used to solve all sorts of problems. For example: - [Here's a script](https://github.com/adafruit/Adafruit-PiTFT-Helper/blob/master/adafruit-pitft-helper) adapted from [Lady Ada's PiTFT installation guide](../../../../adafruit-pitft-28-inch-resistive-touchscreen-display-raspberry-pi/easy-install) which does the tedious work of configuring a touchscreen for the Pi. - Here's one for [cross-compiling kernels for the Raspberry Pi](https://github.com/adafruit/Adafruit-Pi-Kernel-o-Matic/blob/master/build.sh). - [raspi-config](https://github.com/adafruit/Adafruit-Pi-Kernel-o-Matic/blob/master/build.sh) runs the first time a Pi boots and covers a wide range of configuration options. ## A Sample Script ![](https://cdn-learn.adafruit.com/assets/assets/000/023/450/medium800/raspberry_pi_io_machine.png?1424753166) Let's assemble a _very_ simple filter script that turns all the non-space characters in its input into little stars. First, open `stars.sh` in Nano: ```auto nano stars.sh ``` Next, type or paste the following lines and hit **Ctrl-x** to save the file. ```auto #!/bin/bash sed 's/\S/★/g' /dev/stdin ``` To break this down: - `#!/bin/bash` is what's known as a **shebang** or a **hashbang**. `#!` are special characters that tell the kernel "this file should be run by feeding it to the following program". (You'll also commonly see this written as `#!/usr/bin/env bash`, which is sometimes considered a more portable way to invoke an interpreter.) - `sed` is the **s** tream **e** ditor. It acts as a filter, taking a stream of text in one side and transforming it by running one or more commands. - `s/\S/★/g` is a command that says, more or less, "substitute strings matching this regular expression (`\S`) with this replacement string (`★`), **g** lobally". The `\S` will match everything that's not whitespace in its input. - `\s` (lowercase) is the metacharacter for **s** pace characters; think of the uppercase version as inverting this. - `/dev/stdin` is a special file that contains the standard input to our script. Now make the file **executable** , and give it a try... ```auto chmod +x ./stars.sh ``` ![](https://cdn-learn.adafruit.com/assets/assets/000/023/451/medium800thumb/raspberry_pi_screencast-2015-02-23-22_22_28.jpg?1448316669) Hit **Enter** to submit text, and **Ctrl-**** d** on a line by itself to end input. You could also, just as easily, pipe the output of some other command to `./stars.sh`. ## Further Reading - [Advanced Bash-Scripting Guide](http://www.tldp.org/LDP/abs/html/), by Mendel Cooper. - [Bash Reference Manual](https://www.gnu.org/software/bash/manual/bashref.html) # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Write Shell Utilities in Other Languages We've established that the shell has lots of the features found in general-purpose programming languages. With that out of the way, I'm going to make the opposite point. Remember that fragment I used as an example of conditional logic in Bash? ```auto # Is it between midnight and 6am? # (We'll assume that counts as "after midnight".) if [ "`date '+%H'`" -lt 6 ]; then echo "do not feed the mogwai" fi ``` This is actually... Well, it's not very much fun dealing with [this kind of syntax](http://tldp.org/LDP/abs/html/comparison-ops.html). The truth is that, while shell scripting is _really useful_, it's often not nearly as well suited to writing complicated programs as other languages. If you already know some Python, Perl, Ruby, Node.js, etc., you'll often have a better time using those tools than trying to shoehorn too much into the shell. As a rough guideline, I usually consider rewriting things in another language once a shell script takes up more than one or two screens in my text editor. Fortunately, working with other languages than Bash doesn't mean you have to lose the benefits of shell pipelines, standard I/O, and interaction with other utilities. Following are the equivalents of `stars.sh` in a handful of popular languages. If you use this basic pattern of reading from standard input and writing to standard out, you can write programs that interact seamlessly with the traditional Unix environment, while benefitting from modern high-level language features and libraries. ## Python: stars.py ```auto #!/usr/bin/env python # encoding: utf-8 import sys import re write = sys.stdout.write line = sys.stdin.readline() while line: stars = re.sub('\S', '★', line) write(stars) line = sys.stdin.readline() ``` ## Perl: stars.pl ```auto #!/usr/bin/env perl use warnings; use strict; while (my $line = <>) { $line =~ s/\S/★/g; print $line . "\n"; } ``` ## Node.js: stars.js For this one to run, you'll need Node.js. We've got a [tutorial on installing Node.js on the Pi](../../../../node-embedded-development/installing-node-dot-js), or you can use [the Adafruit Pi Finder](https://github.com/adafruit/Adafruit-Pi-Finder) to install [Occidentalis](https://github.com/adafruit/Adafruit-Occidentalis), our growing collection of packages for single-board computers, which includes Node along with other development tools and configuration helpers. ```auto #!/usr/bin/env node process.stdin.resume(); process.stdin.setEncoding('utf8'); process.stdin.on('data', function(data) { var stars = data.replace(/\S/g, '★'); process.stdout.write(stars); }); ``` # An Illustrated Guide to Shell Magic: Typing Less & Doing More ## Concluding Remarks ![](https://cdn-learn.adafruit.com/assets/assets/000/023/452/medium800/raspberry_pi_sigils.png?1424768754) The shell can be a complicated place, but hopefully by now you're starting to get a sense of how the pieces fit together. No one concept defines the Unix environment. Instead, Unix is something like a library of ideas. Many of them are old, dusty ideas, and most have at least a few sharp edges, but they tend to be in the library because they can be combined with each other to solve new and complicated problems. ![](https://cdn-learn.adafruit.com/assets/assets/000/023/453/medium800/raspberry_pi_pieces.png?1424770458) This series will continue with specific projects utilizing the Raspberry Pi and other single-board Linux computers. In the meanwhile, if you haven't, you might want to check out the huge pile of[Adafruit guides already available for the Raspberry Pi](../../../../category/raspberry-pi) and [the Beaglebone](../../../../category/beaglebone). ## Featured Products ### Linux "Tux" Penguin - Skill badge, iron-on patch [Linux "Tux" Penguin - Skill badge, iron-on patch](https://www.adafruit.com/product/553) You hacked, modded or made something with Linux! Adafruit offers a fun and exciting "badges" of achievement for electronics, science and engineering. We believe everyone should be able to be rewarded for learning a useful skill, a badge is just one of the many ways to show and... No Longer Stocked [Buy Now](https://www.adafruit.com/product/553) [Related Guides to the Product](https://learn.adafruit.com/products/553/guides) ### Linux "Tux" Penguin - Sticker [Linux "Tux" Penguin - Sticker](https://www.adafruit.com/product/663) You hacked, modded or made something with Linux! Adafruit offers a fun and exciting stickers to achievement for electronics, science and engineering. We believe everyone should be able to be rewarded for learning a useful skill, a sticker is just one of the many ways to show and share. No Longer Stocked [Buy Now](https://www.adafruit.com/product/663) [Related Guides to the Product](https://learn.adafruit.com/products/663/guides) ### Raspberry Pi 2 Model B Starter Pack - Includes a Raspberry Pi 2 [Raspberry Pi 2 Model B Starter Pack - Includes a Raspberry Pi 2](https://www.adafruit.com/product/2380) Why not trick out your fresh new board with some accessories? The Pi 2 is a big deal - a big, big deal.  It has an upgraded ARMv7 multicore procssor and a full Gigabyte of RAM - meaning you're going to see ~2x the performance on processor-upgrade only and 4x on average for... No Longer Stocked [Buy Now](https://www.adafruit.com/product/2380) [Related Guides to the Product](https://learn.adafruit.com/products/2380/guides) ### 4GB SD card for Raspberry Pi preinstalled with Raspbian Wheezy [4GB SD card for Raspberry Pi preinstalled with Raspbian Wheezy](https://www.adafruit.com/product/1121) ![Raspbian](../images/rasp.png) This is a 4GB microSDHC card (SDC44GB) that comes with a SD adapter and is pre-programmed to contain the latest Raspberry Pi Linux distribution "Raspbian Wheezy" which was released on 2015-01-31... No Longer Stocked [Buy Now](https://www.adafruit.com/product/1121) [Related Guides to the Product](https://learn.adafruit.com/products/1121/guides) ## Related Guides - [Make a Pi Trash Classifier with Machine Learning and Lobe](https://learn.adafruit.com/lobe-trash-classifier-machine-learning.md) - [CircuitPython Libraries on MicroPython using the Raspberry Pi Pico](https://learn.adafruit.com/circuitpython-libraries-on-micropython-using-the-raspberry-pi-pico.md) - [Using the Slamtec RPLIDAR on a Raspberry Pi](https://learn.adafruit.com/slamtec-rplidar-on-pi.md) - [Adafruit Speaker Bonnet for Raspberry Pi](https://learn.adafruit.com/adafruit-speaker-bonnet-for-raspberry-pi.md) - [An Illustrated Guide to Shell Magic: Standard I/O & Redirection](https://learn.adafruit.com/basic-shell-magic.md) - [Running PyPortal Code on Blinka with Displayio](https://learn.adafruit.com/running-pyportal-code-on-blinka-with-displayio.md) - [Adafruit 2.9" eInk Display Breakouts and FeatherWings](https://learn.adafruit.com/adafruit-2-9-eink-display-breakouts-and-featherwings.md) - [Use Docker to Compile Linux for ESP32-S3](https://learn.adafruit.com/docker-esp32-s3-linux.md) - [Mini Mac Pi](https://learn.adafruit.com/mini-mac-pi.md) - [A DigitalOcean droplet in 10 minutes](https://learn.adafruit.com/a-digitalocean-droplet-in-10-minutes.md) - [Adafruit CRICKIT HAT for Raspberry Pi Linux Computers](https://learn.adafruit.com/adafruit-crickit-hat-for-raspberry-pi-linux-computers.md) - [CompuCanvas 101 - Transitions](https://learn.adafruit.com/compucanvas-101.md) - [Adafruit Raspberry Pi Educational Linux Distro](https://learn.adafruit.com/adafruit-raspberry-pi-educational-linux-distro.md) - [Adafruit Triple LED Matrix Bonnet for Raspberry Pi with HUB75](https://learn.adafruit.com/adafruit-triple-led-matrix-bonnet-for-raspberry-pi-with-hub75.md) - [3D Printed Case for Raspberry Pi Pico](https://learn.adafruit.com/raspberry-pi-pico-case.md)