-
- Weitere Informationen zu diesem Buch:
Inhaltsverzeichnis | Index | Probekapitel | Kolophon | Rezensionen |
- Weitere Informationen zu diesem Buch:
Solutions and Examples for bash Users
First Edition Juni 2007
ISBN 978-0-596-52678-8
Weitere Informationen zu diesem Buch
Inhaltsverzeichnis |
Index |
Probekapitel |
Kolophon |
Rezensionen |
Inhaltsverzeichnis
- Chapter 1: Beginning bash
- InhaltsvorschauWhat's a shell, and why should you care about it?Any recent computer operating system (by recent, we mean since about 1970) has some sort of user interface—some way of specifying commands for the operating system to execute. But in lots of operating systems, that command interface was really built in and there was only one way to talk to the computer. Furthermore, an operating system's command interface would let you execute commands, but that was about all. After all, what else was there to do?The Unix operating system popularized the notion of separating the shell (the part of the system that lets you type commands) from everything else: the input/output system, the scheduler, memory management, and all of the other things the operating system takes care of for you (and that most users don't want to care about). The shell was just one more program; it was a program whose job was executing other programs on behalf of users.But that was the beginning of a revolution. The shell was just another program that ran on Unix, if you didn't like the standard one, you could create your own. So by the end of Unix's first decade, there were at least two competing shells: the Bourne Shell, sh (which was a descendant of the original Thomson shell), plus the C Shell, csh. By the end of Unix's second decade, there were a few more alternatives: the Korn shell, (ksh), and the first versions of the bash shell (bash). By the end of Unix's third decade, there were probably a dozen different shells.You probably don't sit around saying "should I use csh or bash or ksh today?" You're probably happy with the standard shell that came with your Linux (or BSD or Mac OS X or Solaris or HP/UX) system. But disentangling the shell from the operating system itself made it much easier for software developers (such as Brian Fox, the creator of bash, and Chet Ramey, the current developer and maintainer of bash), to write better shells—you could create a new shell without modifying the operating system itself. It was much easier to get a new shell accepted, since you didn't have to talk some operating vendor into building the shell into their system; all you had to do was package the shell so that it could be installed just like any other program.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Why bash?
- InhaltsvorschauWhy is this book about bash, and not some other shell? Because bash is everywhere. It may not be the newest, and it's arguably not the fanciest or the most powerful (though if not, it comes close), nor is it the only shell that's distributed as open source software, but it is ubiquitous.The reason has to do with history. The first shells were fairly good programing tools, but not very convenient for users. The C shell added a lot of user conveniences (like the ability to repeat a command you just typed), but as a programming language it was quirky. The Korn shell, which came along next (in the early 80s), added a lot of user conveniences, and improved the programming language, and looked like it was on the path to widespread adoption. But ksh wasn't open source software at first; it was a proprietary software product, and was therefore difficult to ship with a free operating system like Linux. (The Korn shell's license was changed in 2000, and again in 2005.)In the late 1980s, the Unix community decided standardization was a good thing, and the POSIX working groups (organized by the IEEE) were formed. POSIX standardized the Unix libraries and utilities, including the shell. The standard shell was primarily based on the 1988 version of the Korn Shell, with some C shell features and a bit of invention to fill in the gaps. bash was begun as part of the GNU project's effort to produce a complete POSIX system, which naturally needed a POSIX shell.bash provided the programming features that shell programmers needed, plus the conveniences that command-line users liked. It was originally conceived as an alternative to the Korn shell, but as the free software movement became more important, and as Linux became more popular, bash quickly overshadowed ksh.As a result, bash is the default user shell on every Linux distribution we know about (there are a few hundred Linux distros, so there are probably a few with some oddball default shell), as well as Mac OS X. It's also available for just about every other Unix operating system, including BSD Unix and Solaris. In the rare cases whereEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- The bash Shell
- Inhaltsvorschaubash is a shell: a command interpreter. The main purpose of bash (or of any shell) is to allow you to interact with the computer's operating system so that you can accomplish whatever you need to do. Usually that involves launching programs, so the shell takes the commands you type, determines from that input what programs need to be run, and launches them for you. You will also encounter tasks that involve a sequence of actions to perform that are recurring, or very complicated, or both. Shell programming, usually referred to as shell scripting, allows you to automate these tasks for ease of use, reliability, and reproducibility.In case you're new to bash, we'll start with some basics. If you've used Unix or Linux at all, you probably aren't new to bash—but you may not have known you were using it. bash is really just a language for executing commands—so the commands you've been typing all along (e.g., ls, cd, grep, cat) are, in a sense, bash commands. Some of these commands are built into bash itself; others are separate programs. For now, it doesn't make a difference which is which.We'll end this chapter with a few recipes on getting bash. Most systems come with bash pre-installed, but a few don't. Even if your system comes with bash, it's always a good idea to know how to get and install it—new versions, with new features, are released from time to time.If you're already running bash, and are somewhat familiar with it, you may want to go straight to . You are not likely to read this book in order, and if you dip into the middle, you should find some recipes that demonstrate what bash is really capable of. But first, the basics.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Decoding the Prompt
- InhaltsvorschauYou'd like to know what all the punctuation on your screen means.All command-line shells have some kind of prompt to alert you that the shell is ready to accept your input. What the prompt looks like depends on many factors including your operating system type and version, shell type and version, distribution, and how someone else may have configured it. In the Bourne family of shells, a trailing $ in the prompt generally means you are logged in as a regular user, while a trailing # means you are root. The root account is the administrator of the system, equivalent to the System account on Windows (which is even more powerful than the Administrator account), or the Supervisor account on Netware. root is all-powerful and can do anything on a typical Unix or Linux system.Default prompts also often display the path to the directory that you are currently in; however, they usually abbreviate it. So a ~ means you are in your home directory. Some default prompts may also display your username and the name of the machine you are logged into. If that seems silly now, it won't when you're logged into five machines at once possibly under different usernames.Here is a typical Linux prompt for a user named jp on a machine called adams, sitting in the home directory. The trailing $ indicates this is a regular user, not root.
jp@adams:~$
Here's the prompt after changing to the /tmp directory. Notice how ~, which really meant /home/jp, has changed to /tmp.jp@adams:/tmp$
The shell's prompt is the thing you will see most often when you work at the command line, and there are many ways to customize it more to your liking. But for now, it's enough to know how to interpret it. Of course, your default prompt may be different, but you should be able to figure out enough to get by for now.There are some Unix or Linux systems where the power of root may be shared, using commands like su and sudo. Or root may not even be all-powerful, if the system is running some kind of mandatory access control (MAC) system such as the NSA's SELinux.- , "Showing Where You Are"
- , "Using sudo More Securely"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Showing Where You Are
- InhaltsvorschauYou are not sure what directory you are in, and the default prompt is not helpful.Use the pwd built-in command, or set a more useful prompt (as in , "Customizing Your Prompt"). For example:
bash-2.03$ pwd /tmp bash-2.03$ export PS1='[\u@\h \w]$ ' [jp@solaris8 /tmp]$
pwd stands for print working directory and takes two options.-Ldisplays your logical path and is the default.s displays your physical location, which may differ from your logical path if you have followed a symbolic link.bash-2.03$ pwd /tmp/dir2 bash-2.03$ pwd -L /tmp/dir2 bash-2.03$ pwd -P /tmp/dir1
- , "Customizing Your Prompt"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding and Running Commands
- InhaltsvorschauYou need to find and run a particular command under bash.Try the type, which, apropos, locate, slocate, find, and ls commands.bash keeps a list of directories in which it should look for commands in an environment variable called
$PATH. The bash built-in type command searches your environment (including aliases, keywords, functions, built-ins, and files in the$PATH) for executable commands matching its arguments and displays the type and location of any matches. It has several arguments, notably the-aflag, which causes it to print all matches instead of stopping at the first one. The which command is similar but only searches your$PATH(and csh aliases). It may vary from system to system (it's usually a csh shell script on BSD, but a binary on Linux), and usually has a-aflag like type. Use these commands when you know the name of a command and need to know exactly where it's located, or to see if it's on this computer. For example:$ type which which is hashed (/usr/bin/which) $ type ls ls is aliased to `ls -F -h' $ type -a ls ls is aliased to `ls -F -h' ls is /bin/ls $ which which /usr/bin/which
Almost all commands come with some form of help on how to use them. Usually there is online documentation called manpages, where "man" is short for manual. These are accessed using the man command, soman lswill give you documentation about the ls command. Many programs also have a built-in help facility, accessed by providing a "help me" argument such as-hor--help. Some programs, especially on other operating systems, will give you help if you don't give them arguments. Some Unix commands will also do that, but a great many of them will not. This is due to the way that Unix commands fit together into something called pipelines, which we'll cover later. But what if you don't know or can't remember the name of the command you need? apropos searches manpage names and descriptions for regular expressions supplied as arguments. This is incredibly useful when you don't remember the name of the command you need. This is the same asman -k.$ apropos music cms (4) - Creative Music System device driver $ man -k music cms (4) - Creative Music System device driver
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Information About Files
- InhaltsvorschauYou need more information about a file, such as what it is, who owns it, if it's executable, how many hard links it has, or when it was last accessed or changed.Use the ls, stat, file, or find commands.
$ touch /tmp/sample_file $ ls /tmp/sample_file /tmp/sample_file $ ls -l /tmp/sample_file -rw-r--r-- 1 jp jp 0 Dec 18 15:03 /tmp/sample_file $ stat /tmp/sample_file File: "/tmp/sample_file" Size: 0 Blocks: 0 IO Block: 4096 Regular File Device: 303h/771d Inode: 2310201 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 501/ jp) Gid: ( 501/ jp) Access: Sun Dec 18 15:03:35 2005 Modify: Sun Dec 18 15:03:35 2005 Change: Sun Dec 18 15:03:42 2005 $ file /tmp/sample_file /tmp/sample_file: empty $ file -b /tmp/sample_file empty $ echo '#!/bin/bash -' > /tmp/sample_file $ file /tmp/sample_file /tmp/sample_file: Bourne-Again shell script text executable $ file -b /tmp/sample_file Bourne-Again shell script text executable
For much more on the find command, see all of .The commandlsshows only filenames, while-lprovides more details about each file. ls has many options; consult the manpage on your system for the ones it supports. Useful options include:ls- -a
- Do not hide files starting with . (dot)
- -F
- Show the type of file with one of these trailing type designators: /*@%=|
- -l
- Long listing
- -L
- Show information about the linked file, rather than the symbolic link itself
- -Q
- Quote names (GNU extension, not supported on all systems)
- -r
- Reverse sort order
- -R
- Recurse though subdirectories
- -S
- Sort by file size
- -1
- Short format but only one file per line
When using-Fa slash (/) indicates a directory, an asterisk (*) means the file is executable, an at sign (@) indicates a symbolic link, a percent sign (%) shows a whiteout, an equal sign (=) is a socket, and a pipe or vertical bar (|) is a FIFO.stat, file, and find all have many options that control the output format; see the manpages on your system for supported options. For example, these options produce output that is similar tols -l:$ ls -l /tmp/sample_file -rw-r--r-- 1 jp jp 14 Dec 18 15:04 /tmp/sample_file $ stat -c'%A %h %U %G %s %y %n' /tmp/sample_file -rw-r--r-- 1 jp jp 14 Sun Dec 18 15:04:12 2005 /tmp/sample_file $ find /tmp/ -name sample_file -printf '%m %n %u %g %t %p' 644 1 jp jp Sun Dec 18 15:04:12 2005 /tmp/sample_file
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Showing All Hidden (dot) Files in the Current Directory
- InhaltsvorschauYou want to see only hidden (dot) files in a directory to edit a file you forget the name of or remove obsolete files.
ls -ashows all files, including normally hidden ones, but that is often too noisy, andls -a.* doesn't do what you think it will.Usels-dalong with whatever other criteria you have.ls -d .* ls -d .b* ls -d .[!.]*
Or construct your wildcard in such a way that . and .. don't match.$ grep -l 'PATH' ~/.[!.]* /home/jp/.bash_history /home/jp/.bash_profile
Due to the way the shell handles file wildcards, the sequence .* does not behave as you might expect or desire. The way filename expansion or globbing works is that any string containing the characters *, ?, or [ is treated as a pattern, and replaced by an alphabetically sorted list of file names matching the pattern. * matches any string, including the null string, while ? matches any single character. Characters enclosed in [] specify a list or range of characters, any of which will match. There are also various extended pattern-matching operators that we're not going to cover here (see "Pattern-Matching Characters" and "extglob Extended Pattern-Matching Operators" in ). So*.txtmeans any file ending in .txt, while*txtmeans any file ending intxt(no dot).f?owould matchfooorfaobut notfooo. So you'd think that .* would match any file beginning with a dot.The problem is that .* is expanded to include . and .., which are then both displayed. Instead of getting just the dot files in the current directory, you get those files, plus all the files and directories in the current directory (.), all the files and directories in the parent directory (..), and the names and contents of any subdirectories in the current directory that start with a dot. This is very confusing, to say the least.You can experiment with the same ls command with-dand without, then tryecho.*. The echo trick simply shows you what the shell expanded your .* to. Tryecho.[!.]*also..[!.]* is a filename expansion pattern where [] denotes a list of characters to match, but the leading ! negates the list. So we are looking for a dot, followed by any character that isEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Shell Quoting
- InhaltsvorschauYou need a rule of thumb for using command-line quoting.Enclose a string in single quotes unless it contains elements that you want the shell to interpolate.Unquoted text and even text enclosed in double quotes is subject to shell expansion and substitution. Consider:
$ echo A coffee is $5?! A coffee is ?! $ echo "A coffee is $5?!" -bash: !": event not found $ echo 'A coffee is $5?!' A coffee is $5?!
In the first example,$5is treated as a variable to expand, but since it doesn't exist it is set to null. In the second example, the same is true, but we never even get there because !" is treated as a history substitution, which fails in this case because it doesn't match anything in the history. The third example works as expected.To mix some shell expansions with some literal strings you may use the shell escape character \ or change your quoting. The exclamation point is a special case because the preceding backslash escape character is not removed. You can work around that by using single quotes or a trailing space as shown here.$ echo 'A coffee is $5 for' "$USER" '?!' A coffee is $5 for jp ?! $ echo "A coffee is \$5 for $USER?\!" A coffee is $5 for jp?\! $ echo "A coffee is \$5 for $USER?! " A coffee is $5 for jp?!
Also, you can't embed a single quote inside single quotes, even if using a backslash, since nothing (not even the backslash) is interpolated inside single quotes. But you can work around that by using double quotes with escapes, or by escaping a single quote outside of surrounding single quotes.# We'll get a continuation prompt since we now have unbalanced quotes $ echo '$USER won't pay $5 for coffee.' > ^C # WRONG $ echo "$USER won't pay $5 for coffee." jp won't pay for coffee. # Works $ echo "$USER won't pay \$5 for coffee." jp won't pay $5 for coffee. # Also works $ echo 'I won'\''t pay $5 for coffee.' I won't pay $5 for coffee.
- for more about shell variable and the
$VARsyntax - for more about ! and the history commands
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using or Replacing Built-ins and External Commands
- InhaltsvorschauYou want to replace a built-in command with your own function or external command, and you need to know exactly what your script is executing (e.g., /bin/echo or the built-in echo). Or you've created a new command and it may be conflicting with an existing external or built-in command.Use the type and which commands to see if a given command exists and whether it is built-in or external.
# type cd cd is a shell builtin # type awk awk is /bin/awk # which cd /usr/bin/which: no cd in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/ sbin:/usr/bin/X11:/usr/X11R6/bin:/root/bin) # which awk /bin/awk
A built-in command is just that; it is built into the shell itself, while an external command is an external file launched by the shell. The external file may be a binary, or it may be a shell script itself, and its important to understand the difference for a couple of reasons. First, when you are using a given version of a particular shell, built-ins will always be available but external programs may or may not be installed on a particular system. Second, if you give one of your own programs the same name as a built-in, you will be very confused about the results since the built-in will always take precedence (see , "Naming Your Script Test"). It is possible to use the enable command to turn built-in commands off and on, though we strongly recommend against doing so unless you are absolutely sure you understand what you are doing.enable -awill list all built-ins and their enabled or disabled status.One problem with built-in commands is that you generally can't use a-hor--helpoption to get usage reminders, and if a manpage exists it's often just a pointer to the large bash manpage. That's where the help command, which is itself a built-in, comes in handy. help displays help about shell built-ins.# help help help: help [-s] [pattern ...] Display helpful information about builtin commands. If PATTERN is specified, gives detailed help on all commands matching PATTERN, otherwise a list of the builtins is printed. The -s option restricts the output for each builtin command matching PATTERN to a short usage synopsis.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Determining If You Are Running Interactively
- InhaltsvorschauYou have some code you want to run only if you are (or are not) running interactively.Use the following
casestatement:#!/usr/bin/env bash # cookbook filename: interactive case "$-" in *i*) # Code for interactive shell here ;; *) # Code for non-interactive shell here ;; esac
$- is a string listing of all the current shell option flags. It will containiif the shell is interactive.You may also see code like the following (this will work, but the solution above is the preferred method):if [ "$PS1" ]; then echo This shell is interactive else echo This shell is not interactive fi
- help case
- help set
- , "Branching Many Ways," for more explanation of the
casestatement
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting bash As Your Default Shell
- InhaltsvorschauYou're using a BSD system, Solaris, or some other Unix variant for which bash isn't the default shell. You're tired of starting bash explicitly all the time, and want to make bash your default shell.First, make sure bash is installed. Try typing bash --version at a command line. If you get a version, it's installed:
$ bash --version GNU bash, version 3.00.16(1)-release (i386-pc-solaris2.10) Copyright (C) 2004 Free Software Foundation, Inc.
If you don't see a version number, you may be missing a directory from your path.chsh -lorcat /etc/shellsmay give you a list of valid shells on some systems. Otherwise, ask your system administrator where bash is, or if it can be installed.chsh -lprovides a list of valid shells on Linux, but opens an editor and allows you to change settings on BSD.-lis not a valid option tochshon Mac OS X, but just runningchshwill open an editor to allow you to change settings, andchpass -sshell will change your shell.If bash is installed, use thechsh -scommand to change your default shell. For example,chsh -s /bin/bash. If for any reason that fails trychsh, passwd -e, passwd -l chpass,orusermod -s /usr/bin/bash. If you still can't change your shell ask your system administrator, who may need to edit the /etc/passwd file. On most systems, /etc/passwd will have lines of the form:cam:pK1Z9BCJbzCrBNrkjRUdUiTtFOh/:501:100:Cameron Newham:/home/cam:/bin/bash cc:kfDKDjfkeDJKJySFgJFWErrElpe/:502:100:Cheshire Cat:/home/cc:/bin/bash
As root, you can just edit the last field of the lines in the password file to the full pathname of whatever shell you choose. If your system has a vipw command, you should use it to ensure password file consistency.Some systems will refuse to allow a login shell that is not listed in /etc/shells. If bash is not listed in that file, you will have to have your system administrator add it.Some operating systems, notably the BSD Unixes, typically place bash in the /usr partition. You may want to think twice about changing root's shell on such systems. If the system runs into trouble while booting, and you have to work on it beforeEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting bash for Linux
- InhaltsvorschauYou want to get bash for your Linux system, or you want to make sure you have the latest version.bash is included in virtually all modern Linux distributions. To make sure you have the latest version available for your distribution, use the distribution's built-in packaging tools. You must be root or have the root password to upgrade or install applications.Some Linux distributions (notably Debian) include bash version 2.x as plain bash and version 3.x as bash3, so you need to watch out for that. lists the default versions as of early 2007 (distributions update their repositories often, so versions might have changed from this listing).
Table 1-1: Default Linux distributions Distribution2.x in base install2.x in updates3.x in base install3.x in updatesDebian Woody2.05aN/AN/AN/ADebian Sarge2.05b3.1dfsg-8 (testing & unstable)3.0-12(1)-release3.00.16(1)-releaseFedora Core 1bash-2.05b-31.i386.rpmbash-2.05b-34.i386.rpmN/AN/AFedora Core 2bash-2.05b-38.i386.rpmN/AN/AN/AFedora Core3N/AN/Abash-3.0-17.i386.rpmbash-3.0-18.i386.rpmFedora Core 4N/AN/Abash-3.0-31.i386.rpmN/AFedora Core 5N/AN/Abash-3.1-6.2.i386.rpmbash-3.1-9.fc5.1.i386.rpmFedora Core 6N/AN/Abash-3.1-16.1.i386.rpmN/AKnoppix 3.9 & 4.0.2N/AN/A3.0-15N/AMandrake 9.2bash-2.05b-14mdk.i586.rpmN/AN/AN/AMandrake 10.1bash-2.05b-22mdk.i586.rpmN/AN/AN/AMandrake 10.2N/AN/Abash-3.0-2mdk.i586.rpmN/AMandriva 2006.0N/AN/Abash-3.0-6mdk.i586.rpmN/AMandriva 2007.0N/AN/Abash-3.1-7mdv2007.0.i586.rpmN/AOpenSUSE 10.0N/AN/A3.00.16(1)-release3.0.17(1)-releaseOpenSUSE 10.1N/AN/A3.1.16(1)-releaseN/AOpenSUSE 10.2N/AN/Abash-3.1-55.i586.rpmN/ASLED 10 RC3N/AN/A3.1.17(1)-releaseN/ARHEL 3.6, CentOS 3.6bash-2.05b.0(1)N/AN/AN/ARHEL 4.4, CentOS 4.4N/AN/A3.00.15(1)-releaseN/AMEPIS 3.3.1N/AN/A3.0-14N/AUbuntu 5.10N/AN/A3.0.16(1)N/AUbuntu 6.06N/AN/A3.1.17(1)-releaseN/AUbuntu 6.10N/AN/A3.1.17(1)-releaseN/AFor Debian and Debian-derived systems such as Knoppix, Ubuntu, and MEPIS, make sure your /etc/apt/sources.list file is pointing at an up-to-date Debian mirror; then use the graphical Synaptic, kpackage, gnome-aptEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting bash for xBSD
- InhaltsvorschauYou want to get bash for your FreeBSD, NetBSD, or OpenBSD system, or you want to make sure you have the latest version.To see if bash is installed, check the /etc/shells file. To install or update bash, use the
pkg_addcommand. If you are an experienced BSD user, you may prefer using the ports collection, but we will not cover that here.FreeBSD:pkg_add -vr bash
For NetBSD, browse to Application Software for NetBSD at http://netbsd.org/Documentation/software/ and locate the latest bash package for your version and architecture, then use a command such as:pkg_add -vu ftp://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc-2005Q3/NetBSD-2.0/i386/ All/bash-3.0pl16nb3.tgz
For OpenBSD, you use thepkg_add -vrcommand. You may have to adjust the FTP path for your version and architecture. Also, there may be a statically compiled version. For example: ftp://ftp.openbsd.org/pub/OpenBSD/3.8/packages/i386/bash-3.0.16p1-static.tgz.pkg_add -vr ftp://ftp.openbsd.org/pub/OpenBSD/3.8/packages/i386/bash-3.0.16p1.tgz
FreeBSD and OpenBSD place bash in /usr/local/bin/bash while NetBSD uses /usr/pkg/ bin/bash.Interestingly, PC-BSD 1.2, a "rock-solid Unix operating system based on FreeBSD," comes with bash 3.1.17(0) in /usr/local/bin/bash, though the default shell is still csh.- , "Setting bash As Your Default Shell"
- , "Testing Scripts in VMware"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting bash for Mac OS X
- InhaltsvorschauYou want to get bash for your Mac, or you want to make sure you have the latest version.According to Chet Ramey's bash page (http://tiswww.tis.case.edu/~chet/bash/bashtop.html), Mac OS 10.2 (Jaguar) and newer ship with bash as /bin/sh. 10.4 (Tiger) has version 2.05b.0(1)-release (powerpc-apple-darwin8.0). There are also precompiled OS X packages of bash-2.05 available from many web sites. One such package is at HMUG. Bash for Darwin (the base for Mac OS X) is available from Fink or DarwinPorts.It is also possible to build a more recent version of bash from source, but this is recommended only for experienced users.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Getting bash for Unix
- InhaltsvorschauYou want to get bash for your Unix system, or you want to make sure you have the latest version.If it's not already installed or in your operating system's program repository, check Chet Ramey's bash page for binary downloads, or build it from source (see ).According to Chet Ramey's bash page (http://tiswww.tis.case.edu/~chet/bash/bashtop.html):Solaris 2.x, Solaris 7, and Solaris 8 users can get a precompiled version of bash-3.0 from the Sunfreeware site. Sun ships bash-2.03 with Solaris 8 distributions, ships bash-2.05 as a supported part of Solaris 9, and ships bash-3.0 as a supported part of Solaris 10 (directly on the Solaris 10 CD).AIX users can get precompiled versions of older releases of bash for various versions of AIX from Groupe Bull, and sources and binaries of current releases for various AIX releases from UCLA. IBM makes bash-3.0 available for AIX 5L as part of the AIX tool-box for [GNU/]Linux applications. They use RPM format; you can get RPM for AIX from there, too.SGI users can get an installable version of bash-2.05b from the SGI Freeware page.HP-UX users can get bash-3.0 binaries and source code from the Software Porting and Archive Center for HP-UX.Tru64 Unix users can get sources and binaries for bash-2.05b from the HP/Compaq Tru64 Unix Open Source Software Collection.
- , "Setting bash As Your Default Shell"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting bash for Windows
- InhaltsvorschauYou want to get bash for your Windows system, or you want to make sure you have the latest version.Use Cygwin.Download http://www.cygwin.com/setup.exe and run it. Follow the prompts and choose the packages to install, including bash, which is located in the shells category and is selected by default. As of early 2007, bash-3.1-6 and 3.2.9-11 are available.Once Cygwin is installed, you will have to configure it. See the User Guide at http://cygwin.com/cygwin-ug-net.From the Cygwin site:What Is CygwinCygwin is a Linux-like environment for Windows. It consists of two parts:
- A DLL (cygwin1.dll), which acts as a Linux API emulation layer providing substantial Linux API functionality.
- A collection of tools, which provide Linux look and feel.The Cygwin DLL works with all non-beta, non "release candidate," x86 32-bit versions of Windows since Windows 95, with the exception of Windows CE.What Isn't Cygwin
- Cygwin is not a way to run native Linux apps on Windows. You have to rebuild your application from source if you want to get it running on Windows.
- Cygwin is not a way to magically make native Windows apps aware of Unix functionality (e.g., signals, ptys). Again, you need to build your apps from source if you want to take advantage of Cygwin functionality.
Cygwin is a true Unix-like environment running on top of Windows. It is an excellent tool, but sometimes it might be overkill. For Windows native binaries of the GNU Text Utils (not including bash), see http://unxutils.sourceforge.net/.Microsoft Services for Unix (http://www.microsoft.com/windowsserversystem/sfu/default.mspx) may also be of interest, but note that it is not under active development anymore, though it will be supported until at least 2011 (http://www.eweek.com/article2/0,1895,1855274,00.asp).For powerful character-based and GUI command-line shells with a more consistent interface, but a DOS/Windows flavor, see http://jpsoft.com/. None of the authors are affiliated with this company, but one is a long-time satisfied user.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting bash Without Getting bash
- InhaltsvorschauYou want to try out a shell or a shell script on a system you don't have the time or the resources to build or buy.Or, you feel like reading a Zen-like recipe just about now.Get a free or almost free shell account from HP, Polar Home, or another vendor.HP maintains a free "test drive" program that provides free shell accounts on many operating systems on various HP hardware. See http://www.testdrive.hp.com/ for details.Polar Home provides many free services and almost free shell accounts. According to their web site:polarhome.com is non commercial, educational effort for popularization of shell enabled operating systems and Internet services, offering shell accounts, mail and other online services on all available systems (currently on Linux, OpenVMS, Solaris, AIX, QNX, IRIX, HP-UX, Tru64, FreeBSD, OpenBSD, NetBSD and OPENSTEP).[…]Note: this site is continuously under construction and running on slow lines and low capacity servers that have been retired, therefore as a non commercial site user/visitor, nobody should have too high expectations in any meaning of the word. Even if polarhome.com does all to provide services on professional level, users should not expect more than "AS-IS".polarhome.com is a distributed site, but more than 90% of polarhome realm is located in Stockholm, Sweden.
- List of free shell accounts: http://www.ductape.net/~mitja/freeunix.shtml
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Learning More About bash Documentation
- InhaltsvorschauYou'd like to read more about bash but don't know where to start.Well you're reading this book, which is a great place to start! The other O'Reilly books about bash and shell scripting are: Learning the bash Shell by Cameron Newham (O'Reilly) and Classic Shell Scripting by Nelson H.F. Beebe and Arnold Robbins (O'Reilly).Unfortunately, the official bash documentation has not been easily accessible online—until now! Previously, you had to download several different tarballs, locate all the files that contain documentation, and then decipher the file names to find what you wanted. Now, our companion web site (http://www.bashcookbook.com/) has done all this work for you and provides the official bash reference documentation online so it's easy to refer to. Check it out, and refer others to it as needed.
Section 1.16.2.1: Official documentation
The official bash FAQ is at: ftp://ftp.cwru.edu/pub/bash/FAQ. See especially "H2) What kind of bash documentation is there?" The official reference guide is also strongly recommended; see below for details.Chet Ramey's (the current bash maintainer) bash page (called bashtop) contains a ton of very useful information (http://tiswww.tis.case.edu/~chet/bash/bashtop.html). Chet also maintains the following (listed in bashtop):- README
- A file describing bash: http://tiswww.tis.case.edu/chet/bash/README
- NEWS
- A file tersely listing the notable changes between the current and previous versions: http://tiswww.tis.case.edu/chet/bash/NEWS
- CHANGES
- A complete bash change history: http://tiswww.tis.case.edu/chet/bash/CHANGES
- INSTALL
- Installation instructions: http://tiswww.tis.case.edu/chet/bash/INSTALL
- NOTES
- Platform-specific configuration and operation notes: http://tiswww.tis.case.edu/chet/bash/NOTES
- COMPAT
The latest bash source code and documentation are always available at: http://ftp.gnu.org/gnu/bash/.We highly recommend downloading both the source and the documentation even if you are using prepackaged binaries. Here is a brief list of the documentation. See for an index of the included examples and source code. See the source tarball'sEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 2: Standard Output
- InhaltsvorschauNo software is worth anything if there is no output of some sort. But I/O (Input/ Output) has long been one of the nastier areas of computing. If you're ancient, you remember the days most of the work involved in running a program was setting up the program's input and output. Some of the problems have gone away; for example, you no longer need to get operators to mount tapes on a tape drive (not on any laptop or desktop system that I've seen). But many of the problems are still with us.One problem is that there are many different types of output. Writing something on the screen is different from writing something in a file—at least, it sure seems different. Writing something in a file seems different from writing it on a tape, or in flash memory, or on some other kind of device. And what if you want the output from one program to go directly into another program? Should software developers be tasked with writing code to handle all sorts of output devices, even ones that haven't been invented yet? That's certainly inconvenient. Should users have to know how to connect the programs they want to run to different kinds of devices? That's not a very good idea, either.One of the most important ideas behind the Unix operating system was that everything looked like a file (an ordered sequence of bytes). The operating system was responsible for this magic. It didn't matter whether you were writing to a file on the disk, the terminal, a tape drive, a memory stick, or something else; your program only needed to know how to write to a file, and the operating system would take it from there. That approach greatly simplified the problem. The next question was, simply, "which file?" How does a program know whether to write to the file that represents a terminal window, a file on the disk, or some other kind of file? Simple: that's something that can be left to the shell.When you run a program, you still have to connect it to output files and input files (which we'll see in the next chapter). That task doesn't go away. But the shell makes it trivially easy. A command as simple as:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Writing Output to the Terminal/Window
- InhaltsvorschauYou want some simple output from your shell commands.Use the echo built-in command. All the parameters on the command line are printed to the screen. For example:
echo Please wait.
producesPlease wait.
as we see in this simple session where we typed the command at the bash prompt (the $ character):$ echo Please wait. Please wait. $
The echo command is one of the most simple of all bash commands. It prints the arguments of the command line to the screen. But there are a few points to keep in mind. First, the shell is parsing the arguments on the echo command line (like it does for every other command line). This means that it does all its substitutions, wildcard matching, and other things before handing the arguments off to the echo command. Second, since they are parsed as arguments, the spacing between arguments is ignored. For example:$ echo this was very widely spaced this was very widely spaced $
Normally the fact that the shell is very forgiving about white space between arguments is a helpful feature. Here, with echo, it's a bit disconcerting.- help echo
- help printf
- , "Writing Output with More Formatting Control"
- , "Using echo Portably"
- , "Forgetting to Set Execute Permissions"
- "echo Options and Escape Sequences" in
- "printf" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Writing Output but Preserving Spacing
- InhaltsvorschauYou want the output to preserve your spacing.Enclose the string in quotes. The previous example, but with quotes added, will preserve our spacing.
$ echo "this was very widely spaced" this was very widely spaced $
or:$ echo 'this was very widely spaced' this was very widely spaced $
Since the words are enclosed in quotes, they form a single argument to the echo command. That argument is a string and the shell doesn't need to interfere with the contents of the string. In fact, by using the single quotes ('') the shell is told explicitly not to interfere with the string at all. If you use double quotes ("), some shell substitutions will take place (variable and tilde expansions and command substitutions), but since we have none in this example, the shell has nothing to change. When in doubt, use the single quotes.- help echo
- help printf
- for more information about substitution
- , "Writing Output with More Formatting Control"
- , "Using echo Portably"
- , "Seeing Odd Behavior from printf"
- "echo Options and Escape Sequences" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Writing Output with More Formatting Control
- InhaltsvorschauYou want more control over the formatting and placement of output.Use the printf built-in command.For example:
$ printf '%s = %d\n' Lines $LINES Lines = 24 $
or:$ printf '%-10.10s = %4.2f\n' 'GigaHerz' 1.92735 GigaHerz = 1.93 $
The printf built-in command behaves like the C language library call, where the first argument is the format control string and the successive arguments are formatted according to the format specifications (%).The numbers between the % and the format type (sorfin our example) provide additional formatting details. For the floating-point type (f), the first number (4in the4.2specifier) is the width of the entire field. The second number (2) is how many digits should be printed to the right of the decimal point. Note that it rounds the answer.For a string, the first digit is the maximum field width, and the second is the minimum field width. The string will be truncated (if longer than max) or blank padded (if less than min) as needed. When the max and min specifiers are the same, then the string is guaranteed to be that length. The negative sign on the specifier means to left align the string (within its field width). Without the minus sign, the string would right justify, thus:$ printf '%10.10s = %4.2f\n' 'GigaHerz' 1.92735 GigaHerz = 1.93 $
The string argument can either be quoted or unquoted. Use quotes if you need to preserve embedded spacing (there were no spaces needed in our one-word strings), or if you need to escape the special meaning of any special characters in the string (again, our example had none). It's a good idea to be in the habit of quoting any string that you pass to printf, so that you don't forget the quotes when you need them.- help printf
- Learning the bash Shell, Cameron Newham (O'Reilly), , or any C refer-ence on its printf function
- , "Using echo Portably"
- , "Seeing Odd Behavior from printf"
- "printf" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Writing Output Without the Newline
- InhaltsvorschauYou want to produce some output without the default newline that echo provides.Using printf it's easy—just leave off the ending
\nin your format string. With echo, use the-noption.$ printf "%s %s" next prompt next prompt$
or:$ echo -n prompt prompt$
Since there was no newline at the end of the printf format string (the first argument), the prompt character ($) appears right where the printf left off. This feature is much more useful in shell scripts where you may want to do partial output across several statements before completing the line, or where you want to display a prompt to the user before reading input.With the echo command there are two ways to eliminate the newline. First, the-noption suppresses the trailing newline. The echo command also has several escape sequences with special meanings similar to those in C language strings (e.g.,\nfor newline). To use these escape sequences, you must invoke echo with the-eoption. One of echo's escape sequences is\c, which doesn't print a character, but rather inhibits printing the ending newline. Thus, here's a third solution:$ echo -e 'hi\c' hi$
Because of the powerful and flexible formatting that printf provides, and because it is a built-in with very little over head to invoke (unlike other shells or older versions of bash, where printf was a standalone executable), we will use printf for many of our examples throughout the book.- help echo
- help printf
- See , particularly , "Getting User Input"
- , "Writing Output with More Formatting Control"
- , "Using echo Portably"
- , "Seeing Odd Behavior from printf"
- "echo Options and Escape Sequences" in
- "printf" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving Output from a Command
- InhaltsvorschauYou want to keep the output from a command by putting it in a file.Use the > symbol to tell the shell to redirect the output into a file. For example:
$ echo fill it up fill it up $ echo fill it up > file.txt $
Just to be sure, let's look at what is inside file.txt to see if it captured our output:$ cat file.txt fill it up $
The first line of the example shows an echo command with three arguments that are printed out. The second line of code uses the > to capture that output into a file named file.txt, which is why no output appears after that echo command.The second part of the example uses the cat command to display the contents of the file. We can see that the file contains what the echo command would have otherwise sent as output.The cat command gets its name from the longer word concatenation. The cat command concatenates the output from the several files listed on its command line, as in:cat file1 filetwo anotherfile morefiles—the contents of those files would be sent, one after another, to the terminal window. If a large file had been split in half then it could be glued back together (i.e., concatenated) by capturing the output into a third file:$ cat first.half second.half > whole.file
So our simple command,cat file.txt, is really just the trivial case of concatenating only one file, with the result sent to the screen. That is to say, while cat is capable of more, its primary use is to dump the contents of a file to the screen.- man cat
- , "Numbering Lines"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving Output to Other Files
- InhaltsvorschauYou want to save the output with a redirect to elsewhere in the filesystem, not in the current directory.Use more of a pathname when you redirect the output.
$ echo some more data > /tmp/echo.out
or:$ echo some more data > ../../over.here
The filename that appears after the redirection character (the >) is actually a path-name. If it begins with no other qualifiers, the file will be placed in the current directory.If that filename begins with a slash (/) then this is an absolute pathname, and will be placed where it specifies in the filesystem hierarchy (i.e., tree) beginning at the root (provided all the intermediary directories exist and have permissions that allow you to traverse them). We used /tmp since it is a well-known, universally available scratch directory on virtually all Unix systems. The shell, in this example, will create the file named echo.out in the /tmp directory.Our second example, placing the output into ../../over.here, uses a relative path-name, and the .. is the specially-named directory inside every directory that refers to the parent directory. So each reference to .. moves up a level in the filesystem tree (toward the root, not what we usually mean by up in a tree). The point here is that we can redirect our output, if we want, into a file that is far away from where we are running the command.- Learning the bash Shell by Cameron Newham (O'Reilly), , , , for an introduction to files, directories, and the dot notation (i.e., . and .. )
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving Output from the ls Command
- InhaltsvorschauYou tried to save output from the ls command with a redirect, but when you look at the resulting file, the format is not what you expected.Use the
-Coption on ls when you redirect the output.Here's the ls command showing the contents of a directory:$ ls a.out cong.txt def.conf file.txt more.txt zebra.list $
But when we save the output with the > to redirect it to a file, and then show the file contents, we get this:$ ls > /tmp/save.out $ cat /tmp/save.out a.out cong.txt def.conf file.txt more.txt. zebra.list $
This time we'll use the-Coption:$ ls -C > /tmp/save.out $ cat /tmp/save.out a.out cong.txt def.conf file.txt more.txt zebra.list $
Alternatively, if we use the-1option on ls when we don't redirect, then we get out-put like this:$ ls -1 a.out Cong.txt def.conf. file.txt more.txt save.out zebra.list $
Then the original attempt at redirection matches this output.Just when you thought that you understood redirection and you tried it on a simple ls command, it didn't quite work right. What's going on here?The shell's redirection is meant to be transparent to all programs, so programs don't need special code to make their output redirect-able. The shell takes care of it when you use the > to send the output elsewhere. But it turns out that code can be added to a program to figure out when its output is being redirected. Then, the program can behave differently in those two cases—and that's what ls is doing.The authors of ls figured that if your output is going to the screen then you probably want columnar output (-Coption), as screen real estate is limited. But they assumed if you're redirecting it to a file, then you'll want one file per line (the minus one-1option) since there are more interesting things you can do (i.e., other processing) that is easier if each filename is on a line by itself.- man ls
- , "Saving Output to Other Files"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sending Both Output and Error Messages to Different Files
- InhaltsvorschauYou are expecting output from a program but you don't want it to get littered with error messages. You'd like to save your error messages, but it's harder to find them mixed among the expected output.Redirect output and error messages to different files.
$ myprogram 1> messages.out 2> message.err
Or more commonly:$ myprogram > messages.out 2> message.err
This example shows two different output files that will be created by the shell. The first, messages.out, will get all the output from the hypothetical myprogram redirected into it. Any error messages from myprogram will be redirected into message.err.In the constructs1>and2>the number is the file descriptor, so 1 is STDOUT and 2 is STDERR. When no number is specified, STDOUT is assumed.- , "Saving Output to Other Files"
- , "Throwing Output Away"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sending Both Output and Error Messages to the Same File
- InhaltsvorschauUsing redirection, you can redirect output or error messages to separate files, but how do you capture all the output and error messages to a single file?Use the shell syntax to redirect standard error messages to the same place as standard output.Preferred:
$ both >& outfile
or:$ both &> outfile
or older and slightly more verbose:$ both > outfile 2>&1
wherebothis just our (imaginary) program that is going to generate output to both STDERR and STDOUT.&> or >& is a shortcut that simply sends both STDOUT and STDERR to the same place—exactly what we want to do.In the third example, the1appears to be used as the target of the redirection, but the>&says to interpret the1as a file descriptor instead of a filename. In fact, the2>&are a single entity, indicating that standard output (2) will be redirected (>) to a file descriptor (&) that follows (1). The2>&all have to appear together without spaces, otherwise the2would look just like another argument, and the & actually means something completely different when it appears by itself. (It has to do with running the command in the background.)It may help to think of all redirection operators as taking a leading number (e.g.,2>) but that the default number for > is1, the standard output file descriptor.You could also do the redirection in the other order, though it is slightly less read-able, and redirect standard output to the same place to which you have already redirected standard error:$ both 2> outfile 1>&2
The1is used to indicate standard output and the2for standard error. By our reasoning (above) we could have written just>&2for that last redirection, since1is the default for >, but we find it more readable to write the number explicitly when redirecting file descriptors.Note the order of the contents of the output file. Sometimes the error messages may appear sooner in the file than they do on the screen. That has to do with the unbuffered nature of standard error, and the effect becomes more pronounced when writing to a file instead of the screen.- , "Saving Output to Other Files"
- , "Throwing Output Away"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Appending Rather Than Clobbering Output
- InhaltsvorschauEach time you redirect your output, it creates that output file anew. What if you want to redirect output a second (or third, or …) time, and don't want to clobber the previous output?The double greater-than sign (>>) is a bash redirector that means append the output:
$ ls > /tmp/ls.out $ cd ../elsewhere $ ls >> /tmp/ls.out $ cd ../anotherdir $ ls >> /tmp.ls.out $
The first line includes a redirect that removes the file if it exists and starts with a clean (empty) file, filling it with the output from the ls command.The second and third invocations of ls use the double greater than sign (>>) to indicate appending to, rather than replacing, the output file.- , "Saving Output to Other Files"
- , "Throwing Output Away"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Just the Beginning or End of a File
- InhaltsvorschauYou need to display or use just the beginning or end of a file.Use the head or tail commands. By default, head will output the first 10 lines and tail will output the last 10 lines of the given file. If more than one file is given, the appropriate lines from each of them are output. Use the -number switch (e.g.,
-5) to change the number of lines. tail also has the-fand-Fswitches, which follow the end of the file as it is written to. And it has an interesting + switch that we cover in , "Skipping a Header in a File."head and tail, along with cat, grep, sort, cut, and uniq, are some of the most commonly used Unix text processing tools out there. If you aren't already familiar with them, you'll soon wonder how you ever got along without them.- , "Skipping a Header in a File"
- , "Sifting Through Files for a String"
- , "Sorting Your Output"
- , "Cutting Out Parts of Your Output"
- , "Removing Duplicate Lines"
- , "Numbering Lines"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Skipping a Header in a File
- InhaltsvorschauYou have a file with one or more header lines and you need to process just the data, and skip the header.Use the tail command with a special argument. For example, to skip the first line of a file:
$ tail +2 lines Line 2 Line 4 Line 5
An argument to tail, which is a number starting dash (-), will specify a line offset relative to the end of the file. Sotail -10file shows the last 10 lines of file, which also happens to be the default if you don't specify anything. But a number starting with a plus (+) sign is an offset relative to the top of the file. Thus,tail+1file gives you the entire file, the same as cat.+2skips the first line, and so on.- man tail
- , "Setting Up a Database with MySQL"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Throwing Output Away
- InhaltsvorschauSometimes you don't want to save the output into a file; in fact, sometimes you don't even want to see it at all.Redirect the output to /dev/null as shown in these examples:
$ find / -name myfile -print 2> /dev/null
or:$ noisy >/dev/null 2>&1
We could redirect the unwanted output into a file, then remove the file when we're done. But there is an easier way. Unix and Linux systems have a special device that isn't real hardware at all, just a bit bucket where we can dump unwanted data. It's called /dev/null and is perfect for these situations. Any data written there is simply thrown away, so it takes up no disk space. Redirection makes it easy.In the first example, only the output going to standard error is thrown away. In the second example, both standard output and standard error are discarded.In rare cases, you may find yourself in a situation where /dev is on a read-only file system (for example, certain information security appliances), in which case you are stuck with the first suggestion of writing to a file and then removing it.- , "Saving Output to Other Files"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving or Grouping Output from Several Commands
- InhaltsvorschauYou want to capture the output with a redirect, but you're typing several commands on one line.
$ pwd; ls; cd ../elsewhere; pwd; ls > /tmp/all.out
The final redirect applies only to the last command, the last ls on that line. All the other output appears on the screen (i.e., does not get redirected).Use braces { } to group these commands together, then redirection applies to the output from all commands in the group. For example:$ { pwd; ls; cd ../elsewhere; pwd; ls; } > /tmp/all.outThere are two very subtle catches here. The braces are actually reserved words, so they must be surrounded by whitespace. Also, the trailing semicolon is required before the closing space.Alternately, you could use parentheses () to tell bash to run the commands in a subshell, then redirect the output of the entire subshell's execution. For example:$ (pwd; ls; cd ../elsewhere; pwd; ls) > /tmp/all.out
While these two solutions look very similar, there are two important differences. The first difference is syntactic, the second is semantic. Syntactically, the braces need to have white space around them and the last command inside the list must terminate with a semicolon. That's not required when you use parentheses. The bigger difference, though, is semantic—what these constructs mean. The braces are just a way to group several commands together, more like a shorthand for our redirecting, so that we don't have to redirect each command separately. Commands enclosed in parentheses, however, run in another instance of the shell, a child of the current shell called a subshell.The subshell is almost identical to the current shell's environment, i.e., variables, including$PATH, are all the same, but traps are handled differently (for more on traps, see , "Trapping Interrupts"). Now here is the big difference in using the subshell approach: because a subshell is used to execute the cd commands, when the subshell exits, your main shell is back where it started, i.e., its current directory hasn't moved, and its variables haven't changed.With the braces used for grouping, you end up in the new directory (Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Connecting Two Programs by Using Output As Input
- InhaltsvorschauYou want to take the output from one program and use it as the input of another program.You could redirect the output from the first program into a temporary file, then use that file as input to the second program. For example:
$ cat one.file another.file > /tmp/cat.out $ sort < /tmp/cat.out ... $ rm /tmp/cat.out
Or you could do all of that in one step by sending the output directly to the next program by using the pipe symbol | to connect them. For example:$ cat one.file another.file | sort
You can also link a sequence of several commands together by using multiple pipes:$ cat my* | tr 'a-z' 'A-Z' | uniq | awk -f transform.awk | wc
By using the pipe symbol we don't have to invent a temporary filename, remember it, and remember to delete it.Programs like sort can take input from standard in (redirected via the < symbol) but they can also take input as a filename—for example:$ sort /tmp/cat.out
rather than redirecting the input into sort:$ sort < /tmp/cat.out
That behavior (of using a filename if supplied, and if not, of using standard input) is a typical Unix/Linux characteristic, and a useful model to follow so that commands can be connected one to another via the pipe mechanism. If you write your programs and shell scripts that way, they will be more useful to you and to those with whom you share your work.Feel free to be amazed at the powerful simplicity of the pipe mechanism. You can even think of the pipe as a rudimentary parallel processing mechanism. You have two commands (programs) running in parallel, sharing data—the output of one as the input to the next. They don't have to run sequentially (where the first runs to completion before the second one starts)—the second one can get started as soon as data is available from the first.Be aware, however, that commands run this way (i.e., connected by pipes), are run in separate subshells. While such a subtlety can often be ignored, there are a few times when the implications of this are important. We'll discuss that in , "Forgetting That Pipelines Make Subshells."Also consider a command such assvn -v log | lessEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving a Copy of Output Even While Using It As Input
- InhaltsvorschauYou want to debug a long sequence of piped I/O, such as:
$ cat my* | tr 'a-z' 'A-Z' | uniq | awk -f transform.awk | wc
How can you see what is happening betweenuniqandawkwithout disrupting the pipe?The solution to these problems is to use what plumbers call a T-joint in the pipes. For bash, that means using the tee command to split the output into two identical streams, one that is written to a file and the other that is written to standard out, so as to continue the sending of data along the pipes.For this example where we'd like to debug a long string of pipes, we insert the tee command between uniq and awk:$ ... uniq | tee /tmp/x.x | awk -f transform.awk ...
The tee command writes the output to the filename specified as its parameter and also write that same output to standard out. In our example, that sends a copy to /tmp/x.x and also sends the same data to awk, the command to which the output of tee is connected via the | pipe symbol.Don't worry about what each different piece of the command line is doing in these examples; we just want to illustrate how tee can be used in any sequence of commands.Let's back up just a bit and start with a simpler command line. Suppose you'd just like to save the output from a long-running command for later reference, while at the same time seeing it on the screen. After all, a command like:find / -name '*.c' -print | less
could find a lot of C source files, so it will likely scroll off the window. Using more or less will let you look at the output in manageable pieces, but once completed they don't let you go back and look at that output without re-running the command. Sure, you could run the command and save it to a file:find / -name '*.c' -print > /tmp/all.my.sources
but then you have to wait for it to complete before you can see the contents of the file. (OK, we know abouttail -fbut that's just getting off topic here.) The tee command can be used instead of the simple redirection of standard output:find / -name '*.c' -print | tee /tmp/all.my.sources
In this example, since the output of tee isn't redirected anywhere, it will print to the screen. But the copy that is diverted into a file will be there for later use (Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Connecting Two Programs by Using Output As Arguments
- InhaltsvorschauWhat if one of the programs to which you would like to connect with a pipe doesn't work that way? For example, you can remove files with the rm command, specifing the files to be removed as parameters to the command:
$ rm my.java your.c their.*
but rm doesn't read from standard input, so you can't do something like:find . -name '*.c' | rm
Since rm only takes its filenames as arguments or parameters on the command line, how can we get the output of a previously-run command (e.g., echo or ls) onto the command line?Use the command substitution feature of bash:$ rm $(find . -name '*.class') $
The $() encloses a command that is run in a subshell. The output from that command is substituted in place of the $() phrase. Newlines in the output are replaced with a space character (actually it uses the first character of $IFS, which is a space by default, during word splitting), so several lines of output become several parameters on the command line.The earlier shell syntax was to use back-quotes instead of $()for enclosing the sub-command. The $() syntax is preferred over the older backward quotes `` syntax because it easier to nest and arguably easier to read. However, you will probably see `` more often than $() especially in older scripts or from those who grew up with the original Bourne or C shells.In our example, the output from find, typically a list of names, will become the arguments to the rm command.Warning: be very careful when doing something like this because rm is very unforgiving. If your find command finds more than you expect, rm will remove it with no recourse. This is not Windows; you cannot recover deleted files from the trashcan. You can mitigate the danger withrm-i, which will prompt you to verify each delete. That's OK on a small number of files, but interminable on a large set.One way to use such a mechanism in bash with greater safety is to run that inner command first by itself. When you can see that you are getting the results that you want, only then do you use it in the command with back-quotes.For example:$ find . -name '*.class' First.class Other.class $ rm $(find . -name '*.class') $
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Multiple Redirects on One Line
- InhaltsvorschauYou want to redirect output to several different places.Use redirection with file numbers to open all the files that you want to use. For example:
$ divert 3> file.three 4> file.four 5> file.five 6> else.where $
wheredivertmight be a shell script with various commands whose output you want to send to different places. For example, you might write divert to contain lines like this:echo option $OPTSTR >&5. That is, our divert shell script could direct its output to various different descriptors which the invoking program can send to different destinations.Similarly, if divert was a C program executable, you could actually write to descriptors 3, 4, 5, and 6 without any need foropen()calls.In an earlier recipe we explained that each file descriptor is indicated by a number, starting at 0 (zero). So standard input is 0, out is 1, and error is 2. That means that you could redirect standard output with the slightly more verbose1>(rather than a simple >) followed by a filename, but there's no need. The shorthand> is fine. It also means that you can have the shell open up any number of arbitrary file descriptors and have them set to write various files so that the program that the shell then invokes from the command line can use these opened file descriptors without further ado.While we don't recommend this technique, it is intriguing.- , "Saving Output to Other Files"
- , "Sending Both Output and Error Messages to Different Files"
- , "Throwing Output Away"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Saving Output When Redirect Doesn't Seem to Work
- InhaltsvorschauYou tried using > but some (or all) of the output still appears on the screen.For example, the compiler is producing some error messages.
$ gcc bad.c bad.c: In function `main': bad.c:3: error: `bad' undeclared (first use in this function) bad.c:3: error: (Each undeclared identifier is reported only once bad.c:3: error: for each function it appears in.) bad.c:3: error: parse error before "c" $
You wanted to capture those messages, so you tried redirecting the output:$ gcc bad.c > save.it bad.c: In function `main': bad.c:3: error: `bad' undeclared (first use in this function) bad.c:3: error: (Each undeclared identifier is reported only once bad.c:3: error: for each function it appears in.) bad.c:3: error: parse error before "c" $
However, it doesn't seem to have redirected anything. In fact, when you examine the file into which you were directing the output, that file is empty (zero bytes long):$ ls -l save.it -rw-r--r-- 1 albing users 0 2005-11-13 15:30 save.it $ cat save.it $
Redirect the error output, as follows:$ gcc bad.c 2> save.it $
The contents of save.it are now the error messages that we had seen before.So what's going on here? Every process in Unix and Linux typically starts out with three open file descriptors: one for input called standard input (STDIN), one for out-put called standard output (STDOUT), and one for error messages called standard error (STDERR). It is really up to the programmer, who writes any particular program, to stick to these conventions and write error messages to standard error and to write the normally expected output to standard out, so there is no guarantee that every error message that you ever get will go to standard error. But most of the long-established utilities are well behaved this way. That is why these compiler messages are not being diverted with a simple > redirect; it only redirects standard output, not standard error.Each file descriptor is indicated by a number, starting at 0. So standard input is 0, output is 1, and error is 2. That means that you could redirect standard output with the slightly more verbose:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Swapping STDERR and STDOUT
- InhaltsvorschauYou need to swap STDERR and STDOUT so you can send STDOUT to a logfile, but then send STDERR to the screen and to a file using the tee command. But pipes only work with STDOUT.Swap STDERR and STDOUT before the pipe redirection using a third file descriptor:
$ ./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile
Whenever you redirect file descriptors, you are duplicating the open descriptor to another descriptor. This gives you a way to swap descriptors, much like how any program swaps two values—by means of a third, temporary holder. It looks like: copy A into C, copy B into A, copy C into B and then you have swapped the values of A and B. For file descriptors, it looks like this:$ ./myscript 3>&1 1>&2 2>&3
Read the syntax3>&1as "give file descriptor 3 the same value as output file descriptor 1." What happens here is that it duplicates file descriptor 1 (i.e., STDOUT) into file descriptor 3, our temporary holding place. Then it duplicates file descriptor 2 (i.e.,STDERR) intoSTDOUT, and finally duplicates file descriptor 3 into STDERR. The net effect is thatSTDERRandSTDOUTfile descriptors have swapped places.So far so good. Now we just change this slightly. Once we've made the copy ofSTDOUT(into file descriptor 3), we are free to redirectSTDOUTinto the logfile we want to have capture the output of our script or other program. Then we can copy the file descriptor from its temporary holding place (fd 3) into STDERR. Adding the pipe will now work because the pipe connects to the (original)STDOUT. That gets us to the solution we wrote above:$ ./myscript 3>&1 1>stdout.logfile 2>&3- | tee -a stderr.logfile
Note the trailing -on the2>&3-term. We do that so that we close file descriptor 3 when we are done with it. That way our program doesn't have an extra open file descriptor. We are tidying up after ourselves.- Linux Server Hacks, First Edition, hack #5 "n>&m: Swap STDOUT and STDERR," by Rob Flickenger (O'Reilly)
- , "Saving Output When Redirect Doesn't Seem to Work"
- , ""Daemonizing" Your Script"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping Files Safe from Accidental Overwriting
- InhaltsvorschauYou don't want to delete the contents of a file by mistake. It can be too easy to mistype a filename and find that you've redirected output into a file that you meant to save.Tell the shell to be more careful, as follows:
$ set -o noclobber $
If you decide you don't want to be so careful after all, then turn the option off:$ set +o noclobber $
Thenoclobberoption tells bash not to overwrite any existing files when you redirect output. If the file to which you redirect output doesn't (yet) exist, everything works as normal, with bash creating the file as it opens it for output. If the file already exists, however, you will get an error message.Here it is in action. We begin by turning the option off, just so that your shell is in a known state, regardless of how your particular system may be configured.$ set +o noclobber $ echo something > my.file $ echo some more > my.file $ set -o noclobber $ echo something > my.file bash: my.file: cannot overwrite existing file $ echo some more >> my.file $
The first time we redirect output to my.file the shell will create it for us. The second time we redirect, bash overwrites the file (it truncates the file to 0 bytes and starts writing from there). Then we set thenoclobberoption and we get an error message when we try to write to that file. As we show in the last part of this example, we can append to the file (using >>) just fine.Beware! Thenoclobberoption only refers to the shell's clobbering of a file when redirecting output. It will not stop other file manipulating actions of other programs from clobbering files (see , "Setting Permissions").$ echo useless data > some.file $ echo important data > other.file $ set -o noclobber $ cp some.file other.file $
Notice that no error occurs; the file is copied over the top of an existing file. That copy is done via the cp command. The shell doesn't get involved.If you're a good and careful typist this may not seem like an important option, but we will look at other recipes where filenames are generated with regular expressions or passed as variables. Those filenames could be used as the filename for output redirection. In such cases, havingEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Clobbering a File on Purpose
- InhaltsvorschauYou like to have
noclobberset, but every once in a while you do want to clobber a file when you redirect output. Can you override bash's good intentions, just once?Use >| to redirect your output. Even ifnoclobberis set, bash ignores its setting and overwrites the file.Consider this example:$ echo something > my.file $ set -o noclobber $ echo some more >| my.file $ cat my.file some more $ echo once again > my.file bash: my.file: cannot overwrite existing file $
Notice that no error message occurs on the second echo, but on the third echo, when we are no longer using the vertical bar but just the plain > character by itself, the shell warns us and does not clobber the existing file.Usingnoclobberdoes not take the place of file permissions. If you don't have write permission in the directory, you won't be able to create the file, whether or not you use the >| construct. Similarly, you must have write permission on the file itself to overwrite that existing file, whether or not you use the >|.So why the vertical bar? Perhaps because the exclamation point was already used by bash for other things, and the vertical bar is close, visually, to the exclamation point. But why would ! be the appropriate symbol? Well, for emphasis of course. Its use in English (with the imperative mood) fits that sense of "do it anyway!" when telling bash to overwrite the file if need be. Secondly, the vi (and ex) editors use the ! in that same meaning in their write (:w!filename) command. Without a !, the editor will complain if you try to overwrite an existing file. With it, you are telling the editor to "do it!"- , "Setting Permissions"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 3: Standard Input
- InhaltsvorschauWhether it is data for a program to crunch, or simple commands to direct the behavior of a script, input is as fundamental as output. The first part of any program is the beginning of the "input/output" yin and yang of computing.You want your shell commands to read data from a file.Use input redirection, indicated by the < character, to read data from a file
$ wc < my.file
Just as the > sends output to a file, so < takes input from a file. The choice and shape of the characters was meant to give a visual clue as to what was going on with redirection. Can you see it? (Think "arrowhead.")Many shell commands will take one or more filenames as arguments, but when no filename is given, will read from standard input. Those commands can then be invoked as either: command filename or as command < filename with the same result. That's the case here with wc, but also with cat and others.It may look like a simple feature, and be familiar if you've used the DOS command line before, but it is a significant feature to shell scripting (which the DOS command line borrowed) and was radical in both its power and simplicity when first introduced.- , "Saving Output to Other Files"
You need input to your script, but don't want a separate file.Use a here-document, with the << characters, redirecting the text from the command line rather than from a file. When put into a shell script, the script file then contains the data along with the script.Here's an example of a shell script in a file we call ext:$ cat ext # # here is a "here" document # grep $1 <<EOF mike x.123 joe x.234 sue x.555 pete x.818 sara x.822 bill x.919 EOF $
It can be used as a shell script for simple phone number lookups:$ ext bill bill x.919 $
or:$ ext 555 sue x.555 $
The grep command looks for occurrences of the first argument in the files that are named, or if no files are named it looks to standard input.A typical use of grep is something like this:$ grep somestring file.txt
or:$ grep myvar *.c
In our ext script we've parameterized the grep by making the string that we're searching for be the parameter of our shell script (Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Input from a File
- InhaltsvorschauYou want your shell commands to read data from a file.Use input redirection, indicated by the < character, to read data from a file
$ wc < my.file
Just as the > sends output to a file, so < takes input from a file. The choice and shape of the characters was meant to give a visual clue as to what was going on with redirection. Can you see it? (Think "arrowhead.")Many shell commands will take one or more filenames as arguments, but when no filename is given, will read from standard input. Those commands can then be invoked as either: command filename or as command < filename with the same result. That's the case here with wc, but also with cat and others.It may look like a simple feature, and be familiar if you've used the DOS command line before, but it is a significant feature to shell scripting (which the DOS command line borrowed) and was radical in both its power and simplicity when first introduced.- , "Saving Output to Other Files"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping Your Data with Your Script
- InhaltsvorschauYou need input to your script, but don't want a separate file.Use a here-document, with the << characters, redirecting the text from the command line rather than from a file. When put into a shell script, the script file then contains the data along with the script.Here's an example of a shell script in a file we call ext:
$ cat ext # # here is a "here" document # grep $1 <<EOF mike x.123 joe x.234 sue x.555 pete x.818 sara x.822 bill x.919 EOF $
It can be used as a shell script for simple phone number lookups:$ ext bill bill x.919 $
or:$ ext 555 sue x.555 $
The grep command looks for occurrences of the first argument in the files that are named, or if no files are named it looks to standard input.A typical use of grep is something like this:$ grep somestring file.txt
or:$ grep myvar *.c
In our ext script we've parameterized the grep by making the string that we're searching for be the parameter of our shell script ($1). Whereas we often think of grep as searching for a fixed string through various different files, here we are going to vary what we search for, but search through the same data every time.We could have put our phone numbers in a file, say phonenumbers.txt, and then used that filename on the line that invokes the grep command:grep $1 phonenumbers.txt
However, that requires two separate files (our script and our datafile) and raises the question of where to put them and how to keep them together.So rather than supplying one or more filenames (to search through), we set up a here-document and tell the shell to redirect standard input to come from that (temporary) document.The << syntax says that we want to create such a temporary input source, and theEOFis just an arbitrary string (you can choose what you like) to act as the terminator of the temporary input. It is not part of the input, but acts as the marker to show where it ends. The regular shell script (if any) resumes after the marker.We also might add-ito the grep command to make our search is case-insensitive. Thus, usinggrep -i $1 << EOFwould allow us to search for "Bill" as well as "bill."Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Preventing Weird Behavior in a Here-Document
- InhaltsvorschauYour here-document is behaving weirdly. You tried to maintain a simple list of donors using the method described previously for phone numbers. So you created a file called donors that looked like this:
$ cat donors # # simple lookup of our generous donors # grep $1 <<EOF # name amt pete $100 joe $200 sam $ 25 bill $ 9 EOF $
But when you tried running it you got weird output:$ ./donors bill pete bill00 bill $ 9 $ ./donors pete pete pete00 $
Turn off the shell scripting features inside the here-document by escaping any or all of the characters in the ending marker:# solution grep $1 <<EOF pete $100 joe $200 sam $ 25 bill $ 9 EOF
It's a very subtle difference, but the <<EOFis replaced with<<\EOF, or<<'EOF'or even<<E\OF—they all work. It's not the most elegant syntax, but it's enough to tell bash that you want to treat the "here" data differently.Normally (i.e., unless we use this escaping syntax), says the bash man page, "…all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion."So what's happening in our original donor script is that the amounts are being interpreted as shell variables. For example,$100is being seen as the shell variable$1followed by two zeros. That's what gives uspete00when we search for "pete" andbill00when we search for "bill."When we escape some or all of the characters of theEOF, bash knows not to do the expansions, and the behavior is the expected behavior:$ ./donors pete pete $100 $
Of course you may want the shell expansion on your data—it can be useful in the correct circumstances, but isn't what we want here. We've found it to be a useful practice to always escape the marker as in<<'EOF'or<<\EOFto avoid unexpected results, unless you know that you really want the expansion to be done on your data.Trailing whitespace (e.g., even just a single blank space) on your closingEOFmarker will cause it not to be recognized as the closing marker. bash will swallow up the rest of your script, treating it as input too, and looking for thatEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Indenting Here-Documents
- InhaltsvorschauThe here-document is great, but it's messing up your shell script's formatting. You want to be able to indent for readability.Use <<- and then you can use tab characters (only!) at the beginning of lines to indent this portion of your shell script.
$ cat myscript.sh ... grep $1 <<-'EOF' lots of data can go here it's indented with tabs to match the script's indenting but the leading tabs are discarded when read EOF ls ... $
The hyphen just after the << is enough to tell bash to ignore the leading tab characters. This is for tab characters only and not arbitrary white space. This is especially important with theEOFor any other marker designation. If you have spaces there, it will not recognize theEOFas your ending marker, and the "here" data will continue through to the end of the file (swallowing the rest of your script). Therefore, you may want to always left-justify theEOF(or other marker) just to be safe, and let the formatting go on this one line.Just as trailing whitespace of any kind on your closing EOF delimiter prevents it from being recognized as the closing delimiter (see the warning in , "Preventing Weird Behavior in a Here-Document"), so too will using a leading character other than just the tab character. If your script indents with spaces or a combination of spaces and tabs, don't use that technique on here-documents. Either use just tabs, or keep it all flush left. Also, watch out for text editors that automatically replace tabs with spaces.- , "Keeping Your Data with Your Script"
- , "Preventing Weird Behavior in a Here-Document"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting User Input
- InhaltsvorschauYou need to get input from the user.Use the
readstatement:read
or:read - p "answer me this " ANSWER
or:read PRE MID POST
In its simplest form, areadstatement with no arguments will read user input and place it into the shell variableREPLY.If you want bash8 to print a prompt string before reading the input, use the-poption. The next word following the-pwill be the prompt, but quoting allows you to supply multiple words for a prompt. Remember to end the prompt with punctuation and/or a blank, as the cursor will wait for input right at the end of the prompt string.If you supply multiple variable names on thereadstatement, then thereadwill parse the input into words, assigning them in order. If the user enters fewer words, the extra variables will be set blank. If the user enters more words than there are variables on thereadstatement, then all of the extra words will be part of the last variable in the list.- help read
- building robust code
- , "Prompting for a Password"
- , "Looping with a read"
- , "Parsing Text with a read Statement"
- , "Validating Input"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Yes or No Input
- InhaltsvorschauYou need to get a simple yes or no input from the user, and you want to be as user-friendly as possible. In particular, you do not want to be case sensitive, and you want to provide a useful default if the user presses the Enter key.If the actions to take are simple, use this self-contained function:
# cookbook filename: func_choose # Let the user make a choice about something and execute code based on # the answer # Called like: choose <default (y or n)> <prompt> <yes action> <no action> # e.g. choose "y" \ # "Do you want to play a game?" \ # /usr/games/GlobalThermonucularWar \ # 'printf "%b" "See you later Professor Falkin."' >&2 # Returns: nothing function choose { local default="$1" local prompt="$2" local choice_yes="$3" local choice_no="$4" local answer read -p "$prompt" answer [ -z "$answer" ] && answer="$default" case "$answer" in [yY1] ) exec "$choice_yes" # error check ;; [nN0] ) exec "$choice_no" # error check ;; * ) printf "%b" "Unexpected answer '$answer'!" >&2 ;; esac } # end of function chooseIf the actions are complicated, use this function and handle the results in your main code.# cookbook filename: func_choice.1 # Let the user make a choice about something and return a standardized # answer. How the default is handled and what happens next is up to # the if/then after the choice in main # Called like: choice <promtp> # e.g. choice "Do you want to play a game?" $ Returns: global variable CHOICE function choice { CHOICE='' local prompt="$*" local answer read -p "$prompt" answer case "$answer" in [yY1] ) CHOICE='y';; [nN0] ) CHOICE='n';; * ) CHOICE="$answer";; esac } # end of function choiceThe following code calls thechoicefunction to prompt for and verify a package date. Assuming$THISPACKAGEis set, the function displays the date and asks for verification. If the user types y, Y, or Enter, then that date is accepted. If the user enters a new date, the function loops and verifies it (for a different treatment of this problem, see , "Figuring Out Date and Time Arithmetic"):Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Selecting from a List of Options
- InhaltsvorschauYou need to provide the user with a list of options to choose from and you don't want to make them type any more than necessary.Use bash's built-in
selectconstruct to generate a menu, then have the user choose by typing the number of the selection:# cookbook filename: select_dir directorylist="Finished $(ls /)" PS3='Directory to process? ' # Set a useful select prompt until [ "$directory" == "Finished" ]; do printf "%b" "\a\n\nSelect a directory to process:\n" >&2 select directory in $directorylist; do # User types a number which is stored in $REPLY, but select # returns the value of the entry if [ "$directory" = "Finished" ]; then echo "Finished processing directories." break elif [ -n "$directory" ]; then echo "You chose number $REPLY, processing $directory..." # Do something here break else echo "Invalid selection!" fi # end of handle user's selection done # end of select a directory done # end of while not finished
Theselectfunction makes it trivial to present a numbered list to the user on STDERR, from which they may make a choice. Don't forget to provide an "exit" or "finished" choice.The number the user typed is returned in$REPLY, and the value of that entry is returned in the variable you specified in theselectconstruct.- help select
- help read
- , "Getting Yes or No Input"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Prompting for a Password
- InhaltsvorschauYou need to prompt the user for a password, but you don't want it echoed on the screen.
read -s -p "password: " PASSWD printf "%b" "\n"
The-soption tells thereadcommand not to echo the characters typed (s is for silent) and the-poption says that the next argument is the prompt to be displayed prior to reading input.The line of input that is read from the user is put into the environment variable named$PASSWD.We followreadwith aprintfto print out a newline. Theprintfis necessary becauseread -sturns off the echoing of characters. With echoing disabled, when the user presses the Enter key, no newline is echoed and any subsequent output would appear on the same line as the prompt. Printing the newline gets us to the next line, as you would expect. It may even be handy for you to write the code all on one line to avoid intervening logic; putting it on one line also prevents mistakes should you cut and paste this line elsewhere:read -s -p "password: " PASSWD ; printf "%b" "\n"
Be aware that if you read a password into an environment variable it is in memory in plain text, and thus may be accessed via a core dump or /proc/core. It is also in the process environment, which may be accessible by other processes. You may be better off using certificates with SSH, if possible. In any case, it is wise to assume that root and possibly other users on the machine may gain access to the password, so you should handle the situation accordingly.Some older scripts may usesto disable the screen echo while a password is being entered. The problem with that is this if the user breaks the script, echo will still be off. Experienced users will know to typestty saneto fix it, but it's very confusing. If you still need to use this method, set a trap to turn echo back on when the script terminates. See , "Trapping Interrupts."- help read
- , "Trapping Interrupts"
- , "Leaking Passwords into the Process List"
- , "Using Passwords in Scripts"
- , "Using SSH Without a Password"
- , "Making Your Terminal Sane Again"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 4: Executing Commands
- InhaltsvorschauThe main purpose of bash (or of any shell) is to allow you to interact with the computer's operating system so that you can accomplish whatever you need to do. Usually that involves launching programs, so the shell takes the commands you type, determines from that input what programs need to be run, and launches them for you.Let's take a look at the basic mechanism for launching jobs and explore some of the features bash offers for launching programs in the foreground or the background, sequentially or in parallel, indicating whether programs succeeded and more.You need to run a command on a Linux or Unix system.Use bash and type the name of the command at the prompt.
$ someprogThis seems rather simple, and in a way it is, but a lot goes on behind the scenes that you never see. What's important to understand about bash is that its basic operation is to load and execute programs. All the rest is just window dressing to get ready to run programs. Sure there are shell variables and control statementsforlooping andif/then/elsebranching, and there are ways to control input and output, but they are all icing on the cake of program execution.So where does it get the program to run?bash will use a shell variable called$PATHto locate your executable. The$PATHvariable is a list of directories. The directories are separated by colons (:). bash will search in each of those directories for a file with the name that you specified. The order of the directories is important—bash looks at the order in which the directories are listed in the variable, and takes the first executable found.$ echo $PATH /bin:/usr/bin:/usr/local/bin:. $
In the$PATHvariable shown above, four directories are included. The last directory in that list is just a single dot (called the dot directory, or just dot), which represents the current directory. The dot is the name of the directory found within every directory on a Linux or Unix file system—wherever you are, that's the directory to which dot refers. For example, when you copy a file from someplace to dot (i.e.,cp /other/place/file.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Any Executable
- InhaltsvorschauYou need to run a command on a Linux or Unix system.Use bash and type the name of the command at the prompt.
$ someprogThis seems rather simple, and in a way it is, but a lot goes on behind the scenes that you never see. What's important to understand about bash is that its basic operation is to load and execute programs. All the rest is just window dressing to get ready to run programs. Sure there are shell variables and control statementsforlooping andif/then/elsebranching, and there are ways to control input and output, but they are all icing on the cake of program execution.So where does it get the program to run?bash will use a shell variable called$PATHto locate your executable. The$PATHvariable is a list of directories. The directories are separated by colons (:). bash will search in each of those directories for a file with the name that you specified. The order of the directories is important—bash looks at the order in which the directories are listed in the variable, and takes the first executable found.$ echo $PATH /bin:/usr/bin:/usr/local/bin:. $
In the$PATHvariable shown above, four directories are included. The last directory in that list is just a single dot (called the dot directory, or just dot), which represents the current directory. The dot is the name of the directory found within every directory on a Linux or Unix file system—wherever you are, that's the directory to which dot refers. For example, when you copy a file from someplace to dot (i.e.,cp /other/place/file.), you are copying the file into the current directory. By having the dot directory listed in our path, bash will look for commands not just in those other directories, but also in the current directory (.).Many people feel that putting dot on your$PATHis too great a security risk—some-one could trick you and get you to run their own (malicious) version of a command in place of one that you were expecting. Now if dot were listed first, then someone else's version of ls would supersede the normal ls command and you might unwittingly run that command. Don't believe us? Try this:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Telling If a Command Succeeded or Not
- InhaltsvorschauYou need to know whether the command you ran succeeded.The shell variable $? will be set with a non-zero value if the command fails—provided that the programmer who wrote that command or shell script followed the established convention:
$ somecommand it works... $ echo $? 0 $ badcommand it fails... $ echo $? 1 $
The exit status of a command is kept in the shell variable referenced with $?. Its value can range from 0 to 255. When you write a shell script, it's a good idea to have your script exit with a non-zero value if you encounter an error condition. (Just keep it below 255, or the numbers will wrap around.) You return an exit status with theexitstatement (e.g.,exit 1orexit 0). But be aware that you only get one shot at reading the exit status:$ badcommand it fails... $ echo $? 1 $ echo $? 0 $
Why does the second time give us0as a result? Because the second time is reporting on the status of the immediately preceding echo command. The first time we typedecho $?it returneda 1, which was the return value of bad command. But the echo command itself succeeds, therefore the new, most-recent status is success (i.e.,a 0value). So you only get one chance to check it. Therefore, many shell scripts will immediately assign the status to another shell variable, as in:$ badcommand it fails... $ STAT=$? $ echo $STAT 1 $ echo $STAT 1 $
We can keep the value around in the variable$STATand check its value later on.Although we're showing this in command-line examples, the real use of variables like $? comes in writing scripts. You can usually see if a command worked or not if you are watching it run on your screen. But in a script, the commands may be running unattended.One of the great features of bash is that the scripting language is identical to commands as you type them at a prompt in a terminal window. This makes it much easier to check out syntax and logic as you write your scripts.The exit status is more often used in scripts, and often inifstatements, to take different actions depending on the success or failure of a command. Here's a simple example for now, but we will revisit this topic in future recipes:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Several Commands in Sequence
- InhaltsvorschauYou need to run several commands, but some take a while and you don't want to wait for the last one to finish before issuing the next command.There are three solutions to this problem, although the first is rather trivial: just keep typing. A Linux or Unix system is advanced enough to be able to let you type while it works on your previous commands, so you can simply keep typing one command after another.Another rather simple solution is to type those commands into a file and then tell bash to execute the commands in the file—i.e., a simple shell script.Assume that we want to run three commands: long, medium, and short, each of whose execution time is reflected in its name. We need to run them in that order, but don't want to wait around for long to finish before starting the other commands. We could use a shell script (aka batch file). Here's a primitive way to do that:
$ cat > simple.script long medium short ^D # Ctrl-D, not visible $ bash ./simple.script
The third, and arguably best, solution is to run each command in sequence. If you want to run each program, regardless if the preceding ones fail, separate them with semicolons:$ long ; medium ; short
If you only want to run the next program if the preceding program worked, and all the programs correctly set exit codes, separate them with double-ampersands:$ long && medium && short
The cat example was just a very primitive way to enter text into a file. We redirect the output from the command into the file named simple.script (for more on redirecting output, see ). Better you should use a real editor, but such things are harder to show in examples like this. From now on, when we want to show a script, we'll just either show the text as disembodied text not on a command line, or we will start the example with a command likecat filenameto dump the contents of the file to the screen (rather than redirecting output from our typing into the file), and thus display it in the example.The main point of this simple solution is to demonstrate that more than one command can be put on theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Several Commands All at Once
- InhaltsvorschauYou need to run three commands, but they are independent of each other, and don't need to wait for each other to complete.You can run a command in the background by putting an ampersand (&) at the end of the command line. Thus, you could fire off all three jobs in rapid succession as follows:
$ long & [1] 4592 $ medium & [2] 4593 $ short $
Or better yet, you can do it all on one command line:$ long & medium & short [1] 4592 2] 4593 $
When we run a command in the background (there really is no such place in Linux), all that really means is that we disconnect keyboard input from the command and the shell doesn't wait for the command to complete before it gives another prompt and accepts more command input. Output from the job (unless we take explicit action to do otherwise) will still come to the screen, so all three jobs will be interspersing output to the screen.The odd bits of numerical output are the job number in square brackets, followed by the process ID of the command that we just started in the background. In our example, job 1 (process 4592) is the long command, and job 2 (process 4593) is medium.We didn't put short into the background since we didn't put an ampersand at the end of the line, so bash will wait for it to complete before giving us the shell prompt (the$).The job number or process ID can be used to provide limited control over the job. You can kill the long job withkill %1(since its job number was1). Or you could specify the process number (e.g.,kill 4592) with the same deadly results.You can also use the job number to reconnect to a background job. Connect it back to the foreground withfg %1. But if you only had one job running in the background, you wouldn't even need the job number, justfgby itself.If you start a job and then realize it will take longer to complete than you thought, you can pause it using Ctrl-Z, which will return you to a prompt. You can then type bg to un-pause the job so it will continue running in the background. This is basically adding a trailing & after the fact.- on redirecting output
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Deciding Whether a Command Succeeds
- InhaltsvorschauYou need to run some commands, but you only want to run certain commands if certain other ones succeed. For example, you'd like to change directories (using the cd command) into a temporary directory and remove all the files. However, you don't want to remove any files if the cd fails (e.g., if permissions don't allow you into the directory, or if you spell the directory name wrong).We can use the exit status ($?) of the cd command in combination with an
ifstatement to do the rm only if the cd was successful.cd mytmp if (( $? )); then rm * ; fi
Obviously, you wouldn't need to do this if you were typing the commands by hand. You would see any error messages from the cd command, and thus you wouldn't type the rm command. But scripting is another matter, and this test is very well worth doing to make sure that you don't accidentally erase all the files in the directory where you are running.Let's say you ran that script from the wrong directory, one that didn't have a subdirectory named mytmp. When it runs, the cd would fail, so the current directory remains unchanged. Without the if check (the cd having failed) the script would just continue on to the next statement. Running therm* would remove all the files in your current directory. Ouch. Theifis worth it.So how does$?get its value? It is the exit code of the command. For C Language programmers, you'll recognize this as the value of the argument supplied to theexit( )function; e.g.,exit(4); would return a 4. For the shell, zero is considered success and a non-zero value means failure.If you're writing bash scripts, you'll want to be sure that your bash scripts explicitly set return values, so that $? is set properly from your script. If you don't, the value set will be the value of the last command run, which you may not want as your result.- , "Telling If a Command Succeeded or Not"
- , "Using Fewer if Statements"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Fewer if Statements
- InhaltsvorschauAs a conscientious programmer, you took to heart what we described in the previous recipe, , "Deciding Whether a Command Succeeds." You applied the concept to your latest shell script, and now you find that the shell script is unreadable, if with all those
ifstatements checking the return code of every command. Isn't there an alternative?Use the double-ampersand operator in bash to provide conditional execution:$ cd mytmp && rm *
Two commands separated by the double ampersands tells bash to run the first command and then to run the second command only if the first command succeeds (i.e., its exit status is0). This is very much like using anifstatement to check the exit status of the first command in order to protect the running of the second command:cd mytmp if (( $? )); then rm * ; fi
The double ampersand syntax is meant to be reminiscent of the logical and operator in C Language. If you know your logic (and your C) then you'll recall that if you are evaluating the logical expressionA AND B, then the entire expression can only be true if both (sub)expressionAand (sub)expressionBevaluate to true. If either one is false, the whole expression is false. C Language makes use of this fact, and when you code an expression likeif (A && B){ … }, it will evaluate expressionAfirst. If it is false, it won't even bother to evaluateBsince the overall outcome (false) has already been determined (byAbeing false).So what does this have to do with bash? Well, if the exit status of the first command (the one to the left of the &&) is non-zero (i.e., failed) then it won't bother to evaluate the second expression—i.e., it won't run the other command at all.If you want to be thorough about your error checking, but don't wantifstatements all over the place, you can have bash exit any time it encounters a failure (i.e., a non-zero exit status) from every command in your script (except inwhileloops andifstatements where it is already capturing and using the exit status) by setting the-eflag.set -e cd mytmp rm *
Setting the-eflag will cause the shell to exit when a command fails. If theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Long Jobs Unattended
- InhaltsvorschauYou ran a job in the background, then exited the shell and went for coffee. When you came back to check, the job was no longer running and it hadn't completed. In fact, your job hadn't progressed very far at all. It seems to have quit as soon as you exited the shell.If you want to run a job in the background and expect to exit the shell before the job completes, then you need tonohup the job:
$ nohup long & nohup: appending output to `nohup.out' $
When you put the job in the background (via the &), it is still a child process of the bash shell. When you exit an instance of the shell, bash sends a hangup (hup) signal to all of its child processes. That's why your job didn't run for very long. As soon as you exited bash, it killed your background job. (Hey, you were leaving; how was it supposed to know?)The nohup command simply sets up the child process to ignore hang-up signals. You can still kill a job with the kill command, because kill sends aSIGTERMsignal not aSIGHUPsignal. But with nohup, bash won't inadvertently kill your job when you exit.The message that nohup gives about appending your output is just nohup trying to be helpful. Since you are likely to exit the shell after issuing a nohup command, your output destination will likely go away—i.e., the bash session in your terminal window would no longer be active. So, where would the job be able to write? More importantly, writing to a non-existent destination would cause a failure. So nohup redirects the output for you, appending it (not overwriting, but adding at the end) to a file named nohup.out in the current directory. You can explicitly redirect the out-put elsewhere on the command line and nohup is smart enough to detect that this has happened and doesn't use nohup.out for your output.- for various recipes on redirecting output, since you probably want to do that for a background job
- , ""Daemon-izing" Your Script"
- , "Recovering Disconnected Sessions Using screen"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Displaying Error Messages When Failures Occur
- InhaltsvorschauYou need your shell script to be verbose about failures. You want to see error messages when commands don't work, but
ifstatements tend to distract from the visual flow of statements.A common idiom among some shell programmers is to use the||with commands to spit out debug or error messages. Here's an example:cmd || printf "%b" "cmd failed. You're on your own\n"
Similar to how the&&didn't bother to evaluate the second expression if the first was false, the||tells the shell not to bother to evaluate the second expression if the first one is true (i.e., succeeds). As with&&, the||syntax harkens back to logic and C Language where the outcome is determined (astrue) if the first expression inA OR Bevaluates to true—so there's no need to evaluate the second expression. In bash, if the first expression returns 0 (i.e., succeeds) then it just continues on. Only if the first expression (i.e., exit value of the command) returns a non-zero value must it evaluate the second part, and thus run the other command.Warning—don't be fooled by this:cmd || printf "%b" "FAILED.\n" ; exit 1
The exit will be executed in either case! The OR is only between those two commands. If we want to have the exit happen only on error, we need to group it with the printf so that both are considered as a unit. The desired syntax would be:cmd || { printf "%b" "FAILED.\n" ; exit 1 ; }Due to an oddity of bash syntax, the semicolon after the last command and just before the } is required, and that closing brace must be separated by whitespace from the surrounding text.- , "Saving or Grouping Output from Several Commands"
- , "Using Fewer if Statements" for an explanation of && syntax
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Commands from a Variable
- InhaltsvorschauYou want to run different commands in your script depending on circumstances. How can you vary which commands run?There are many solutions to this problem—it's what scripting is all about. In coming chapters we'll discuss various programming logic that can be used to solve this problem, such as
if/then/else, casestatements, and more. But here's a slightly different approach that reveals something about bash. We can use the contents of a variable (more on those in ) not just for parameters, but also for the command itself.FN=/tmp/x.x PROG=echo $PROG $FN PROG=cat $PROG $FN
We can assign the program name to a variable (here we use$PROG), and then when we refer to that variable in the place where a command name would be expected, it uses the value of that variable ($PROG) as the command to run. The bash shell parses the command line, substitutes the values of its variables and takes the result of all the substitutions and then treats that as the command line, as if it had been typed that way verbatim.Be careful about the variable names you use. Some programs such as InfoZip use environment variables such as$ZIPand$UNZIPto pass settings to the program itself. So if you do something likeZIP='/usr/bin/ zip', you can spend days pulling your hair out wondering why it works fine from the command line, but not in your script. Trust us. We learned this one the hard way. Also, RTFM.- , "Setting a Secure $PATH"
- , "Creating Self-Contained, Portable RC Files"
- , "Getting Started with a Custom Configuration"
- for a descripton of all the various substitutions that are preformed on a command line; you'll want to read a few more chapters before tackling that subject
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running All Scripts in a Directory
- InhaltsvorschauYou want to run a series of scripts, but the list keeps changing; you're always adding new scripts, but you don't want to continuously modify a master list.Put the scripts you want to run in a directory, and let bash run everything that it finds. Instead of keeping a master list, simply look at the contents of that directory. Here's a script that will run everything it finds in a directory:
for SCRIPT in /path/to/scripts/dir/* do if [ -f $SCRIPT -a -x $SCRIPT ] then $SCRIPT fi done
We will discuss theforloop and theifstatement in greater detail in , but this gives you a taste. The variable$SCRIPTwill take on successive values for each file that matches the wildcard pattern *, which matches everything in the current directory (except invisible dot files, which begin with a period). If it is a file (the-ftest) and has execute permissions set (the-xtest), the shell will then try to run that script.In this simple example, we have provided no way to specify any arguments to the scripts as they are executed. This simple script may work well for your personal needs, but wouldn't be considered robust; some might consider it downright dangerous. But we hope it gives you an idea of what lies ahead: some programming-language-style scripting capabilities.- for more about
forloops andifstatements
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 5: Basic Scripting: Shell Variables
- Inhaltsvorschaubash shell programming is a lot like any kind of programming, and that includes having variables—containers that hold strings and numbers, which can be changed, compared, and passed around. bash variables have some very special operators that can be used when you refer to the variable. bash also has some important built-in variables, ones that provide important information about the other variables in your script. This chapter takes a look at bash variables and some special mechanisms for referencing variables, and shows how they can be put to use in your scripts.Variables in a bash script are often written as all-uppercase names, though that is not required—just a common practice. You don't need to declare them; just use them where you want them. They are basically all of type string, though some bash operations can treat their contents as a number. They look like this in use:
# trivial script using shell variables # (but at least it is commented!) MYVAR="something" echo $MYVAR # similar but with no quotes MY_2ND=anotherone echo $MY_2ND # quotes are needed here: MYOTHER="more stuff to echo" echo $MYOTHER
There are two significant aspects of bash variable syntax that may not be intuitively obvious regarding shell variables. First, on the assignment, thename=valuesyntax is straightforward enough, but there cannot be any spaces around the equal sign.Let's consider for a moment why this is the case. Remember that the basic semantics of the shell is to launch programs—you name the program on the command line and that is the program that gets launched. Any words of text that follow after it on the command line are passed along as arguments to the program. For example when you type:$ ls filename
the wordlsis the name of the command andfilenameis the first and only argument in this example.Why is that relevant? Well, consider what a variable assignment in bash would look like if you allowed spaces around the equal sign, like this:MYVAR = something
Can you see that the shell would have a hard time distinguishing between the name of a command to invoke (like theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Documenting Your Script
- InhaltsvorschauBefore we say one more word about shell scripts or variables, we have to say something about documenting your scripts. After all, you need to be able to understand your script even when several months have passed since you wrote it.Document your script with comments. The # character denotes the beginning of a comment. All the characters after it on that line are ignored by the shell.
# # This is a comment. # # Use comments frequently. # Comments are your friends.
Some people have described shell syntax, regular expressions, and other parts of shell scripting as write only syntax, implying that it is nearly impossible to understand the intricacies of many shell scripts.One of your best defenses against letting your shell scripts fall into this trap is the liberal use of comments (another is the use of meaningful variable names). It helps to put a comment before strange syntax or terse expressions.# replace the semi with a blank NEWPATH=${PATH/;/ } # # switch the text on either side of a semi sed -e 's/^\(.*\);\(.*\)$/\2;\1/' < $FILEComments can even be typed in at the command prompt with an interactive shell. This can be turned off, but it is on by default. There may be a few occasions when it is useful to make interactive comments.- "shopt Options" in gives the option for turning interactive comments on or off
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Embedding Documentation in Shell Scripts
- InhaltsvorschauYou want a simple way to provide formatted end-user documentation (e.g., man or html pages) for your script. You want to keep both code and documentation markup in the same file to simplify updates, distribution, and revision control.Embed documentation in the script using the "do nothing" built-in (a colon) and a here-document:
#!/usr/bin/env bash # cookbook filename: embedded_documentation echo 'Shell script code goes here' # Use a : NOOP and here document to embed documentation, : <<'END_OF_DOCS' Embedded documentation such as Perl's Plain Old Documentation (POD), or even plain text here. Any accurate documentation is better than none at all. Sample documentation in Perl's Plain Old Documentation (POD) format adapted from CODE/ch07/Ch07.001_Best_Ex7.1 and 7.2 in Perl Best Practices. =head1 NAME MY~PROGRAM--One line description here =head1 SYNOPSIS MY~PROGRAM [OPTIONS] <file> =head1 OPTIONS -h = This usage. -v = Be verbose. -V = Show version, copyright and license information. =head1 DESCRIPTION A full description of the application and its features. May include numerous subsections (i.e. =head2, =head3, etc.) [...] =head1 LICENSE AND COPYRIGHT =cut END_OF_DOCS
Then to extract and use that POD documentation, try these commands.# To read on-screen, automatically paginated $ perldoc myscript # Just the "usage" sections $ pod2usage myscript # Create an HTML version $ pod2html myscript > myscript.html # Create a man page $ pod2man myscript > myscript.1
Any plain text documentation or mark-up can be used this way, either interspersed throughout the code or better yet collected at the end of the script. Since computer systems that have bash will probably also have Perl, its Plain Old Documentation (POD) may be a good choice. Perl usually comes with pod2* programs to convert POD to HTML, LaTeX, man, text, and usage files.Damian Conway's Perl Best Practices (O'Reilly) has some excellent library module and application documentation templates that could be easily translated into any documentation format including plain text. In that book, seeEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Promoting Script Readability
- InhaltsvorschauYou'd like to make your script as readable as possible for ease of understanding and future maintenance.
- Document your script as noted in , "Documenting Your Script" and , "Embedding Documentation in Shell Scripts"
- Indent and use vertical whitespace wisely
- Use meaningful variable names
- Use functions, and give them meaningful names
- Break lines at meaningful places at less than 76 characters or so
- Put the most meaningful bits to the left
Document your intent, not the trivial details of the code. If you follow the rest of the points, the code should be pretty clear. Write reminders, provide sample data layouts or headers, and make a note of all the details that are in your head now, as you write the code. But document the code itself too if it is subtle or obscure.We recommend indenting using four spaces per level, with no tabs and especially no mixed tabs. There are many reasons for this, though it often is a matter of personal preference or company standards. After all, four spaces is always four spaces, no matter how your editor (excepting proportional fonts) or printer is set. Four spaces is big enough to be easily visible as you glance across the script but small enough that you can have several levels of indenting without running the lines off the right side of your screen or printed page. We also suggest indenting continued lines with two additional spaces, or as needed, to make the code the most clear.Use vertical white space, with separators if you like them, to create blocks of similar code. Of course you'll do that with functions as well.Use meaningful names for variables and functions, and spell them out. The only time$ior$xis ever acceptable is in aforloop. You may think that short, cryptic names are saving you time and typing now, but we guarantee that you will lose that time 10- or 100-fold somewhere down the line when you have to fix or modify that script.Break long lines at around 76 characters. Yes, we know that most of the screens (or rather terminal programs) can do a lot more than that. But 80 character paper and screens are still the default, and it never hurts to have some white space to the right of the code. Constantly having to scroll to the right or having lines wrap on the screen or printout is annoying and distracting. Don't cause it.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Separating Variable Names from Surrounding Text
- InhaltsvorschauYou need to print a variable along with other text. You are using the dollar sign in referring to the variable. But how do you distinguish the end of the variable name from other text that follows? For example, say you wanted to use a shell variable as part of a filename, as in:
for FN in 1 2 3 4 5 do somescript /tmp/rep$FNport.txt done
How will the shell read that? It will think that the variable name starts with the $ and ends with the punctuation. In other words, it will think that$FNportis the variable name, not the intended$FN.Use the full syntax for a variable reference, which includes not just the dollar sign, but also braces around the variable name:somescript /tmp/rep${SUM}bay.txtBecause shell variables are only alphanumeric characters, there are many instances where you won't need to use the braces. Any whitespace or punctuation (except underscore) provides enough of a clue to where the variable name ends. But when in doubt, use the braces.- , "Using Shell Quoting"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Exporting Variables
- InhaltsvorschauYou defined a variable in one script, but when you called another script it didn't know about the variable.Export variables that you want to pass on to other scripts:
export MYVAR export NAME=value
Sometimes it's a good thing that one script doesn't know about the other script's variables. If you called a shell script from within aforloop in the first script, you wouldn't want the second script messing up the iterations of yourforloop.But sometimes you do want the information passed along. In those cases, you can export the variable so that its value is passed along to any other program that it invokes.If you want to see a list of all the exported variables, just type the built-in command env (orexport -p) for a list of each variable and its value. All of these are available for your script when it runs. Many have already been set up by the bash startup scripts (see for more on configuring and customizing bash).You can have the export statement just name the variable that will be exported. Though the export statement can be put anywhere prior to where you need the value to be exported, script writers often group these export statements together like variable declarations at the front of a script. You can also make theexportpart of any variable assignment, though that won't work in old versions of the shell.Once exported, you can assign repeatedly to the variable without exporting it each time. So, sometimes you'll see statements like:export FNAME export SIZE export MAX ... MAX=2048 SIZE=64 FNAME=/tmp/scratch
and at other times you'll see:export FNAME=/tmp/scratch export SIZE=64 export MAX=2048 ... FNAME=/tmp/scratch2 ... FNAME=/tmp/stillexported
One word of caution: the exported variables are, in effect, call by value. Changing the value of the exported value in the called script does not change that variable's value back in the calling script.This begs the question: "How would you pass back a changed value from the called script?" Answer: you can't.Is there a better answer? Unfortunately, there isn't. You can only design your scripts so that they don't need to do this. What mechanisms have people used to cope with this limitation?Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Seeing All Variable Values
- InhaltsvorschauHow can I see which variables have been exported and what values they have? Do I have to echo each one by hand? How would I tell if they are exported?Use the set command to see the value of all variables and function definitions in the current shell.Use the env (or
export -p) command to see only those variables that have been exported and would be available to a subshell.The set command, with no other arguments, produces (on standard out) a list of all the shell variables currently defined along with their values, in aname=valueformat. The env command is similiar. If you run either, you will find a rather long list of variables, many of which you might not recognize. Those variables have been created for you, as part of the shell's startup process.The list produced by env is a subset of the list produced by set, since not all variables are exported.If there are particular variables or values that are of interest, and you don't want the entire list, just pipe it into a grep command. For example:$ set | grep MY
will show only those variables whose name or value has the two-character sequence MY somewhere in it.- help set
- help export
- man env
- for more on configuring and customizing bash
- for reference lists for all of the built-in shell variables
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Parameters in a Shell Script
- InhaltsvorschauYou also want users to be able to invoke your script with a parameter. You could require that users set a shell variable, but that seems clunky. You also need to pass data to another script. You could agree on environment variables, but that ties the two scripts together too closely.Use command-line parameters. Any words put on the command line of a shell script are available to the script as numbered variables:
# simple shell script echo $1
The script will echo the first parameter supplied on the command line when it is invoked. Here it is in action:$ cat simplest.sh # simple shell script echo ${1} $ ./simplest.sh you see what I mean you $ ./simplest.sh one more time one $The other parameters are available as${2}, ${3}, ${4}, ${5}, and so on. You don't need the braces for the single-digit numbers, except to separate the variable name from the surrounding text. Typical scripts have only a handful of parameters, but when you get to${10}you better use the braces or else the shell will interpret that as${1}followed immediately by the literal string0as we see here:$ cat tricky.sh echo $1 $10 ${10} $ ./tricky.sh I II III IV V VI VII VIII IX X XI I I0 X $The tenth argument has the valueXbut if you write$10in your script, then the shell will give you$1, the first parameter, followed immediately by a zero, the literal character that you put next to the$1in yourechostatement.- , "Separating Variable Names from Surrounding Text"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Looping Over Arguments Passed to a Script
- InhaltsvorschauYou want to take some set of actions for a given list of arguments. You could write your shell script to do that for one argument and use
$1to reference the parameter. But what if you'd like to do this for a whole bunch of files? You would like to be able to invoke your script like this:actall *.txt
knowing that the shell will pattern match and build a list of filenames that match the*.txtpattern (any filename ending with.txt).Use the shell special variable $* to refer to all of your arguments, and use that in aforloop like this:#!/usr/bin/env bash # cookbook filename: chmod_all.1 # # change permissions on a bunch of files # for FN in $* do echo changing $FN chmod 0750 $FN done
The variable$FNis our choice; we could have used any shell variable name we wanted there. The $* refers to all the arguments supplied on the command line. For example, if the user types:$ ./actall abc.txt another.txt allmynotes.txt
the script will be invoked with$1equal to abc.txt and$2equal to another.txt and$3equal to allmynotes.txt, but $* will be equal to the entire list. In other words, after the shell has substituted the list for $* in theforstatement, it will be as if the script had read:for FN in abc.txt another.txt allmynotes.txt do echo changing $FN chmod 0750 $FN done
Theforloop will take one value at a time from the list, assign it to the variable$FNand proceed through the list of statements between the do and thedone. It will then repeat that loop for each of the other values.But you're not finished yet! This script works fine when filenames have no spaces in them, but sometimes you encounter filenames with spaces. Read the next two recipes to see how this script can be improved.- help for
- , "Looping with a Count"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Handling Parameters with Blanks
- InhaltsvorschauYou wrote a script that took a filename as a parameter and it seemed to work, but then one time your script failed. The filename, it turns out, had an embedded blank.You'll need to be careful to quote any shell parameters that might contain filenames. When referring to a variable, put the variable reference inside double quotes.Thanks a lot, Apple! Trying to be user friendly, they popularized the concept of space characters as valid characters in filenames, so users could name their files with names like My Report and Our Dept Data instead of the ugly and unreadable MyReport and Our_Dept_Data. (How could anyone possibly understand what those old-fashioned names meant?) Well, that makes life tough for the shell, because the space is the fundamental separator between words, and so filenames were always kept to a single word. Not so anymore.So how do we handle this?Where a shell script once had simply
ls -l $1, it is better to writels -l "$1"with quotes around the parameter. Otherwise, if the parameter has an embedded blank, it will be parsed into separate words, and only part of the name will be in$1. Let's show you how this doesn't work:$ cat simpls.sh # simple shell script ls -l ${1} $ $ ./simple.sh Oh the Waste ls: Oh: No such file or directory $When we don't put any quotes around the filename as we invoke the script, then bash sees three arguments and substitutes the first argument (Oh) for$1. The ls command runs withOhas its only argument and can't find that file.So now let's put quotes around the filename when we invoke the script:$ ./simpls.sh "Oh the Waste" ls: Oh: No such file or directory ls: the: No such file or directory ls: Waste: No such file or directory $
Still not good. bash has taken the three-word filename and substituted it for$1on the ls command line in our script. So far so good. Since we don't have quotes around the variable reference in our script, however, ls sees each word as a separate argument, i.e., as separate filenames. It can't find any of them.Let's try a script that quotes the variable reference:$ cat quoted.sh # note the quotes ls -l "${1}" $ $ ./quoted.sh "Oh the Waste" -rw-r--r-- 1 smith users 28470 2007-01-11 19:22 Oh the Waste $Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Handling Lists of Parameters with Blanks
- InhaltsvorschauOK, you have quotes around your variable as the previous recipe recommended. But you're still getting errors. It's just like the script from the , "Looping Over Arguments Passed to a Script," but it fails when a file has a blank in its name:
# for FN in $* do chmod 0750 "$FN" done
It has to do with the $* in the script, used in theforloop. For this case we need to use a different but related shell variable, $@. When it is quoted, the resulting list has quotes around each argument separately. The shell script should be written as follows:#!/usr/bin/env bash # cookbook filename: chmod_all.2 # # change permissions on a bunch of files # with better quoting in case of filenames with blanks # for FN in "$@" do chmod 0750 "$FN" done
The parameter $* expands to the list of arguments supplied to the shell script. If you invoke your script like this:$ myscript these are args
then $* refers to the three argumentsthese are args. And when used in aforloop, such as:for FN in $*
then the first time through the loop,$FNis assigned the first word (these) and the second time, the second word (are), etc.If the arguments are filenames and they are put on the command line by pattern matching, as when you invoke the script this way:$ myscript *.mp3
then the shell will match all the files in the current directory whose names end with the four characters.mp3, and they will be passed to the script. So consider an example where there are three MP3 files whose names are:vocals.mp3 cool music.mp3 tophit.mp3
The second song title has a blank in the filename betweencoolandmusic. When you invoke the script with:$ myscript *.mp3
you'll get, in effect:$ myscript vocals.mp3 cool music.mp3 tophit.mp3
If your script contains the line:for FN in $*
that will expand to:for FN in vocals.mp3 cool music.mp3 tophit.mp3
which has four words in its list, not three. The second song title has a blank as the fifth character (cool music.mp3), and the blank causes the shell to see that as two separate words (cool and music.mp3), so$FNwill be cool on the second iteration through theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Counting Arguments
- InhaltsvorschauYou need to know with how many parameters the script was invoked.Use the shell built-in variable ${#}. Here's some scripting to enforce an exact count of three arguments:
#!/usr/bin/env bash # cookbook filename: check_arg_count # # Check for the correct # of arguments: # Use this syntax or use: if [ $# -lt 3 ] if (( $# < 3 )) then printf "%b" "Error. Not enough arguments.\n" >&2 printf "%b" "usage: myscript file1 op file2\n" >&2 exit 1 elif (( $# > 3 )) then printf "%b" "Error. Too many arguments.\n" >&2 printf "%b" "usage: myscript file1 op file2\n" >&2 exit 2 else printf "%b" "Argument count correct. Proceeding...\n" fi
And here is what it looks like when we run it, once with too many arguments and once with the correct number of arguments:$ ./myscript myfile is copied into yourfile Error. Too many arguments. usage: myscript file1 op file2 $ ./myscript myfile copy yourfile Argument count correct. Proceeding...
After the opening comments (always a helpful thing to have in a script), we have theiftest to see whether the number of arguments supplied (found in $#) is greater than three. If so, we print an error message, remind the user of the correct usage, and exit.The output from the error messages are redirected to standard error. This is in keeping with the intent of standard error as the channel for all error messages.The script also has a different return value depending on the error that was detected. While not that significant here, it is useful for any script that might be invoked by other scripts, so that there is a programmatic way not only to detect failure (non-zero exit value), but to distinguish between error types.One word of caution: don't confuse ${#} with${#VAR}or even${VAR#alt}just because they all use the # inside of braces. The first gives the number of arguments the second gives the length of the value in the variableVAR, and the third does a certain kind of substitution.- , "Telling If a Command Succeeded or Not"
- , "Documenting Your Script"
- , "Consuming Arguments"
- , "Changing Pieces of a String"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Consuming Arguments
- InhaltsvorschauFor any serious shell script, you are likely to have two kinds of arguments—options that modify the behavior of the script and the real arguments with which you want to work. You need a way to get rid of the option argument(s) after you've processed them.Remember this script:
for FN in "$@" do echo changing $FN chmod 0750 "$FN" done
It's simple enough—it echoes the filename that it is working on, then it changes that file's permissions. What if you want it to work quietly sometimes, not echoing the filename? How would we add an option to turn off this verbose behavior while preserving theforloop?#!/usr/bin/env bash # cookbook filename: use_up_option # # use and consume an option # # parse the optional argument VERBOSE=0; if [[ $1 = -v ]] then VERBOSE=1; shift; fi # # the real work is here # for FN in "$@" do if (( VERBOSE == 0 )) then echo changing $FN fi chmod 0750 "$FN" done
We add a flag variable,$VERBOSE, to tell us whether or not to echo the filename as we work. But once the shell script has seen the-vand set the flag, we don't want the-vin the argument list any more. The shift statement tells bash to shift its arguments down one position, getting rid of the first argument ($1) as$2becomes$1, and$3becomes$2, and so on.That way, when theforloop runs, the list of parameters (in $@) no longer contains the-vbut starts with the next parameter.This approach of parsing arguments is alright for handling a single option. But if you want more than one option, you need a bit more logic. By convention, options to a shell script (usually) are not dependent on position; e.g.,myscript -a -pshould be the same asmyscript -p -a. Moreover, a robust script should be able to handle repeated options and either ignore them or report an error. For more robust parsing, see the recipe on bash'sgetoptsbuilt-in (, "Parsing Arguments for Your Shell Script").- help shift
- , "Looping Over Arguments Passed to a Script"
- , "Counting Arguments"
- , "Consuming Arguments"
- , "Parsing Command-Line Arguments"
- , "Parsing Arguments for Your Shell Script"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Default Values
- InhaltsvorschauYou have a shell script that takes arguments supplied on the command line. You'd like to provide default values so that the most common value(s) can be used without needing to type them every time.Use the ${:-} syntax when referring to the parameter, and use it to supply a default value:
FILEDIR=${1:-"/tmp"}There are a series of special operators available when referencing a shell variable. This one, the : -operator, says that if$1is not set or is null then it will use what follows,/tmpin our example, as the value. Otherwise it will use the value that is already set in$1. It can be used on any shell variable, not just the positional parameters (1, 2, 3, etc.), but they are probably the most common use.Of course you could do this the long way by constructing anifstatement and checking to see if the variable is null or unset (we leave that as an exercise to the reader), but this sort of thing is so common in shell scripts that this syntax has been welcomed as a convenient shorthand.- bash manpage on parameter substitution
- Learning the bash Shell by Cameron Newham (O'Reilly), -
- Classic Shell Scripting by Nelson H.F. Beebe and Arnold Robbins (O'Reilly), pages 113–114
- , "Setting Default Values"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting Default Values
- InhaltsvorschauYour script may rely on certain environment variables, either widely used ones (e.g.,
$USER) or ones specific to your own business. If you want to build a robust shell script, you should make sure that these variables do have a reasonable value. You want to guarantee a reasonable default value. How?Use the assignment operator in the shell variable reference the first time you refer to it to assign a value to the variable if it doesn't already have one, as in:cd ${HOME:=/tmp}The reference to$HOMEin the example above will return the current value of$HOMEunless it is empty or not set at all. In those cases (empty or not set), it will return the value/tmp, which will also be assigned to$HOMEso that further references to$HOMEwill have this new value.We can see this in action here:$ echo ${HOME:=/tmp} /home/uid002 $ unset HOME # generally not wise to do $ echo ${HOME:=/tmp} /tmp $ echo $HOME /tmp $ cd ; pwd /tmp $Once we unset the variable it no longer had any value. When we then used the := operator as part of our reference to it, the new value (/tmp) was substituted. The subsequent references to$HOMEreturned its new value.One important exception to keep in mind about the assignment operator: this mechanism will not work with positional parameter arguments (e.g.,$1or $*). For those cases, use :- in expressions like${1:-default}, which will return the value without trying to do the assignment.As an aside, it might help you to remember some of these crazy symbols if you think of the visual difference between${VAR:=value}and${VAR:-value}. The := will do an assignment as well as return the value on the right of the operator. The :- will do half of that—it just returns the value but doesn't do the assignment—so its symbol is only half of an equal sign (i.e., one horizontal bar, not two). If this doesn't help, forget that we mentioned it.- , "Getting Default Values"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using null As a Valid Default Value
- InhaltsvorschauYou need to set a default value, but you want to allow an empty string as a valid value. You only want to substitute the default in the case where the value is unset.The ${:=} operator has two cases where the new value will be used: first, when the value of the shell variable has previously not been set (or has been explicitly unset); and second, where the value has been set but is empty, as in
HOME=""orHOME=$OTHER(where$OTHERhad no value).The shell can distinguish between these two cases, and omitting the colon (:) indicates that you want to make the substitution only if the value is unset. If you write only${HOME=/tmp}without the colon, the assignment will take place only in the case where the variable is not set (never set or explicitly unset).Let's play with the$HOMEvariable again, but this time without the colon in the operator:$ echo ${HOME=/tmp} # no substitution needed /home/uid002 $ HOME="" # generally not wise $ echo ${HOME=/tmp} # will NOT substitute $ unset HOME # generally not wise $ echo ${HOME=/tmp} # will substitute /tmp $ echo $HOME /tmp $In the case where we simply made the$HOMEvariable an empty string, the = operator didn't do the substitution since$HOMEdid have a value, albeit null. But when we unset the variable, the substitution occurs. If you want to allow for empty strings, use just the = with no colon. Most times, though, the := is used because you can do little with an empty value, deliberate or not.- , "Getting Default Values"
- , "Setting Default Values"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using More Than Just a Constant String for Default
- InhaltsvorschauYou need something more than just a constant string as the default value for the variable.You can use quite a bit more on the righthand side of these shell variable references. For example:
cd ${BASE:="$(pwd)"}As the example shows, the value that will be substituted doesn't have to be just a string constant. Rather it can be the result of a more complex shell expression, including running commands in a subshell (as in the example). In our example, if$BASEis not set, the shell will run the pwd built-in command (to get the current directory) and use the string that it returns as the value.So what can you do on the righthand side of this (and the other similar) operators? The bash manpage says that what we put to the right of the operator "is subject to tilde expansion, parameter expansion, command substitution, and arithmetic expansion."Here is what that means:- Parameter expansion means that we could use other shell variables in this expression, as in:
${BASE:=${HOME}}. - Tilde expansion means that we can use expressions like
~boband it will expand that to refer to the home directory of the usernamebob. Use${BASE:=~uid17}to set the default value to the home directory for useruid17, but don't put quotes around this string, as that will defeat the tilde expansion. - Command substitution is what we used in the example; it will run the commands and take their output as the value for the variable. Commands are enclosed in the single parentheses syntax,
$( cmds ). - Arithmetic expansion means that we can do integer arithmetic, using the
$((…))syntax in this expression. Here's an example:echo ${BASE:=/home/uid$((ID+1))}
- , "Getting Default Values"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Giving an Error Message for Unset Parameters
- InhaltsvorschauThose shorthands for giving a default value are cool, but maybe you need to force the users to give you a value, otherwise you don't want to proceed. Perhaps if they left off a parameter, they don't really understand how to invoke your script. You want to leave nothing to guesswork. Is there anything shorter than lots of
ifstatements to check each of your several parameters?Use the ${:?} syntax when referring to the parameter. bash will print an error message and then exit if the parameter is unset or null.#!/usr/bin/env bash # cookbook filename: check_unset_parms # USAGE="usage: myscript scratchdir sourcefile conversion" FILEDIR=${1:?"Error. You must supply a scratch directory."} FILESRC=${2:?"Error. You must supply a source file."} CVTTYPE=${3:?"Error. ${USAGE}"}Here's what happens when we run that script with insufficient arguments:$ ./myscript /tmp /dev/null ./myscript: line 5: 3: Error. usage: myscript scracthdir sourcefile conversion $
The check is made to see if the first parameter is set (or null) and if not, it will print an error message and exit.The third variable uses another shell variable in its message. You can even run another command inside it:CVTTYPE=${3:?"Error. $USAGE. $(rm $SCRATCHFILE)"}If parameter three is not set, then the error message will contain the phrase "Error.", along with the value of the variable named$USAGEand then any output from the command which removes the filename named by the variable$SCRATCHFILE. OK, so we're getting carried away. You can make your shell script awfully compact, and we do mean awfully. It is better to waste some whitespace and a few bytes to make the logic ever so much more readable, as in:if [ -z "$3" ] then echo "Error. $USAGE" rm $SCRATCHFILE fi
One other consideration: the error message produced by the ${:?} feature comes out with the shell script filename and line number. For example:./myscript: line 5: 3: Error. usage: myscript scracthdir sourcefile conversion
Because you have no control over this part of the message, and since it looks like an error in the shell script itself, combined with the issue of readability, this technique is not so popular in commercial-grade shell scripts. (It is handy for debugging, though.)Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Changing Pieces of a String
- InhaltsvorschauYou want to rename a number of files. The filenames are almost right, but they have the wrong suffix.Use a bash parameter expansion feature that will remove text that matches a pattern.
#!/usr/bin/env bash # cookbook filename: suffixer # # rename files that end in .bad to be .bash for FN in *.bad do mv "${FN}" "${FN%bad}bash" doneTheforloop will iterate over a list of filenames in the current directory that all end in .bad. The variable$FNwill take the value of each name one at a time. Inside the loop, themvcommand will rename the file (move it from the old name to the new name). We need to put quotes around each filename in case the filename contains embedded spaces.The crux of this operation is the reference to$FNthat includes an automatic deletion of the trailingbadcharacters. The ${ } delimit the reference so that thebashadjacent to it is just appended right on the end of the string.Here it is broken down into a few more steps:NOBAD="${FN%bad}" NEWNAME="${NOBAD}bash" mv "${FN}" "${NEWNAME}"This way you can see the individual steps of stripping off the unwanted suffix, creating the new name, and then renaming the files. Putting it all on one line isn't so bad though, once you get used to the special operators.Since we are not just removing a substring from the variable but are replacing thebadwithbash, we could have used the substitution operator for variable references, the slash (/). Similar to editor commands (e.g., those found in vi and sed) that use the slash to delimit substitutions, we could have written:mv "${FN}" "${FN/.bad/.bash}"(Unlike the editor commands, you don't use a final slash—the right-brace serves that function.)However, one reason that we didn't do it this way is because the substitution isn't anchored, and will make the substitution anywhere in the variable. If, for example, we had a file named subaddon.bad then the substitution would leave us with subashdon.bad, which is not what we want. If we used a double slash for the first slash, it would substitute every occurrence within the variable. That would result inEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Array Variables
- InhaltsvorschauThere have been plenty of scripts so far with variables, but can bash deal with an array of variables?Yes. bash now has an array syntax for single-dimension arrays.Arrays are easy to initialize if you know the values as you write the script. The format is simple:
MYRA=(first second third home)
Each element of the array is a separate word in the list enclosed in parentheses. Then you can refer to each this way:echo runners on ${MYRA[0]} and ${MYRA[2]}This output is the result:runners on first and third
If you write only$MYRA, you will get only the first element, just as if you had written${MYRA[0]}.- Learning the bash Shell by Cameron Newham (O'Reilly), pages 157–161 for more information about arrays
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 6: Shell Logic and Arithmetic
- InhaltsvorschauOne of the big improvements that modern versions of bash have when compared with the original Bourne shell is in the area of arithmetic. Early versions of the shell had no built-in arithmetic; it had to be done by invoking a separate executable, even just to add 1 to a variable. In a way it's a tribute to how useful and powerful the shell was and is—that it can be used for so many tasks despite that awful mechanism for arithmetic. Maybe no one expected the shell to be so useful and so well used but, after a while, the simple counting useful for automating repetitive tasks needed simple, straightforward syntax. The lack of such capability in the original Bourne shell contributed to the success of the C shell (csh) when it introduced C Language-like syntax for shell programming, including numeric variables. Well, that was then and this is now. If you haven't looked at shell arithmetic in bash for a while, you're in for a big surprise.Beyond arithmetic, there are the control structures familiar to any programmer. There is an
if/then/elseconstruct for decision making. There arewhileloops andforloops, but you will see some bash peculiarities to all of these. There is acasestatement made quite powerful by its string pattern matching, and an odd construct calledselect. After discussing these features we will end the chapter by using them to build two simple command-line calculators.You need to do some simple arithmetic in your shell script.Use $(( )) or let for integer arithmetic expressions.COUNT=$((COUNT + 5 + MAX * 2)) let COUNT+=5+MAX*2
As long as you keep to integer arithmetic, you can use all the standard (i.e., C-like) operators inside of $(()) for arithmetic. There is one additional operator—you can use ** for raising to a power, as inMAX=$((2**8)), which yields 256.Spaces are not needed nor are they prohibited around operators and arguments (though ** must be together) within a $(( )) expression. But you must not have spaces around the equals sign, as with any bash variable assignment. If you wrote:COUNT = $((COUNT + 5)) # not what you think!
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Doing Arithmetic in Your Shell Script
- InhaltsvorschauYou need to do some simple arithmetic in your shell script.Use $(( )) or let for integer arithmetic expressions.
COUNT=$((COUNT + 5 + MAX * 2)) let COUNT+=5+MAX*2
As long as you keep to integer arithmetic, you can use all the standard (i.e., C-like) operators inside of $(()) for arithmetic. There is one additional operator—you can use ** for raising to a power, as inMAX=$((2**8)), which yields 256.Spaces are not needed nor are they prohibited around operators and arguments (though ** must be together) within a $(( )) expression. But you must not have spaces around the equals sign, as with any bash variable assignment. If you wrote:COUNT = $((COUNT + 5)) # not what you think!
then bash will try to run a program named COUNT and its first argument would be an equal sign, and its second argument would be the number you get adding5to the value of$COUNT. Remember not to put spaces around the equal sign.Another oddity to these expressions is that the $ that we normally put in front of a shell variable to say we want its value (as in$COUNTor$MAX) is not needed inside the double parentheses. For example,$((COUNT +5 MAX * 2))needs no dollar sign on the shell variables—in effect, the outer $ applies to the entire expression.We do need the dollar sign, though, if we are using a positional parameter (e.g.,$2) to distinguish it from a numeric constant (e.g., "2"). Here's an example:COUNT=$((COUNT + $2 + OFFSET))
There is a similar mechanism for integer arithmetic with shell variables using the bash built-inletstatement. It uses the same arithmetic operators as the $(()) construct:let COUNT=COUNT+5
When usinglet, there are some fancy assignment operators we can use such as this (which will accomplish the same thing as the previous line):let COUNT+=5
(This should look familiar to programmers of C/C++ and Java.)shows a list of those special assignment operators.Table 6-1: Explanation of assignment operators in bash OperatorOperation with assignmentUseMeaning=Simple assignmenta=ba=b*=Multiplicationa*=ba=(a*b)Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Branching on Conditions
- InhaltsvorschauYou want to check if you have the right number of arguments and take actions accordingly. You need a branching construct.The
ifstatement in bash is similar in appearance to that in other programming languages:if [ $# -lt 3 ] then printf "%b" "Error. Not enough arguments.\n" printf "%b" "usage: myscript file1 op file2\n" exit 1 fi
or alternatively:if (( $# < 3 )) then printf "%b" "Error. Not enough arguments.\n" printf "%b" "usage: myscript file1 op file2\n" exit 1 fi
Here's a full-blown if with an elif (bash-talk forelse-if) and anelseclause:if (( $# < 3 )) then printf "%b" "Error. Not enough arguments.\n" printf "%b" "usage: myscript file1 op file2\n" exit 1 elif (( $# > 3 )) then printf "%b" "Error. Too many arguments.\n" printf "%b" "usage: myscript file1 op file2\n" exit 2 else printf "%b" "Argument count correct. Proceeding...\n" fi
You can even do things like this:[ $result = 1 ] \ && { echo "Result is 1; excellent." ; exit 0; } \ || { echo "Uh-oh, ummm, RUN AWAY! " ; exit 120; }(For a discussion of this last example, see ,"Saving or Grouping Output from Several Commands.")We have two things we need to discuss: the basic structure of theifstatement and how it is that we have different syntax (parentheses or brackets, operators or options) for theifexpression. The first may help explain the second. The general form for anifstatement, from the manpage for bash, is:if list; then list; [ elif list; then list; ] ... [ else list; ] fi
The [ and ] in our description here are used to delineate optional parts of the statement (e.g., someifstatements have noelseclause). So let's look for a moment at the if without any optional elements.The simplest form for anifstatement would be:if list; then list; fi
In bash, the semicolon serves the same purpose as a newline—it ends a statement. So in the first examples of the Solution section we could have crammed the example onto fewer lines by using the semicolons, but it is more readable to use newlines.ThethenlistEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing for File Characteristics
- InhaltsvorschauYou want to make your script robust by checking to see if your input file is there before reading from it; you would like to see if your output file has write permissions before writing to it; you would like to see if there is a directory there before you attempt to cd into it. How do you do all that in bash scripts?Use the various file characteristic tests in the test command as part of your
ifstatements. Your specific problems might be solved with scripting that looks something like this:!/usr/bin/env bash # cookbook filename: checkfile # DIRPLACE=/tmp INFILE=/home/yucca/amazing.data OUTFILE=/home/yucca/more.results if [ -d "$DIRPLACE" ] then cd $DIRPLACE if [ -e "$INFILE" ] then if [ -w "$OUTFILE" ] then doscience < "$INFILE" >> "$OUTFILE" else echo "can not write to $OUTFILE" fi else echo "can not read from $INFILE" fi else echo "can not cd into $DIRPLACE" fi
We put all the references to the various filenames in quotes in case they have any embedded spaces in the pathnames. There are none in this example, but if you change the script you might use other pathnames.We tested and executed the cd before we tested the other two conditions. In this example it wouldn't matter, butifINFILE or OUTFILE were relative pathnames (not beginning from the root of the file system, i.e., with a leading "/"), then the test might evaluate true before the cd and not after, or vice versa. This way, we test right before we use the files.We use the double-greater-than operator>>to concatenate output onto our results file, rather than wiping it out. You wouldn't really care if the file had write permissions if you were going to obliterate it. (Then you would only need write permission on its containing directory.)The several tests could be combined into one largeifstatement using the-a(read "and") operator, but then if the test failed you couldn't give a very helpful error message since you wouldn't know which test it didn't pass.There are several other characteristics for which you can test. Three of them are tested using binary operators, each taking two filenames:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing for More Than One Thing
- InhaltsvorschauWhat if you want to test for more than one characteristic? Do you have to nest your
ifstatements?Use the operators for logial AND(-a)and OR(-o)to combine more than one test in an expression. For example:if [ -r $FILE -a -w $FILE ]
will test to see that the file is both readable and writable.All the file test conditions include an implicit test for existence, so you don't need to test if a file exists and is readable. It won't be readable if it doesn't exist.These conjunctions(-afor AND and-ofor OR) can be used for all the various test conditions. They aren't limited to just the file conditions.You can make several and/or conjunctions on one statement. You might need to use parentheses to get the proper precedence, as ina and (b or c), but if you use parentheses, be sure to escape their special meaning from the shell by putting a backslash before each or by quoting each parenthesis. Don't try to quote the entire expression in one set of quotes, however, as that will make your entire expression a single term that will be treated as a test for an empty string (see ,"Testing for String Characteristics").Here's an example of a more complex test with the parentheses properly escaped:if [ -r "$FN" -a \( -f "$FN" -o -p "$FN" \) ]
Don't make the assumption that these expressions are evaluated in quite the same order as in Java or C language. In C and Java, if the first part of the AND expression is false (or the first part true in an OR expression), the second part of the expression won't be evaluated (we say the expression short-circuited). However, because the shell makes multiple passes over the statement while preparing it for evaluation (e.g., doing parameter substitution, etc.), both parts of the joined condition may have been partially evaluated. While it doesn't matter in this simple example, in more complicated situations it might. For example:if [ -z "$V1" -o -z "${V2:=YIKES}" ]Even if$V1is empty, satisfying enough of theifstatement that the second part of the condition (checking if$V2is empty) need not occur, the value of$V2may have already been modified (as a side-effect of the parameter substitution forEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing for String Characteristics
- InhaltsvorschauYou want your script to check the value of some strings before using them. The strings could be user input, read from a file, or environment variables passed to your script. How do you do that with bash scripts?There are some simple tests that you can do with the built-in test command, using the single bracket
ifstatements. You can check to see whether a variable has any text, and you can check to see whether two variables are equal as strings.For example:#!/usr/bin/env bash # cookbook filename: checkstr # # if statement # test a string to see if it has any length # # use the command line argument VAR="$1" # if [ "$VAR" ] then echo has text else echo zero length fi # if [ -z "$VAR" ] then echo zero length else echo has text fi
We use the phrase "has any length" deliberately. There are two types of variables that will have no length—those that have been set to an empty string and those that have not been set at all. This test does not distinguish between those two cases. All it asks is whether there are some characters in the variable.It is important to put quotes around the "$VAR"expression because without them your syntax could be disturbed by odd user input. If the value of$VARwerex -a 7 -lt 5and if there were no quotes around the$VAR, then the expression:if [ -z $VAR ]
would become (after variable substitution):if [ -z x -a 7 -lt 5 ]
which is legitimate syntax for a more elaborate test, but one that will yield a result that is not what you wanted (i.e., one not based on whether the string has characters).- , "Testing with Pattern Matches"
- , "Testing with Regular Expressions"
- , "Avoiding Interpreter Spoofing"
- "Test Operators" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing for Equal
- InhaltsvorschauYou want to check to see if two shell variables are equal, but there are two different test operators:
-eqand = (or ==). So which one should you use?The type of comparison you need determines which operator you should use. Use the-eqoperator for numeric comparisons and the equality primary = (or ==) for string comparisons.Here's a simple script to illustrate the situation:#!/usr/bin/env bash # cookbook filename: strvsnum # # the old string vs. numeric comparison dilemma # VAR1=" 05 " VAR2="5" printf "%s" "do they -eq as equal? " if [ "$VAR1" -eq "$VAR2" ] then echo YES else echo NO fi printf "%s" "do they = as equal? " if [ "$VAR1" = "$VAR2" ] then echo YES else echo NO fi
When we run the script, here is what we get:$ bash strvsnum do they -eq as equal? YES do they = as equal? NO $
While the numeric value is the same (5) for both variables, Characters such as leading zeros and whitespace can mean that the strings are not equal as strings.Both = and == are accepted, but the single equal sign follows the POSIX standard and is more portable.It may help you to remember which comparison to use if you can recognize that the-eqoperator is similar to the FORTRAN .eq. operator. (FORTRAN is a very numbers-oriented language, used for scientific computation.) In fact, there are several numerical comparison operators, each similar to an old FORTRAN operator. The abbreviations, all listed in , are rather mnemonic-like and easy to figure out.Table 6-3: bash's comparison operators NumericStringMeaning-lt<Less than-le<=Less than or equal to-gt>Greater than-ge>=Greater than or equal to-eq=, = =Equal to-ne!=Not equal toOn the other hand, these are the opposite of Perl, in whicheq, ne, etc. are the string operators, while ==, !=, etc. are numeric.- , "Testing with Pattern Matches"
- , "Testing with Regular Expressions"
- , "Validating Input"
- "Test Operators" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing with Pattern Matches
- InhaltsvorschauYou want to test a string not for a literal match, but to see if it fits a pattern. For example, you want to know if a file is named like a JPEG file might be named.Use the double-bracket compound statement in an
ifstatement to enable shell-style pattern matches on the righthand side of the equals operator:if [[ "${MYFILENAME}" == *.jpg ]]The double-brackets is a newer syntax (bash version 2.01 or so). It is not the old-fashioned [ of the test command, but a newer bash mechanism. It uses the same operators that work with the single bracket form, but in the double-bracket syntax the equal sign is a more powerful string comparator. The equal sign operator can be a single equal sign or a double equals as we have used here. They are the same semantically. We prefer to use the double equals (especially when using the pattern matching) to emphasize the difference, but it is not the reason that we get pattern matching—that comes from the double-bracket compound statement.The standard pattern matching includes the * to match any number of characters, the question mark (?) to match a single character, and brackets for including a list of possible characters. Note that these resemble shell file wildcards, and are not regular expressions.Don't put quotes around the pattern if you want it to behave as a pattern. If our string had been quoted, it would have only matched strings with a literal asterisk as the first character.There are more powerful pattern matching capabilities available by turning on some additional options in bash. Let's expand our example to look for filenames that end in either .jpg or .jpeg. We could do that with this bit of code:shopt -s extglob if [[ "$FN" == *.@(jpg|jpeg) ]] then # and so on
Theshopt -scommand is the way to turn on shell options. Theextglobis the option dealing with extended pattern matching (or globbing). With this extended pattern matching we can have several patterns, separated by the | character and grouped by parentheses. The first character preceding the parentheses says whether the list should match just one occurrence of a pattern in the list (using a leading @) or some other criteria. lists the possibilities (see also "extglob Extended Pattern-Matching Operators" in ).Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing with Regular Expressions
- InhaltsvorschauSometimes even the extended pattern matching of the
extgloboption isn't enough. What you really need are regular expressions. Let's say that you rip a CD of classical music into a directory, ls that directory, and see these names:$ ls Ludwig Van Beethoven - 01 - Allegro.ogg Ludwig Van Beethoven - 02 - Adagio un poco mosso.ogg Ludwig Van Beethoven - 03 - Rondo - Allegro.ogg Ludwig Van Beethoven - 04 - "Coriolan" Overture, Op. 62.ogg Ludwig Van Beethoven - 05 - "Leonore" Overture, No. 2 Op. 72.ogg $
You'd like to write a script to rename these files to something simple, such as just the track number. How can you do that?Use the regular expression matching of the =~ operator. Once it has matched the string, the various parts of the pattern are available in the shell variable$BASH_REMATCH. Here is the part of the script that deals with the pattern match:#!/usr/bin/env bash # cookbook filename: trackmatch # for CDTRACK in * do if [[ "$CDTRACK" =~ "([[:alpha:][:blank:]]*)- ([[:digit:]]*) - (.*)$" ]] then echo Track ${BASH_REMATCH[2]} is ${BASH_REMATCH[3]} mv "$CDTRACK" "Track${BASH_REMATCH[2]}" fi doneCaution: this requires bash version 3.0 or newer because older versions don't have the =~ operator. In addition, bash version 3.2 unified the handling of the pattern in the == and =~ conditional command operators but introduced a subtle quoting bug that was corrected in 3.2 patch #3. If the solution above fails, you may be using bash version 3.2 without that patch. You might want to upgrade to a newer version. You might also avoid the bug with a less readable version of the regular expression by removing the quotes around the regex and escaping each parenthesis and space character individually, which gets ugly quickly:if [[ "$CDTRACK" =~ \([[:alpha:][:blank:]]*\)-\ \([[:digit: ]]*\)\ -\ \(.*\)\$ ]]
If you are familiar with regular expressions from sed, awk, and older shells, you may notice a few slight differences with this newer form. Most noticeable are the character classes such as[:alpha:]and that the grouping parentheses don't need to be escaped—we don't write \( here as we would inEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Changing Behavior with Redirections
- InhaltsvorschauNormally you want a script to behave the same regardless of whether input comes from a keyboard or a file, or whether output is going to the screen or a file. Occasionally, though, you want to make that distinction. How do you do that in a script?Use the
test -toption in anifstatement to branch between the two desired behaviors.Think long and hard before you do this. So much of the power and flexibility of bash scripting comes from the fact that scripts can be pipelined together. Be sure you have a really good reason to make your script behave oddly when input or output is redirected.- , "Using Multiple Redirects on One Line"
- , "Saving Output When Redirect Doesn't Seem to Work"
- , "Swapping STDERR and STDOUT"
- , ""Daemonizing" Your Script"
- , "Using bash Net-Redirection"
- , "Redirecting Output for the Life of a Script"
- "I/O Redirection" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Looping for a While
- InhaltsvorschauYou want your shell script to perform some actions repeatedly as long as some condition is met.Use the while looping construct for arithmetic conditions:
while (( COUNT < MAX )) do some stuff let COUNT++ done
for filesystem-related conditions:while [ -z "$LOCKFILE" ] do some things done
or for reading input:while read lineoftext do process $lineoftext done
The double parentheses in our first while statement are just arithmetic expressions, very much like the $(( )) expression for shell variable assignment. They bound an arithmetic expression and assume that variable names mentioned inside the parentheses are meant to be dereferenced. That is, you don't write$VAR, and instead useVARinside the parentheses.The use of the square brackets in while[-z"$LOCKFILE"] is the same as with theifstatement—the single square bracket is the same as using theteststatement.The last example,while read lineoftext, doesn't have any parentheses, brackets, or braces. The syntax of thewhilestatement in bash is defined such that the condition of thewhilestatement is a list of statements to be executed (just like theifstatement), and the exit status of the last one determines whether the condition is true or false. An exit status of zero, and the condition is considered true, otherwise false.Areadstatement returns a 0 on a successful read and a -1 on end-of-file, which means that thewhilewill find it true for any successfulread, but when the end of file is reached (and -1 returned) thewhilecondition will be false and the looping will end. At that point, the next statement to be executed will be the statement after thedonestatement.This logic of "keep looping while the statement returns zero" might seem a bit flipped—most C-like languages use the opposite, namely, "loop while nonzero." But in the shell, a zero return value means everything went well; non-zero return values indicate an error exit.This explains what happens with the (()) construct, too. Any expression inside the parentheses is evaluated, and if the result is nonzero, then the result of the (()) is to return a zero; similarly, a zero result returns a one. This means we can write expressions like Java or C programmers would, but theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Looping with a read
- InhaltsvorschauWhat can you do with a
whileloop? One common technique is to read the output of previous commands. Let's say you're using the Subversion revision control system, which is executable as svn. (This example is very similar to what you would do for cvs as well.) When you check the status of a directory subtree to see what files have been changed, you might see something like this:$ svn status bcb M bcb/amin.c ? bcb/dmin.c ? bcb/mdiv.tmp A bcb/optrn.c M bcb/optson.c ? bcb/prtbout.4161 ? bcb/rideaslist.odt ? bcb/x.maxc $
The lines that begin with question marks are files about which Subversion has not been told; in this case they're scratch files and temporary copies of files. The lines that begin with anAare newly added files, and those that begin withMhave been modified since the last changes were committed.To clean up this directory it would be nice to get rid of all the scratch files, which are those files named in lines that begin with a question mark.Try:svn status mysrc | grep '^?' | cut -c8- | \ while read FN; do echo "$FN"; rm -rf "$FN"; done
or:svn status mysrc | \ while read TAG FN do if [[ $TAG == \? ]] then echo $FN rm -rf "$FN" fi done
Both scripts will do the same thing—remove files that svn reports with a question mark.The first approach uses several subprograms to do its work (not a big deal in these days of gigahertz processors), and would fit on a single line in a typical terminal window. It uses grep to select only the lines that begin (signified by the ^) with a question mark. The expression '^?' is put in single quotes to avoid any special meanings that those characters have for bash. It then uses cut to take only the characters beginning in column eight (through the end of the line). That leaves just the filenames for thewhileloop to read.Thereadwill return a nonzero value when there is no more input, so at that point the loop will end. Until then, thereadwill assign the line of text that it reads each time into the variable "$FN", and that is the filename that we remove. We use the -Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Looping with a Count
- InhaltsvorschauYou need to loop a fixed number of times. You could use a
whileloop and do the counting and testing, but programming languages have for loops for such a common idiom. How does one do this in bash ?Use a special case of theforsyntax, one that looks a lot like C Language, but with double parentheses:$ for (( i=0 ; i < 10 ; i++ )) ; do echo $i ; done
In early versions of the shell, the original syntax for theforloop only included iterating over a fixed list of items. It was a neat innovation for such a word-oriented language as shell scripts, dealing with filenames and such. But when users needed to count, they sometimes found themselves writing:for i in 1 2 3 4 5 6 7 8 9 10 do echo $i done
Now that's not too bad, especially for small loops, but let's face it—that's not going to work for 500 iterations. (Yes, you could nest loops 5 x 10, but come on!) What you really need is aforloop that can count.The special case of theforloop with C-like syntax is a relatively recent addition to bash (appearing in version 2.04). Its more general form can be described as:for (( expr1 ; expr2 ; expr3 )) ; do list ; done
The use of double parentheses is meant to indicate that these are arithmetic expressions. You don't need to use the $ construct (as in$i, except for arguments like$1) when referring to variables inside the double parentheses (just like the other places where double parentheses are used in bash). The expressions are integer arithmetic expressions and offer a rich variety of operators, including the use of the comma to put multiple operations within one expression:for (( i=0, j=0 ; i+j < 10 ; i++, j++ )) do echo $((i*j)) done
Thatforloop initializes two variables (iandj), then has a more complex second expression adding the two together before doing the less-than comparison. The comma operator is used again in the third expression to increment both variables.- , "Looping with Floating-Point Values"
- , "Writing Sequences"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Looping with Floating-Point Values
- InhaltsvorschauThe
forloop with arithmetic expressions only does integer arithmetic. What do I do for floating-point values?Use the seq command to generate your floating-point values, if your system provides it:for fp in $(seq 1.0 .01 1.1) do echo $fp; other stuff too done
or:seq 1.0 .01 1.1 | \ while read fp do echo $fp; other stuff too done
The seq command will generate a sequence of floating-point numbers, one per line. The arguments to seq are the starting value, the increment, and the ending value. This is not the intuitive order if you are used to the C languageforloop, or if you learned your looping from BASIC (e.g.,FOR I=4 TO 10 STEP 2). With seq the increment is the middle argument.In the first example, the $() runs the command in a subshell and returns the result with the newlines replaced by just whitespace, so each value is a string value for theforloop.In the second example, seq is run as a command with its output piped into awhileloop that reads each line and does something with it. This would be the preferred approach for a really long sequence, as it can run the seq command in parallel with thewhile. Theforloop version has to run seq to completion and put all of its output on the command line for theforstatement. For very large sequences, this could be time- and memory-consuming.- , "Looping with a Count"
- , "Writing Sequences"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Branching Many Ways
- InhaltsvorschauYou have a series of comparisons to make, and the
if/then/elseis getting pretty long and repetitive. Isn't there an easier way?Use the case statement for a multiway branch:case $FN in *.gif) gif2png $FN ;; *.png) pngOK $FN ;; *.jpg) jpg2gif $FN ;; *.tif | *.TIFF) tif2jpg $FN ;; *) printf "File not supported: %s" $FN ;; esac
The equivalent to this using if/then/else statements is:if [[ $FN == *.gif ]] then gif2png $FN elif [[ $FN == *.png ]] then pngOK $FN elif [[ $FN == *.jpg ]] then jpg2gif $FN elif [[ $FN == *.tif || $FN == *.TIFF ]] then tif2jpg $FN else printf "File not supported: %s" $FN fi
Thecasestatement will expand the word (including parameter substitution) between thecaseand theinkeywords. It will then try to match the word with the patterns listed in order. This is a very powerful feature of the shell. It is not just doing simple value comparisons, but string pattern matches. We have simple patterns in our example: *.gifmatches any character sequence (signified by the *) that ends with the literal characters.gif.Use |, a vertical bar meaning logical OR, to separate different patterns for which you want to take the same action. In the example above, if$FNends either with .tifor .TIFFthen the pattern will match and the (fictional) tif2jpg command will be executed.Use the double semicolon to end the set of statements or else bash will continue executing into the next set of statements.There is noelseordefaultkeyword to indicate the statements to execute if no pattern matches. Instead, use * as the last pattern—since that pattern will match anything. Placing it last makes it act as the default and match anything that hasn't already been matched.An aside to C/C++ and Java programmers: the bashcaseis similar to theswitchstatement, and each pattern corresponds to a case. Notice though, the variable on which you can switch/case is a shell variable (typically a string value) and the cases are patterns (not just constant values). The patterns end with a right parenthesis (not a colon). The equivalent to theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Command-Line Arguments
- InhaltsvorschauYou want to write a simple shell script to print a line of dashes, but you want to parameterize it so that you can specify different line lengths and specify a character to use other than just a dash. The syntax would look like this:
dashes # would print out 72 dashes dashes 50 # would print out 50 dashes dashes -c= 50 # would print out 50 equal signs dashes -cx # would print out 72 x characters
What's an easy way to parse those simple arguments?For serious scripting, you should use thegetoptsbuilt-in. But we would like to show you thecasestatement in action, so for this simple situation we'll usecasefor argument parsing.Here's the beginning of the script (see ,"Starting Simple by Printing Dashes" for a complete remove):#!/usr/bin/env bash # cookbook filename: dashes # # dashes - print a line of dashes # # options: # how many (default 72) # -c X use char X instead of dashes # LEN=72 CHAR='-' while (( $# > 0 )) do case $1 in [0-9]*) LEN=$1 ;; -c) shift; CHAR=${1:--} ;; *) printf 'usage: %s [-c X] [#]\n' $(basename $0) >&2 exit 2 ;; esac shift done # # more...The default length (72) and the default character (-) are set at the beginning of the script (after some useful comments). Thewhileloop allows us to parse more than one parameter. It will keep looping while the number of arguments ($#) is above zero.Thecasestatement matches three different patterns. First, the[0-9]* will match any digit followed by any other characters. We could have used a more elaborate expression to allow only pure numbers, but we'll assume that any argument that begins with a digit is a number. If that isn't true (e.g., the user types1T4), then the script will error when it tries to use$LEN. We can live with that for now.The second pattern is a literal-c. There is no pattern to this, just an exact match. In that case, we use theshiftbuilt-in command to throw away that argument (now that we know what it is) and we take the next argument (which has now become the first argument, so it is referenced asEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating Simple Menus
- InhaltsvorschauYou have a simple SQL script that you would like to run against different databases to reset them for tests that you want to run. You could supply the name of the database on the command line, but you want something more interactive. How can you write a shell script to choose from a list of names?Use the select statement to create simple character-based screen menus. Here's a simple example:
#!/usr/bin/env bash # cookbook filename: dbinit.1 # DBLIST=$(sh ./listdb | tail +2) select DB in $DBLIST do echo Initializing database: $DB mysql -uuser -p $DB <myinit.sql done
Ignore for a moment how$DBLISTgets its values; just know that it is a list of words (like the output from ls would give). Theselectstatement will display those words, each preceded by a number, and the user will be prompted for input. The user makes a choice by typing the number and the corresponding word is assigned to the variable specified after the keywordselect(in this caseDB).Here's what the running of this script might look like:$ ./dbinit 1) testDB 2) simpleInventory 3) masterInventory 4) otherDB #? 2 Initializing database: simpleInventory #? $
When the user types "2" the variable DB is assigned the wordsimpleInventory. If you really want to get at the user's literal choice, the variable$REPLYwill hold it, in this case it would be "2".Theselectstatement is really a loop. When you have entered a choice it will execute the body of the loop (between thedoand thedone) and then re-prompt you for the next value.It doesn't redisplay the list every time, only if you make no choice and just press the Enter key. So whenever you want to see the list again, just press Enter.It does not re-evaluate the code after thein, that is, you can't alter the list once you've begun. If you modified$DBLISTinside the loop, it wouldn't change your list of choices.The looping will stop when it reaches the end of the file, which for interactive use means when you type Ctrl-D. (If you piped a series of choices into aselectloop, it would end when the input ends.)There isn't any formatting control over the list. If you're going to useEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Changing the Prompt on Simple Menus
- InhaltsvorschauYou just don't like that prompt in the
selectmenus. How can it be changed?The bash environment variable$PS3is the prompt used byselect. Set it to a new value and you'll get a new prompt.This is the third of the bash prompts. The first ($PS1) is the prompt you get before most commands. (We've used $ in our examples, but it can be much more elaborate than that, including user ID or directory names.) If a line of command input needs to be continued, the second prompt is used ($PS2).For select loops, the third prompt,$PS3, is used. Set it before theselectstatement to make the prompt be whatever you want. You can even modify it within the loop to have it change as the loop progresses.Here's a script similar to the previous recipe, but one that counts how many times it has handled a valid input:#!/usr/bin/env bash # cookbook filename: dbinit.2 # DBLIST=$(sh ./listdb | tail +2) PS3="0 inits >" select DB in $DBLIST do if [ $DB ] then echo Initializing database: $DB PS3="$((i++)) inits >" mysql -uuser -p $DB <myinit.sql fi done $
We've added some extra whitespace to make the setting of$PS3stand out more. Theifstatement assures us that we're only counting the times when the user entered a valid choice. Such a check would be useful in the previous version, but we were keeping it simple.- , "Selecting from a List of Options"
- , "Changing the Prompt on Simple Menus"
- , "Customizing Your Prompt"
- , "Using Secondary Prompts: $PS2, $PS3, $PS4"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating a Simple RPN Calculator
- InhaltsvorschauYou may be able to convert binary to decimal, octal, or hex in your head but it seems that you can't do simple arithmetic anymore and you can never find a calculator when you need one. What to do?Create a calculator using shell arithmetic and RPN notation:
#!/usr/bin/env bash # cookbook filename: rpncalc # # simple RPN command line (integer) calculator # # takes the arguments and computes with them # of the form a b op # allow the use of x instead of * # # error check our argument counts: if [ \( $# -lt 3 \) -o \( $(($# % 2)) -eq 0 \) ] then echo "usage: calc number number op [ number op ] ..." echo "use x or '*' for multiplication" exit 1 fi ANS=$(($1 ${3//x/*} $2)) shift 3 while [ $# -gt 0 ] do ANS=$((ANS ${2//x/*} $1)) shift 2 done echo $ANSAny arithmetic done within $(( )) is integer arithmetic only.The idea of RPN (or postfix) style of notation puts the operands (the numbers) first, followed by the operator. If we are using RPN, we don't write5 + 4but rather5 4 +as our expression. If you want to multiply the result by 2, then you just put2* on the end, so the whole expression would be5 4 + 2*, which is great for computers to parse because you can go left to right and never need parentheses. The result of any operation becomes the first operand for the next expression.In our simple bash calculator we will allow the use of lowercasexas a substitute for the multiplication symbol since * has special meaning to the shell. But if you escape that special meaning by writing '*' or \* we want that to work, too.How do we error check the arguments? We will consider it an error if there are less than three arguments (we need two operands and one operator, e.g.,6 3/). There can be more than three arguments, but in that case there will always be an odd number (since we start with three and add two more, a second operand and the next operator, and so on, always adding two more; the valid number of arguments would be 3 or 5 or 7 or 9 or …). We check that with the expression:$(($# % 2)) -eq 0
to see if the result is zero. The $(()) says we're doing some shell arithmetic inside. We are using the % operator (called theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating a Command-Line Calculator
- InhaltsvorschauYou need more than just integer arithmetic, and you've never been very fond of RPN notation. How about a different approach to a command-line calculator?Create a trivial command-line calculator using awk's built-in floating-point arithmetic expressions:
# cookbook filename: func_calc # Trivial command line calculator function calc { awk "BEGIN {print \"The answer is: \" $* }"; }You may be tempted to tryecho The answer is: $(( $* )), which will work fine for integers, but will truncate the results of floating-point operations.We use a function because aliases do not allow the use of arguments.You will probably want to add this function to your global /etc/bashrc or local ~/.bashrc.The operators are what you'd expect and are the same as in C:$ calc 2 + 3 + 4 The answer is: 9 $ calc 2 + 3 + 4.5 The answer is: 9.5
Watch out for shell meta characters. For example:$ calc (2+2-3)*4 -bash: syntax error near unexpected token `2+2-3'
You need to escape the special meaning of the parentheses. You can put the expression inside single quotes, or just use the backslash in front of any special (to the shell) character to escape its meaning. For example:$ calc '(2+2-3)*4' The answer is: 4 $ calc \(2+2-3\)\*4 The answer is: 4 $ calc '(2+2-3)*4.5' The answer is: 4.5
We need to escape the multiplication symbol too, since that has special meaning to bash as the wildcard for filenames. This is especially true if you like to put whitespace around your operators, as in17 + 3 * 21, because then * will match all the files in the current directory, putting their names on the command line in place of the asterisk—definitely not what you want.- man awk
- "ARITHMETIC EVALUATION" in the bash(1) manpage
- , "Creating a Simple RPN Calculator"
- , "Shortening or Changing Command Names"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 7: Intermediate Shell Tools I
- InhaltsvorschauIt is time to expand our repertoire. This chapter's recipes use some utilities that are not part of the shell, but which are so useful that it is hard to imagine using the shell without them.One of the over-arching philosophies of Unix (and thus Linux) is that of small (i.e., limited in scope) program pieces that can be fit together to provide powerful results. Rather than have one program do everything, we have many different programs that each do one thing well.That is true of bash as well. While bash is getting big and feature-rich, it still doesn't try to do everything, and there are times when it is easier to use other commands to accomplish a task even if bash can be stretched to do it.A simple example of this is the ls command. You needn't use ls to see the contents of your current directory. You could just type echo*to have filenames displayed. Or you could even get fancier, using the bash
printfcommand and some formatting, etc. But that's not really the purpose of the shell, and someone has already provided a listing program (ls) to deal with all sorts of variations on filesystem information.Perhaps more importantly, by not expecting bash to provide more filesystem listing features, we avoid additional feature creep pressures on bash and instead give it some measure of independence; ls can be released with new features without requiring that we all upgrade our bash versions.But enough philosophy—back to the practical.What we have here are three of the most useful text-related utilities: grep, sed, and awk.The grep program searches for strings, the sed program provides a way to edit text as it passes through a pipeline, and awk, well, awk is its own interesting beast, a precursor to perl and a bit of a chameleon—it can look quite different depending on how it is used.These utilities, and a few more that we will discuss in an upcoming chapter, become very much a part of most shell scripts and most sessions spent typing commands to bash. If your shell script requires a list of files on which to operate, it is likely that either find or grep will be used to supply that list of files, and thatEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sifting Through Files for a String
- InhaltsvorschauYou need to find all occurrences of a string in one or more files.The grep command searches through files looking for the expression you supply:
$ grep printf *.c both.c: printf("Std Out message.\n", argv[0], argc-1); both.c: fprintf(stderr, "Std Error message.\n", argv[0], argc-1); good.c: printf("%s: %d args.\n", argv[0], argc-1); somio.c: // we'll use printf to tell us what we somio.c: printf("open: fd=%d\n", iod[i]); $The files we searched through in this example were all in the current directory. We just used the simple shell pattern*.cto match all the files ending in .c with no preceding pathname.Not all the files through which you want to search may be that conveniently located. Of course, the shell doesn't care how much pathname you type, so we could have done something like this:$ grep printf ../lib/*.c ../server/*.c ../cmd/*.c */*.c
When more than one file is searched, grep begins its output with the filename, followed by a colon. The text after the colon is what actually appears in the files that grep searched.The search matches any occurrence of the characters, so a line that contained the string "fprintf" was returned, since "printf" is contained within "fprintf".The first (non-option) argument to grep can be just a simple string, as in this example, or it can be a more complex regular expression (RE). These REs are not the same as the shell's pattern matching, though they can look similar at times. Pattern matching is so powerful that you may find yourself relying on it to the point where you'll start using "grep" as a verb, and wishing you could make use of it everywhere, as in "I wish I could grep my desk for that paper you wanted."You can vary the output from grep using options on the command line. If you don't want to see the specific filenames, you may turn this off using the-hswitch to grep:$ grep -h printf *.c printf("Std Out message.\n", argv[0], argc-1); fprintf(stderr, "Std Error message.\n", argv[0], argc-1); printf("%s: %d args.\n", argv[0], argc-1); // we'll use printf to tell us what we printf("open: fd=%d\n", iod[i]); $Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Just the Filename from a Search
- InhaltsvorschauYou need to find the files in which a certain string appears. You don't want to see the line of text that was found, just the filenames.Use the
-l optionof grep to get just the filenames:$ grep -l printf *.c both.c good.c somio.c
If grep finds more than one match per file, it still only prints the name once. If grep finds no matches, it gives no output.This option is handy if you want to build a list of files to be operated on, based on the fact that they contain the string that you're looking for. Put the grep command inside $( ) and those filenames can be used on the command line.For example, to remove the files that contain the phrase "This file is obsolete," you could use this shell command combination:$ rm -i $(grep -l 'This file is obsolete' * )
We've added the-ioption to rm so that it will ask you before it removes each file. That's obviously a safer way to operate, given the power of this combination of commands.bash expands the* tomatch every file in the current directory (but does not descend into sub-directories) and passes them as the arguments to grep. Then grep produces a list of filenames that contain the given string. This list then is handed to the rm command to remove each file.- man grep
- man rm
- man regex (Linux, Solaris, HP-UX) or man re_format (BSD, Mac) for the details of your regular expression library
- Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly)
- , "Connecting Two Programs by Using Output As Input"
- , "Expecting to Change Exported Variables"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting a Simple True/False from a Search
- InhaltsvorschauYou need to know whether a certain string is in a particular file. However, you don't want any output, just a yes or no sort of answer.Use
-q, the "quiet" option for grep. Or, for maximum portability, just throw the output away by redirecting it into /dev/null. Either way, your answer is in the bash return status variable $? so you can use it in anif-test like this:$ grep -q findme bigdata.file $ if [ $? -eq 0 ] ; then echo yes ; else echo nope ; fi nope $
In a shell script, you often don't want the results of the search displayed in the out-put; you just want to know whether there is a match so that your script can branch accordingly.As with most Unix/Linux commands, a return value of 0 indicates successful completion. In this case, success is defined as having found the string in at least one of the given files (in this example, we searched in only one file). The return value is stored in the shell variable $?, which we can then use in anifstatement.If we list multiple filenames aftergrep -q, then grep stops searching after the very first occurrence of the search string being found. It doesn't search all the files, as you really just want to know whether it found any occurrence of the string. If you really need to read through all the files (why?), then rather than use-qyou can do this:$ grep findme bigdata.file >/dev/null $ if [ $? -eq 0 ] ; then echo yes ; else echo nope ; fi nope $
The redirecting to /dev/null sends the output to a special kind of device, a bit bucket, that just throws everything you give it away.The /dev/null technique is also useful if you want to write shell scripts that are portable across the various flavors of grep that are available on Unix and Linux systems, should you find one that doesn't support the-qoption.- man grep
- man regex (Linux, Solaris, HP-UX) or man re_format (BSD, Mac) for the details of your regular expression library
- Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly)
- , "Expecting to Change Exported Variables"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Searching for Text While Ignoring Case
- InhaltsvorschauYou need to search for a string (e.g., "error") in a log file, and you want to do it caseinsensitively to catch all occurrences.Use the
-ioption on grep to ignore case:$ grep -i error logfile.msgs
A case-isensitive search finds messages written "ERROR", "error", "Error," as well as ones like "ErrOR" and "eRrOr." This option is particularly useful for finding words anywhere that you might have mixed-case text, including words that might be capitalized at the beginning of a sentence or email addresses.- man grep
- man regex (Linux, Solaris, HP-UX) or man re_format (BSD, Mac) for the details of your regular expression library
- Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly)
- 's discussion of the find command and its
-inameoption - , "Expecting to Change Exported Variables"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Doing a Search in a Pipeline
- InhaltsvorschauYou need to search for some text, but the text you're searching for isn't in a file; instead, it's in the output of a command or perhaps even the output of a pipeline of commands.Just pipe your results into grep:
$ some pipeline | of commands | grepWhen no filename is supplied to grep, it reads from standard input. Most well-designed utilities meant for shell scripting will do this. It is one of the things that makes them so useful as building blocks for shell scripts.If you also want to have grep search error messages that come from the previous command, be sure to redirect its error output into standard output before the pipe:$ gcc bigbadcode.c 2>&1 | grep -i error
This command attempts to compile some hypothetical, hairy piece of code. We redirect standard error into standard output (2>&1) before we proceed to pipe (|) the output into grep, where it will search case-insensitively (-i) looking for the stringerror.Don't overlook the possibility of grepping the output of grep. Why would you want to do that? To further narrow down the results of a search. Let's say you wanted to find out Bob Johnson's email address:$ grep -i johnson mail/* ... too much output to think about; there are lots of Johnsons in the world ... $ !! | grep -i robert grep -i johnson mail/* | grep -i robert ... more manageable output ... $ !! | grep -i "the bluesman" grep -i johnson mail/* | grep -i robert | grep -i "the bluesman" Robert M. Johnson, The Bluesman <rmj@noplace.org>
You could have re-typed the first grep, but this example also shows the power of the !! history operator. The !! let's you repeat the previous command without retyping it. You can then continue adding to the command line after the !! as we show here. The shell will display the command that it runs, so that you can see what you got as a result of the !! substitution (see , "Repeating the Last Command").You can build up a long grep pipeline very quickly and simply this way, seeing the results of the intermediate steps as you go, and deciding how to refine your search with additional grep expressions. You could also accomplish the same task with a singleEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Paring Down What the Search Finds
- InhaltsvorschauYour search is returning way more than you expected, including many results you don't want.Pipe the results into
grep -vwith an expression that describes what you don't want to see.Let's say you were searching for messages in a log file, and you wanted all the messages from the month of December. You know that your logfile uses the 3-letter abbreviation for December as Dec, but you're not sure if it's always written as Dec,so to be sure to catch them all you type:... error on Jan 01: not a decimal number error on Feb 13: base converted to Decimal warning on Mar 22: using only decimal numbers error on Dec 16 : the actual message you wanted error on Jan 01: not a decimal number ...
A quick and dirty solution in this case is to pipe the first result into a second grep and tell the second grep to ignore any instances of "decimal":$ grep -i dec logfile | grep -vi decimal
It's not uncommon to string a few of these together (as new, unexpected matches are also discovered) to filter down the search results to what you're really looking for:$ grep -i dec logfile | grep -vi decimal | grep -vi decimate
The "dirty" part of this "quick and dirty" solution is that the solution here might also get rid of some of the December log messages, ones that you wanted to keep—if they have the word "decimal" in them, they'll be filtered out by thegrep-v.The-voption can be handy if used carefully; you just have to keep in mind what it might exclude.For this particular example, a better solution would be to use a more powerful regular expression to match the December date, one that looked for "Dec" followed by a space and two digits:$ grep 'Dec [0-9][0-9]' logfile
But that often won't work either because syslog uses a space to pad single digit dates, so we add a space in the first list [0-9]:$ grep 'Dec [0-9 ][0-9]' logfile
We used single quotes around the expression because of the embedded spaces, and to avoid any possible shell interpretation of the bracket characters (not that there would be, but just as a matter of habit). It's good to get into the habit of using single quotes around anything that might possibly be confusing to the shell. We could have written:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Searching with More Complex Patterns
- InhaltsvorschauThe regular expression mechanism of grep provides for some very powerful patterns that can fit most of your needs.A regular expression describes patterns for matching against strings. Any alphabetic character just matches that character in the string. "A" matches "A", "B" matches "B"; no surprise there. But regular expressions define other special characters that can be used by themselves or in combination with other characters to make more complex patterns.We already said that any character without some special meaning simply matches itself—"A" to "A" and so on. The next important rule is to combine letters just by position, so "AB" matches "A" followed by "B". This, too, seems obvious.The first special character is
(.).Aperiod (.) matches any single character. Therefore …. matches any four characters; A. matches an "A" followed by any character; and.A.matches any character, then an "A", then any character (not necessarily the same character as the first).An asterisk(*)means to repeat zero or more occurrences of the previous character. SoA*means zero or more "A" characters, and .* means zero or more characters of any sort (such as "abcdefg", "aaaabc", "sdfgf ;lkjhj", or even an empty line).So what does ..* mean? Any single character followed by zero or more of any character (i.e., one or more characters) but not an empty line.Speaking of lines, the caret ^ matches the beginning of a line of text and the dollar sign $ matches the end of a line; hence ^$ matches an empty line (the beginning followed by the end, with nothing in between).What if you want to match an actual period, caret, dollar sign, or any other special character? Precede it by a backslash (\).So ion. matches the letters "ion" followed by any other letter, bution\. matches "ion" bounded by a period (e.g., at the end of a sentence or wherever else it appears with a trailing dot).A set of characters enclosed in square brackets (e.g., [abc]) matches any one of those characters (e.g., "a" or "b" or "c"). If the first character inside the square brackets is a caret, then it matches any character that isEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Searching for an SSN
- InhaltsvorschauYou need a regular expression to match a Social Security number. These numbers are nine digits long, typically grouped as three digits, then two digits, then a final four digits (e.g., 123-45-6789). Sometimes they are written without hyphens, so you need to make hyphens optional in the regular expression.
$ grep '[0-9]\{3\}-\{0,1\}[0-9]\{2\}-\{0,1\}[0-9]\{4\}' datafileThese kinds of regular expressions are often jokingly referred to as write only expressions, meaning that they can be difficult or impossible to read. We'll take this one apart to help you understand it. In general, though, in any bash script that you write using regular expressions, be sure to put comments nearby explaining what you intended the regular expression to match.If we added some spaces to the regular expression we would improve its readability, making visual comprehension easier, but it would change the meaning—it would say that we'd need to match space characters at those points in the expression. Ignoring that for the moment, let's insert some spaces into the previous regular expression so that we can read it more easily:[0-9]\{3\} -\{0,1\} [0-9]\{2\} -\{0,1\} [0-9]\{4\}The first grouping says "any digit" then "exactly 3 times." The next grouping says "a dash" then "0 or 1 time." The third grouping says "any digit" then "exactly 2 times." The next grouping says "a dash" then "0 or 1 time." The last grouping says "any digit" then "exactly 4 times."- man regex (Linux, Solaris, HP-UX) or man re_format (BSD, Mac) for the details of your regular expression library
- Classic Shell Scripting by Nelson H.F. Beebe and Arnold Robbins (O'Reilly) Section 3.2, for more about regular expressions and the tools that use them
- Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly)
- , "Expecting to Change Exported Variables"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Grepping Compressed Files
- InhaltsvorschauYou need to grep some compressed files. Do you have to uncompress them first?Not if you have zgrep, zcat, or gzcat on your system.zgrep is simply a grep that understands various compressed and uncompressed files (which types are understood varies from system to system). You will commonly run into this when searching syslog messages on Linux, since the log rotation facilities leave the current log file uncompressed (so it can be in use), but gzip archival logs:
$ zgrep 'search term' /var/log/messages*
zcat is simply a cat that understands various compressed and uncompressed files (which types are understood varies from system to system). It might understand more formats than zgrep, and it might be installed on more systems by default. It is also used in recovering damaged compressed files, since it will simply output everything it possibly can, instead of erroring out as gunzip or other tools might.gzcat is similar to zcat, the differences having to do with commercial versus free Unix variants, and backward compatibility:$ zcat /var/log/messages.1.gz
The less utility may also be configured to transparently display various compressed files, which is very handy. See , "Doing More with less."- , "Compressing Files"
- , "Uncompressing Files"
- , "Doing More with less"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping Some Output, Discarding the Rest
- InhaltsvorschauYou need a way to keep some of your output and discard the rest.The following code prints the first word of every line of input:
$ awk '{print $1}' myinput.fileWords are delineated by whitespace. The awk utility reads data from the filename supplied on the command line, or from standard input if no filename is given. Therefore, you can redirect the input from a file, like this:$ awk '{print $1}' < myinput.fileor even from a pipe, like this:$ cat myinput.file | awk '{print $1}'The awk program can be used in several different ways. Its easiest, simplest use is just to print one or more selected fields from its input.Fields are delineated by whitespace (or specified with the-Foption) and are numbered starting at 1. The field $0 represents the entire line of input.awk is a complete programming language; awk scripts can become extremely complex. This is only the beginning.- man awk
- Effective awk Programming by Arnold Robbins (O'Reilly)
- sed & awk by Arnold Robbins and Dale Dougherty (O'Reilly)
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping Only a Portion of a Line of Output
- InhaltsvorschauYou want to keep only a portion of a line of output, such as just the first and last words. For example, you would like ls to list just filenames and permissions, without all of the other information provided by
ls -l. However, you can't find any options to ls that would limit the output in that way.Pipe ls into awk, and just pull out the fields that you need:$ ls -l | awk '{print $1, $NF}' total 151130 -rw-r--r-- add.1 drwxr-xr-x art drwxr-xr-x bin -rw-r--r-- BuddyIcon.png drwxr-xr-x CDs drwxr-xr-x downloads drwxr-sr-x eclipse ... $Consider the output from thels -lcommand. One line of it looks like this:drwxr-xr-x 2 username group 176 2006-10-28 20:09 bin
so it is convenient for awk to parse (by default, whitespace delineates fields in awk). The output fromls -lhas the permissions as the first field and the filename as the last field.We use a bit of a trick to print the filename. Since the various fields are referenced in awk using a dollar sign followed by the field number (e.g.,$1, $2, $3), and since awk has a built-in variable calledNFthat holds the number of fields found on the current line,$NFalways refers to the last field. (For example, the ls output line has eight fields, so the variableNFcontains 8,so$NFrefers to the eighth field of the input line, which in our example is the filename.)Just remember that you don't use a $ to read the value of an awk variable (unlike bash variables).NFis a valid variable reference by itself. Adding a $ before it changes its meaning from "the number of fields on the current line" to "the last field on the current line."- man awk
- Effective awk Programming by Arnold Robbins (O'Reilly)
- sed & awk by Arnold Robbins and Dale Dougherty (O'Reilly)
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Reversing the Words on Each Line
- InhaltsvorschauYou want to print the input lines with words in the reverse order.
$ awk '{ > for (i=NF; i>0; i--) { > printf "%s ", $i; > } > printf "\n" > }'You don't type the > characters; the shell will print those as a prompt to say that you haven't ended your command yet (it is looking for the matching single-quote mark). Because the awk program is enclosed in single quotes, the bash shell lets us type multiple lines, prompting us with the secondary prompt > until we supply the matching end quote. We spaced out the program for readability, even though we could have stuffed it all onto one line like this:$ awk '{for (i=NF; i>0; i--) {printf "%s ", $i;} printf "\n" }'The awk program has syntax for afor loop, very much in theClanguage style. It even supports aprintfmechanism for formatted output, again modeled after theClanguage version (or the bash version, too). We use theforloop to count down from the last to the first field, and print each field as we go. We deliberately don't put a\non that firstprintfbecause we want to keep the several fields on the same line of output. When the loop is done, we add a newline to terminate the line of output.The reference to$iis very different in awk compared to bash. In bash, when we write$iwe are getting at the value stored in the variable namedi. But in awk, as with most programming languages, we simply reference the value iniby naming it—that is by just writingi. So what is meant by$iin awk? The value of the variableiis resolved to a number, and then the dollar-number expression is understood as a reference to a field (or word) of input—that is, thei-thfield. So asicounts down from the last field to the first, this loop will print the fields in that reversed order.- man printf(1)
- man awk
- Effective awk Programming by Arnold Robbins (O'Reilly)
- sed& awk by Arnold Robbins and Dale Dougherty (O'Reilly)
- "printf" in Appendix A
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Summing a List of Numbers
- InhaltsvorschauYou need to sum a list of numbers, including numbers that don't appear on lines by themselves.Use awk both to isolate the field to be summed and to do the summing. Here we'll sum up the numbers that are the file sizes from the output of an
ls -lcommand:$ ls -l | awk '{sum += $5} END {print sum}'We are summing up the fifth field of thels -loutput. The output ofls-llooks like this:-rw-r--r-- 1 albing users 267 2005-09-26 21:26 lilmax
and the fields are: permissions, links, owner, group, size (in bytes), date, time, and filename. We're only interested in the size, so we use$5in our awk program to reference that field.We enclose the two bodies of our awk program in braces ({}); note that there can be more than one body (or block) of code in an awk program. A block of code preceded by the literal keywordENDis only run once, when the rest of the program has finished. Similarly, you can prefix a block of code withBEGINand supply some code that will be run before any input is read. TheBEGINblock is useful for initializing variables, and we could have used one here to initialize sum, but awk guarantees that variables will start out empty.If you look at the output of anls -lcommand, you will notice that the first line is a total, and doesn't fit our expected format for the other lines.We have two choices for dealing with that. We can pretend it's not there, which is the approach taken above. Since that undesired line doesn't have a fifth field, then our reference to $5 will be empty, and our sum won't change.The more conscientious approach would be to eliminate that field. We could do so before we give the output to awk by using grep:$ ls -l | grep -v '^total' | awk '{sum += $5} END {print sum}'or we could do a similar thing within awk:$ ls -l | awk '/^total/{getline} {sum += $5} END {print sum}'The^totalis a regular expression (regex); it means "the letters to-t-a-l occurring at the beginning of a line" (the leading ^ anchors the search to the beginning of a line). For any line of input matching that regex, the associated block of code will be executed. The second block of code (theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Counting String Values
- InhaltsvorschauYou need to count all the occurrences of several different strings, including some strings whose values you don't know beforehand. That is, you're not trying to count the occurrences of a pre-determined set of strings. Rather, you are going to encounter some strings in your data and you want to count these as-yet-unknown strings.Use awk's associative arrays (also known as hashes) for your counting.For our example, we'll count how many files are owned by various users on our system. The username shows up as the third field in an
ls-loutput. So we'll use that field ($3) as the index of the array, and increment that member of the array:# # cookbook filename: asar.awk # NF < 7 { user[$3]++ } END { for (i in user) { printf "%s owns %d files\n", i, user[i] } }We invoke awk a bit differently here. Because this awk script is a bit more complex, we've put it in a separate file. We use the-foption to tell awk where to get the script file:$ ls -lR /usr/local | awk -f asar.awk bin owns 68 files albing owns 1801 files root owns 13755 files man owns 11491 files $
We use the conditionNF > 7as a qualifier to part of the awk script to weed out the lines that do not contain filenames, which appear in thels -lRoutput and are useful for readability because they include blank lines to separate different directories as well as total counts for each subdirectory. Such lines don't have as many fields (or words). The expressionNF>7that precedes the opening brace is not enclosed in slashes, which is to say that it is not a regular expression. It's a logical expression, much like you would use in anifstatement, and it evaluates to true or false. TheNFvariable is a special built-in variable that refers to the number of fields for the current line of input. So only if a line of input has more than seven fields (words of text) will it be processed by the statements within the braces.The key line, however, is this one:user[$3]++
Here the username (e.g., bin) is used as the index to the array. It's called anEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Showing Data As a Quick and Easy Histogram
- InhaltsvorschauYou need a quick screen-based histogram of some data.Use the associative arrays of awk, as discussed in the previous recipe:
# # cookbook filename: hist.awk # function max(arr, big) { big = 0; for (i in user) { if (user[i] > big) { big=user[i];} } return big } NF > 7{ user[$3]++ } EN { # for scaling maxm = max(user); for (i in user) { #printf "%s owns %d files\n", i, user[i] scaled = 60 * user[i] / maxm ; printf "%-10.10s [%8d]:", i, user[i] for (i=0; i<scaled; i++) { printf "#"; } printf "\n"; } }When we run it with the same input as the previous recipe, we get:$ ls -lR /usr/local | awk -f hist.awk bin [ 68]:# albing [ 1801]:####### root [ 13755]:################################################## man [ 11491]:########################################## $
We could have put the code formaxas the first code inside theENDblock, but we wanted to show you that you can define functions in awk. We are using a bit of fancierprintf. The string format%-10.10swill left justify and pad to 10 characters but also truncate at 10 characters. The integer format%8dwill assure that the integer is printed in an 8 character field. This gives each histogram the same starting point, by using the same amount of space regardless of the username or the size of the integer.Like all arithmetic in awk, the scaling calculation is done with floating point unless we explicitly truncate the result with a call to the built-inint()function. We don't do so, which means that theforloop will execute at least once, so that even the smallest amount of data will still display a single hash mark.The order of data returned from the for (i in user) loop is in no particular order, probably based on some convenient ordering of the underlying hash table. If you wanted the histogram displayed in a sorted order, either numeric by count or alphabetical by username, you would have to add some sorting. One way to do this is to break this program apart into two pieces, sending the output from the first part into theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Showing a Paragraph of Text After a Found Phrase
- InhaltsvorschauYou are searching for a phrase in a document, and want to show the paragraph after the found phrase.We're assuming a simple text file, where paragraph means all the text between blank lines, so the occurrence of a blank line implies a paragraph break. Given that, it's a pretty short awk program:
$ cat para.awk /keyphrase/ { flag=1 } { if (flag == 1) { print $0 } } /^$/ { flag=0 } $ $ awk -f para.awk < searchthis.txtThere are just three simple code blocks. The first is invoked when a line of input matches the regular expression (here just the word "keyphrase"). If "keyphrase" occurs anywhere within a line of input, that is a match and this block of code will be executed. All that happens in this block is that the flag is set.The second code block is invoked for every line of input, since there is no regular expression preceding its open brace. Even the input that matches "keyphrase" will also be applied to this code block (if we didn't want that effect, we could use acontinuestatement in the first block). All this second block does is print the entire input line, but only if the flag is set.The third block has a regular expression that, if satisfied, will simply reset (turn off) the flag. That regular expression uses two characters with special meaning—the caret (^), when used as the first character of a regular expression, matches the beginning of the line; the dollar sign ($), when used as the last character, matches the end of the line. So the regular expression ^$ means "an empty line" because it has no characters between the beginning and end of the line.We could have used a slightly more complicated regular expression for an empty line to let it handle any line with just whitespace rather than a completely blank line. That would make the third line look like this:/^[:blank:]*$/ { flag=0 }Perl programmers love the sort of problem and solution discussed in this recipe, but we've implemented it with awk because Perl is (mostly) beyond the scope of this book. If you know Perl, by all means use it. If not, awk might be all you need.- man awk
- Effective awk Programming
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 8: Intermediate Shell Tools II
- InhaltsvorschauOnce again, we have some useful utilities that are not part of the shell but are used in so many shell scripts that you really should know about them.Sorting is such a common task, and so useful for readability reasons, that it's good to know about the sort command. In a similar vein, the tr command will translate or map from one character to another, or even just delete characters.One common thread here is that these utilities are written not just as standalone commands, but also as filters that can be included in a pipeline of commands. These sorts of commands will typically take one to many filenames as parameters (or arguments), but in the absence of any filenames they will read from standard input. They also write to standard output. That combination makes it easy to connect to the command with pipes, as in something
| sort |even more.That makes them especially useful, and avoids the clutter and confusion of a myriad of temporary files.You would like output in a sorted order, but you don't want to write (yet again) a custom sort function for your program or shell script. Hasn't this been done already?Use the sort utility. You can sort one or more files by putting the file names on the command line:$ sort file1.txt file2.txt myotherfile.xyz
With no filenames on the command, sort will read from standard input so you can pipe the output from a previous command into sort:$ somecommands | sortIt can be handy to have your output in sorted order, and handier still not to have to add sorting code to every program you write. The shell's piping allows you to hook up sort to any program's standard output.There a few options to sort, but two of the three most worth remembering are:$ sort -r
to reverse the order of the sort (where, to borrow a phrase, the last shall be first and the first, last); and$ sort -f
to "fold" lower- and uppercase characters together; i.e., to ignore the case differences. This can be done either with the-foption or with a GNU long-format option:$ sort --ignore-case
We decided to keep you in suspense, so see the next recipe, , "Sorting Numbers," for the third coolestEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sorting Your Output
- InhaltsvorschauYou would like output in a sorted order, but you don't want to write (yet again) a custom sort function for your program or shell script. Hasn't this been done already?Use the sort utility. You can sort one or more files by putting the file names on the command line:
$ sort file1.txt file2.txt myotherfile.xyz
With no filenames on the command, sort will read from standard input so you can pipe the output from a previous command into sort:$ somecommands | sortIt can be handy to have your output in sorted order, and handier still not to have to add sorting code to every program you write. The shell's piping allows you to hook up sort to any program's standard output.There a few options to sort, but two of the three most worth remembering are:$ sort -r
to reverse the order of the sort (where, to borrow a phrase, the last shall be first and the first, last); and$ sort -f
to "fold" lower- and uppercase characters together; i.e., to ignore the case differences. This can be done either with the-foption or with a GNU long-format option:$ sort --ignore-case
We decided to keep you in suspense, so see the next recipe, , "Sorting Numbers," for the third coolest sort option.- man sort
- , "Sorting Numbers"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sorting Numbers
- InhaltsvorschauWhen sorting numeric data you notice that the order doesn't seem right:
$ sort somedata 2 200 21 250 $
You need to tell sort that the data should be sorted as numbers. Specify a numeric sort with the-noption:$ sort -n somedata 2 21 200 250 $
There is nothing wrong with the original (if odd) sort order if you realize that it is an alphabetic sort on the data (i.e., 21 comes after 200 because 1 comes after 0 in an alphabetic sort). Of course, what you probably want is numeric ordering, so you need to use the-noption.sort -rncan be very handy in giving you a descending frequency list of something when combined withuniq -c. For example, let's display the most popular shells on this system:$ cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn 20 /bin/sh 10 /bin/false 2 /bin/bash 1 /bin/sync
cut -d':' -f7 /etc/passwdisolates the shell from the /etc/passwd file. Then we have to do an initial sort so that uniq will work.uniq -ccounts consecutive, duplicate lines, which is why we need the pre-sort. Thensort -rngives us a reverse, numerical sort, with the most popular shell at the top.If you don't need to count the occurrences and just want a unique list of values—i.e., if you want sort to remove duplicates—then you can use the-uoption on the sort command (and omit the uniq command). So to find just the list of different shells on this system:cut -d':' -f7 /etc/passwd | sort -u
- man sort
- man uniq
- man cut
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sorting IP Addresses
- InhaltsvorschauYou want to sort a list of numeric IP address, but you'd like to sort by the last portion of the number or by the entire address logically.To sort by the last octet only (old syntax):
$ sort -t. -n +3.0 ipaddr.list 10.0.0.2 192.168.0.2 192.168.0.4 10.0.0.5 192.168.0.12 10.0.0.20 $
To sort the entire address as you would expect (POSIX syntax):$ sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list 10.0.0.2 10.0.0.5 10.0.0.20 192.168.0.2 192.168.0.4 192.168.0.12 $
We know this is numeric data, so we use the-noption. The-toption indicates the character to use as a separator between fields (in our case, a period) so that we can also specify which fields to sort first. In the first example, we start sorting with the third field (zero-based) from the left, and the very first character (again, zero-based) of that field, so+3.0.In the second example, we used the new POSIX specification instead of the traditional (but obsolete)+pos1 -pos2method. Unlike the older method, it is not zero-based, so fields start at 1.$ sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n ipaddr.list
Wow, that's ugly. Here it is in the old format:sort -t. +0n -1 +1n -2 +2n -3 +3n -4, which is just as bad.Using-t.to define the field delimiter is the same, but the sort-key fields are given quite differently. In this case,-k 1,1nmeans "start sorting at the beginning of field one(1)and (,) stop sorting at the end of field one(1)and do a numerical sort(n). Once you get that, the rest is easy. When using more than one field, it's very important to tell sort where to stop. The default is to go to the end of the line, which is often not what you want and which will really confuse you if you don't understand what it's doing.The order that sort uses is affected by your locale setting. If your results are not as expected, that's one thing to check.Your sort order will vary from system to system depending on whether your sort command defaults to using a stable sort. A stable sort preserves the original order in the sorted data when the sort fields are equal. Linux and Solaris do not default to a stable sort, but NetBSD does. And whileEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Cutting Out Parts of Your Output
- InhaltsvorschauYou need to look at only part of your fixed-width or column-based data. You'd like to take a subset of it, based on the column position.Use the cut command with the
-coption to take particular columns: Note that our example'ps'command only works with certain systems; e.g., CentOS-4, Fedora Core 5, and Ubuntu work, but Red Hat 8, NetBSD, Solaris, and Mac OS X all garble the output due to using different columns:$ ps -l | cut -c12-15 PID 5391 7285 7286 $
or:$ ps -elf | cut -c58- (output not shown)
With the cut command we specify what portion of the lines we want to keep. In the first example, we are keeping columns 12 (starting at column one) through 15, inclusive. In the second case, we specify starting at column 58 but don't specify the end of the range so that cut will take from column 58 on through the end of the line.Most of the data manipulation we've looked at has been based on fields, relative positions separated by characters called delimiters. The cut command can do that too, but it is one of the few utilities that you'll use with bash that can also easily deal with fixed-width, columnar data (via the-coption).Using cut to print out fields rather than columns is possible, though more limited than other choices such as awk. The default delimiter between fields is the Tab character, but you can specify a different delimiter with the-doption. Here is an example of a cut command using fields:$ cut -d'#' -f2 < ipaddr.list
and an equivalent awk command:$ awk -F'#' '{print $2}' < ipaddr.listYou can even use cut to handle non-matching delimiters by using more than one cut. You may be better off using a regular expression with awk for this, but sometimes a couple of quick and dirty cuts are faster to figure out and type.Here is how you can get the field out from between square brackets. Note that the first cut uses a delimiter of open square bracket(-d'[')and field 2 (-f2starting at 1). Because the first cut has already removed part of the line, the second cut uses a delimiter of closed square bracket(-d']')and field 1(-f1).$ cat delimited_data Line [l1]. Line [l2]. Line [l3]. $ cut -d'[' -f2 delimited_data | cut -d']' -f1 l1 l2 l3
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Removing Duplicate Lines
- InhaltsvorschauAfter selecting and/or sorting some data you notice that there are many duplicate lines in your results. You'd like to get rid of the duplicates, so that you can see just the unique values.You have two choices available to you. If you've just been sorting your output, add the
-uoption to the sort command:$ somesequence | sort -uIf you aren't running sort, just pipe the output into uniq—provided, that is, that the output is sorted, so that identical lines are adjacent:$ somesequence > myfile $ uniq myfileSince uniq requires the data to be sorted already, we're more likely to just add the-uoption to sort unless we also need to count the number of duplicates (-c, see , "Sorting Numbers"), or see only the duplicates(-d), which uniq can do.Don't accidentally overwrite a valuable file by mistake; the uniq command is a bit odd in its parameters. Whereas most Unix/Linux commands take multiple input files on the command line, uniq does not. In fact, the first (non-option) argument is taken to be the (one and only) input file and any second argument, if supplied, is taken as the output file. So if you supply two filenames on the command line, the second one will get clobbered without warning.- man sort
- man uniq
- , "Sorting Numbers"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Compressing Files
- InhaltsvorschauYou need to compress some files and aren't sure of the best way to do it.First, you need to understand that in traditional Unix, archiving (or combining) and compressing files are two different operations using two different tools, while in the DOS and Windows world it's typically one operation with one tool. A "tarball" is created by combining several files and/or directories using the tar (tape archive) command, then compressed using the compress, gzip, or bzip2 tools. This results in files like tarball.tar.Z, tarball.tar.gz, tarball.tgz, or tarball.tar.bz2. Having said that, many other tools, including zip, are supported.In order to use the correct format, you need to understand where your data will be used. If you are simply compressing some files for yourself, use whatever you find easiest. If other people will need to use your data, consider what platform they will be using and what they are comfortable with.The Unix traditional tarball was tarball.tar.Z, but gzip is now much more common and bzip2 (which offers better compression than gzip) is gaining ground. There is also a tool question. Some versions of tar allow you to use the compression of your choice automatically while creating the archive. Others don't.The universally accepted Unix or Linux format would be a tarball.tar.gz created like this:
$ tar cf tarball_name.tar directory_of_files $ gzip tarball_name.tar
If you have GNU tar, you could use-Zfor compress (don't, this is obsolete),-zfor gzip (safest), or-jfor bzip2 (highest compression). Don't forget to use an appropriate filename, this is not automatic.$ tar czf tarball_name.tgz directory_of_files
While tar and gzip are available for many platforms, if you need to share with Windows you are better off using zip, which is nearly universal. zip and unzip are supplied by the InfoZip packages on Unix and almost any other platform you can possibly think of. Unfortunately, they are not always installed by default. Run the command by itself for some helpful usage information, since these tools are not like most other Unix tools. And note theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Uncompressing Files
- InhaltsvorschauYou need to uncompress one or more files ending in extensions like
tar, tar.gz, gz, tgz, Z, orzip.Figure out what you are dealing with and use the right tool. maps common extensions to programs capable of handling them.Table 8-3: Common file extensions and compression utilities File extensionCommand.tartar tf(list contents),tar xf(extract).tar.gz, .tgzGNU tar:tar tzf(list contents),tar xzf(extract)else:gunzipfile &&tar xffile.tar.bz2GNU tar:tar tjf(list contents),tar xjf(extract)else:gunzip2file &&tar xffile.tar.ZGNU tar:tar tZf(list contents),tar xZf(extract)else:uncompressfile &&tar xffile.zipunzip(often not installed by default)You should also try the file command:$ file what_is_this.* what_is_this.1: GNU tar archive what_is_this.2: gzip compressed data, from Unix $ gunzip what_is_this.2 gunzip: what_is_this.2: unknown suffix -- ignored $ mv what_is_this.2 what_is_this.2.gz $ gunzip what_is_this.2.gz $ file what_is_this.2 what_is_this.2: GNU tar archive
If the file extension matches none of those listed in and the file command doesn't help, but you are sure it's an archive of some kind, then you should do a web search for it.- , "Grepping Compressed Files"
- , "Compressing Files"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Checking a tar Archive for Unique Directories
- InhaltsvorschauYou want to untar an archive, but you want to know beforehand into which directories it is going to write. You can look at the table of contents of the tarfile by using
tar-t, but this output can be very large and it's easy to miss something.Use an awk script to parse off the directory names from the tar archive's table of contents, then usesort -uto leave you with just the unique directory names:$ tar tf some.tar | awk -F/ '{print $1}' | sort -uThetoption will produce the table of contents for the file specified with thefoption whose filename follows. The awk command specifies a non-default field separator by using-F/to specify a slash as the separator between fields. Thus, theprint $1will print the first directory name in the pathname.Finally, all the directory names will be sorted and only unique ones will be printed.If a line of the output contains a single period then some files will be extracted into the current directory when you unpack this tar file, so be sure to be in the directory you desire.Similarly, if the filenames in the archive are all local and without a leading ./ then you will get a list of filenames that will be created in the current directory.If the output contains a blank line, that means that some of the files are specified with absolute pathnames (i.e., beginning with /), so again be careful, as extracting such an archive might clobber something that you don't want replaced.- man tar
- man awk
- , "Sorting Your Output"
- , "Sorting Numbers"
- , "Sorting IP Addresses"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Translating Characters
- InhaltsvorschauYou need to convert one character to another in all of your text.Use the tr command to translate one character to another. For example:
$ tr ';' ',' <be.fore >af.ter
In its simplest form, a tr command replaces occurrences of the first (and only) character of the first argument with the first (and only) character of the second argument.In the example solution, we redirected input from the file named be.fore and sent the output into the file named af.ter and we translated all occurrences of a semicolon into a comma.Why do we use the single quotes around the semicolon and the comma? Well, a semicolon has special meaning to bash, so if we didn't quote it bash would break our command into two commands, resulting in an error. The comma has no special meaning, but we quote it out of habit to avoid any special meaning we may have forgotten about—i.e., it's safer always to use the quotes, then we never forget to use them when we need them.The tr command can do more that one translation at a time by putting the several characters to be translated in the first argument and their corresponding resultant characters in the second argument. Just remember, it's a one-for-one substitution. For example:$ tr ';:.!?' ',' <other.punct >commas.all
will translate all occurrences of the punctuation symbols of semicolon, colon, period, exclamation point and question mark to commas. Since the second argument is shorter than the first, its last (and here, its only) character is repeated to match the length of the first argument, so that each character has a corresponding character for the translation.Now this kind of translation could be done with the sed command, though sed syntax is a bit trickier. The tr command is not as powerful, since it doesn't use regular expressions, but it does have some special syntax for ranges of characters—and that can be quite useful as we'll see in , "Converting Uppercase to Lowercase."- man tr
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Converting Uppercase to Lowercase
- InhaltsvorschauYou need to eliminate case distinctions in a stream of text.You can translate all uppercase characters (A–Z) to lowercase (a–z) using the tr command and specifying a range of characters, as in:
$ tr 'A-Z' 'a-z' <be.fore >af.ter
There is also special syntax in tr for specifying this sort of range for upper-and lower-case conversions:$ tr '[:upper:]' '[:lower:]' <be.fore >af.ter
Although tr doesn't support regular expressions, it does support a range of characters. Just make sure that both arguments end up with the same number of characters. If the second argument is shorter, its last character will be repeated to match the length of the first argument. If the first argument is shorter, the second argument will be truncated to match the length of the first.Here's a very simplistic encoding of a text message using a simple substitution cypher that offsets each character by 13 places (i.e., ROT13). An interesting characteristic of ROT13 is that the same process is used to both encipher and decipher the text:$ cat /tmp/joke Q: Why did the chicken cross the road? A: To get to the other side. $ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke D: Jul qvq gur puvpxra pebff gur ebnq? N: Gb trg gb gur bgure fvqr. $ tr 'A-Za-z' 'N-ZA-Mn-za-m' < /tmp/joke | tr 'A-Za-z' 'N-ZA-Mn-za-m' Q: Why did the chicken cross the road? A: To get to the other side.
- man tr
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Converting DOS Files to Linux Format
- InhaltsvorschauYou need to convert DOS formatted text files to the Linux format. In DOS, each line ends with a pair of characters—the return and the newline. In Linux, each line ends with a single newline. So how can you delete that extra DOS character?Use the
-doption on tr to delete the character(s) in the supplied list. For example, to delete all DOS carriage returns(\r), use the command:$ tr -d '\r' <file.dos >file.txt
This will delete all\rcharacters in the file, not just those at the end of a line. Typical text files rarely have characters like that inline, but it is possible. You may wish to look into the dos2unix and unix2dos programs if you are worried about this.The tr utility has a few special escape sequences that it recognizes, among them\rfor carriage return and\nfor newline. The other special backslash sequences are listed in .Table 8-4: The special escape sequences of the tr utility SequenceMeaning\oooCharacter with octal value ooo (1-3 octal digits)\\A backslash character (i.e., escapes the backslash itself)\a"audible" bell, the ASCII BEL character (since "b" was taken for backspace)\bBackspace\fForm feed\nNewline\rReturn\tTab (sometimes called a "horizontal" tab)\vVertical tab- man tr
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Removing Smart Quotes
- InhaltsvorschauYou want simple ASCII text out of a document in MS Word, but when you save it as text some odd characters still remain.Translate the odd characters back to simple ASCII like this:
$ tr '\221\222\223\224\226\227' '\047\047""--' <odd.txt >plain.txt
Such "smart quotes" come from the Windows-1252 character set, and may also show up in email messages that you save as text. To quote from Wikipedia on this subject:A few mail clients send curved quotes using the Windows-1252 codes but mark the text as ISO-8859-1 causing problems for decoders that do not make the dubious assumption that C1 control codes in ISO-8859-1 text were meant to be Windows-1252 printable characters.To clean up such text, we can use the tr command. The221and222(octal) curved single-quotes will be translated to simple single quotes. We specify them in octal(047)to make it easier on us, since the shell uses single quotes as a delimiter. The223and224(octal) are opening and closing curved quotes, and will be translated to simple double quotes. The double quotes can be typed within the second argument since the single quotes protect them from shell interpretation. The226and227(octal) are dash characters and will be translated to hyphens (and no, that second hyphen in the second argument is not technically needed, since tr will repeat the last character to match the length of the first argument, but it's better to be specific).- man tr
- http://en.wikipedia.org/wiki/Curved_quotes for way more than you might ever have wanted to know about quotation marks and related character set issues
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Counting Lines, Words, or Characters in a File
- InhaltsvorschauYou need to know how many lines, words, or characters are in a given file.Use the wc (word count) command with awk in a command substitution.The normal output of wc is something like this:
$ wc data_file 5 15 60 data_file # Lines only $ wc -l data_file 5 data_file # Words only $ wc -w data_file 15 data_file # Characters (often the same as bytes) only $ wc -c data_file 60 data_file # Note 60B $ ls -l data_file -rw-r--r-- 1 jp users 60B Dec 6 03:18 data_file
You may be tempted to just do something like this:data_file_lines=$(wc -l "$data_file")
That won't do what you expect, since you'll get something like"5 data_file"as the value. Instead, try this:data_file_lines=$(wc -l "$data_file" | awk '{print $1}')If your version of wc is locale aware, the number of characters will not equal the number of bytes in some character sets.- man wc
- , "Splitting Output Only When Necessary"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Rewrapping Paragraphs
- InhaltsvorschauYou have some text with lines that are too long or too short, so you'd like to re-wrap them to be more readable.Use the fmt command, optionally with a goal and maximum line length:
$ fmt mangled_text $ fmt 55 60 mangled_text
One tricky thing about fmt is that it expects blank lines to separate headers and paragraphs. If your input file doesn't have those blanks, it has no way to tell the difference between different paragraphs and extra newlines inside the same paragraph. So you will end up with one giant paragraph, with the correct line lengths.The pr command might also be of some interest for formatting text.- man fmt
- man pr
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Doing More with less
- Inhaltsvorschau"less is more!"You'd like to take better advantage of the features of the
lesspager.Read the less manpage and use the$LESSvariable with ~/.lessfilter and ~/.lesspipe files.less takes options from the$LESSvariable, so rather than creating an alias with your favorite options, put them in that variable. It takes both long and short options, and any command-line options will override the variable. We recommend using the long options in the$LESSvariable since they are easy to read. For example:export LESS="--LONG-PROMPT --LINE-NUMBERS --ignore-case --QUIET"
But that is just the beginning. less is expandable via input preprocessors, which are simply programs or scripts that pre-process the file that less is about to display. This is handled by setting the$LESSOPENand$LESSCLOSEenvironment variables appropriately.You could build your own, but save yourself some time and look into Wolfgang Friebel's lesspipe.sh available at http://www-zeuthen.desy.de/~friebel/unix/lesspipe.html (but see the discussion below first). The script works by setting and exporting the$LESSOPENenvironment variable when run by itself:$ ./lesspipe.sh LESSOPEN="|./lesspipe.sh %s" export LESSOPEN
So you simply run it in an eval statement, likeeval $(/path/to/lessfilter.sh)oreval`/path/to/lessfilter.sh`, and then use less as usual. The list of supported formats for version 1.53 is:gzip, compress, bzip2, zip, rar, tar, nroff, ar archive, pdf, ps, dvi, shared library, executable, directory, RPM, Microsoft Word, OpenOffice 1.x and OASIS (OpenDocument) formats, Debian, MP3 files, image formats (png, gif, jpeg, tiff, …), utf-16 text, iso images and filesystems on removable media via /dev/xxxBut there is a catch. These formats require various external tools, so not all features in the example lesspipe.sh will work if you don't have them. The package also contains ./configure (or make) scripts to generate a version of the filter that will work on your system, given the tools that you have available.less is unique in that it is a GNU tool that was already installed by default on every single test system we tried—every one. Not even bash can say this. And version differences aside, it works the same on all of them. Quite a claim to fame.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 9: Finding Files: find, locate, slocate
- InhaltsvorschauHow easy is it for you to search for files throughout your filesystem?For the first few files that you created, it was easy enough just to remember their names and where you kept them. Then when you got more files, you created subdirectories (or folders in GUI-speak) to clump your files into related groups. Soon there were subdirectories inside of subdirectories, and now you are having trouble remembering where you put things. Of course, with larger and larger disks it is getting easier to just keep creating and never deleting any files (and for some of us, this getting older thing isn't helping either).But how do you find that file you were just editing last week? Or the attachment that you saved in a subdirectory (it seemed such a logical choice at the time). Or maybe your filesystem has become cluttered with MP3 files scattered all over it.Various attempts have been made to provide graphical interfaces to help you search for files, which is all well and good—but how do you use the results from a GUI-style search as input to other commands?bash and the GNU tools can help. They provide some very powerful search capabilities that enable you to search by filename, dates of creation or modification, even content. They send the results to standard output, perfect for use in other commands or scripts.So stop your wondering—here's the information you need.You have MP3 audio files scattered all over your filesystem. You'd like to move them all into a single location so that you can organize them and then copy them onto a music player.The find utility can locate all of those files and then execute a command to move them where you want. For example:
$ find . -name '*.mp3' -print -exec mv '{}' ~/songs \;The syntax for the find utility is unlike other Unix tools. It doesn't use options in the typical way, with dash and single-letter collections up front followed by several words of arguments. Rather, the options look like short words, and are ordered in a logical sequence describing the logic of which files are to be found, and what to do with them, if anything, when they are found. These word-like options are often calledEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding All Your MP3 Files
- InhaltsvorschauYou have MP3 audio files scattered all over your filesystem. You'd like to move them all into a single location so that you can organize them and then copy them onto a music player.The find utility can locate all of those files and then execute a command to move them where you want. For example:
$ find . -name '*.mp3' -print -exec mv '{}' ~/songs \;The syntax for the find utility is unlike other Unix tools. It doesn't use options in the typical way, with dash and single-letter collections up front followed by several words of arguments. Rather, the options look like short words, and are ordered in a logical sequence describing the logic of which files are to be found, and what to do with them, if anything, when they are found. These word-like options are often called predicates.A find command's first arguments are the directory or directories in which to search. A typical use is simply (.) for the current directory. But you can provide a whole list of directories, or even search the entire filesystem (permissions allowing) by specifying the root of the filesystem (/) as the starting point.In our example the first option (the-namepredicate) specifies the pattern we will search for. Its syntax is like the bash pattern matching syntax, so*.mp3will match all filenames that end in the characters ".mp3". Any file that matches this pattern is considered to return true and will thus continue to the next predicate of the command.Think of it this way: find will climb around on the filesystem and each filename that it finds it will present to this gauntlet of conditions that must be run. Any condition that is true is passed. Encounter a false and that filename's turn is immediately over, and the next filename is processed.Now the-printcondition is easy. It is always true and it has the side effect of printing the name to standard output. So any file that has made it this far in the sequence of conditions will have its name printed.The-execis a bit odd. Any filename making it this far will become part of a command that is executed. The remainder of the lineup to the \; is the command to be executed. The {} is replaced by the name of the file that was found. So in our example, ifEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Handling Filenames Containing Odd Characters
- InhaltsvorschauYou used a find command like the one in , "Finding All Your MP3 Files" but the results were not what you intended because many of your filenames contain odd characters.First, understand that to Unix folks, odd means "anything not a lowercase letter, or maybe a number." So uppercase, spaces, punctuation, and character accents are all odd. But you'll find all of those and more in the names of many songs and bands.Depending on the oddness of the characters, your system, tools, and goal, it might be enough to simply quote the replacement string (i.e., put single quotes around the {}, as in '{}') . You did test your command first, right?If that's no good, try using the
-print0argument to find and the-0argument to xargs.-print0tells find to use the null character (\0) instead of whitespace as the output delimiter between pathnames found.-0then tells xargs the input delimiter. These will always work, but they are not supported on every system.The xargs command takes whitespace delimited (except when using -0) pathnames from standard input and executes a specified command on as many of them as possible (up to a bit less than the system'sARG_MAXvalue; see , "Working Around "argument list too long" Errors"). Since there is a lot of overhead associated with calling other commands, using xargs can drastically speed up operations because you are calling the other command as few times as possible, rather than each time a pathname is found.So, to rewrite the solution from , "Finding All Your MP3 Files" to handle odd characters:$ find . -name '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songsHere is a similar example demonstrating how to use xargs to work around spaces in a path or filename when locating and then coping files:$ locate P1100087.JPG PC220010.JPG PA310075.JPG PA310076.JPG | xargs -i cp '{}' .There are two problems with this approach. One is that not all versions of xargs support the-ioption, and the other is that the-ioption eliminates argument grouping, thus negating the speed increase we were hoping for. The problem is that the mv command needs the destination directory as the final argument, but traditionalEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Speeding Up Operations on Found Files
- InhaltsvorschauYou used a find command like the one in , "Finding All Your MP3 Files" and the resulting operations take a long time because you found a lot of files, so you want to speed it up.See the discussion on xargs , "Handling Filenames Containing Odd Characters."
- , "Finding All Your MP3 Files"
- , "Handling Filenames Containing Odd Characters"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files Across Symbolic Links
- InhaltsvorschauYou issued a find command to find your .mp3 files but it didn't find all of them—it missed all those that were part of your filesystem but were mounted via a symbolic link. Is find unable to cross that kind of boundary?Use the
-followpredicate. The example we used before becomes:$ find . -follow -name '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songsSometimes you don't want find to cross over onto other filesystems, which is where symbolic links originated. So the default for find is not to follow a symbolic link. If you do want it to do so, then use the-followoption as the first option in the list on your find command.- man find
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files Irrespective of Case
- InhaltsvorschauSome of your MP3 files end with .MP3 rather than .mp3. How do you find those?Use the
-inamepredicate (if your version of find supports it) to run a case-insensitive search, rather than just-name. For example:$ find . -follow -iname '*.mp3' -print0 | xargs -i -0 mv '{}' ~/songsSometimes you care about the case of the filename and sometimes you don't. Use the-inameoption when you don't care, in situations like this, where .mp3 or .MP3 both indicate that the file is probably an MP3 file. (We say probably because on Unix-like systems you can name a file anything that you want. It isn't forced to have a particular extension.)One of the most common places where you'll see the upper- and lowercase issue is when dealing with Microsoft Windows-compatible filesystems, especially older or "lowest common denominator" filesystems. A digital camera that we use stores its files with filenames like PICT001.JPG, incrementing the number with each picture. If you were to try:$ find . -name '*.jpg' -print
you wouldn't find many pictures. In this case you could also try:$ find . -name '*.[Jj][Pp][Gg]' -print
since that regular expression will match either letter in brackets, but that isn't as easy to type, especially if the pattern that you want to match is much longer. In practice,-inameis an easier choice. The catch is that not every version of find supports the-inamepredicate. If your system doesn't support it, you could try tricky regular expressions as shown above, use multiple-nameoptions with the case variations you expect, or install the GNU version of find.- man find
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files by Date
- InhaltsvorschauSuppose someone sent you a JPEG image file that you saved on your filesystem a few months ago. Now you don't remember where you put it. How can you find it?Use a find command with the
-mtimepredicate, which checks the date of last modification. For example:find . -name '*.jpg' -mtime +90 -print
The-mtimepredicate takes an argument to specify the timeframe for the search. The90stands for 90 days. By using a plus sign on the number (+90) we indicate that we're looking for a file modified more than 90 days ago. Write-90(using a minus sign) for less than 90 days. Use neither a plus nor minus to mean exactly 90 days.There are several predicates for searching based on file modification times and each take a quantity argument. Using a plus, minus, or no sign indicates greater than, less than, or equals, respectively, for all of those predicates.The find utility also has logical AND, OR, and NOT constructs so if you know that the file was at least one week old (7 days) but not more than 14 days old, you can combine the predicates like this:$ find . -mtime +7 -a -mtime -14 -print
You can get even more complicated using OR as well as AND and even NOT to combine conditions, as in:$ find . -mtime +14 -name '*.text' -o \( -mtime -14 -name '*.txt' \) -print
This will print out the names of files ending in.textthat are older than 14 days, as well as those that are newer than 14 days but have.txtas their last 4 characters.You will likely need parentheses to get the precedence right. Two predicates in sequence are like a logical AND, which binds tighter than an OR (in find as in most languages). Use parentheses as much as you need to make it unambiguous.Parentheses have a special meaning to bash, so we need to escape that meaning, and write them as \( and \) or inside of single quotes as '(' and ')'. You cannot use single quotes around the entire expression though, as that will confuse the find command. It wants each predicate as its own word.- man find
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files by Type
- InhaltsvorschauYou are looking for a directory with the word "java" in it. When you tried:
$ find . -name '*java*' -print
you got way too many files—including all the Java source files in your part of the filesystem.Use the-typepredicate to select only directories:$ find . -type d -name '*java*' -print
We put the-type dfirst followed by the-name *java*. Either order would have found the same set of files. By putting the-type dfirst in the list of options, though, the search will be slightly more efficient: as each file is encountered, the test will be made to see if it is a directory and then only directories will have their names checked against the pattern. All files have names; relatively few are directories. So this ordering eliminates most files from further consideration before we ever do the string comparison. Is it a big deal? With processors getting faster all the time, it matters less so. With disk sizes getting bigger all the time, it matters more so. There are several types of files for which you can check, not just directories. lists the single characters used to find these types of files.Table 9-1: Characters used by find's -type predicate KeyMeaningbblock special fileccharacter special fileddirectoryppipe (or "fifo")fplain ol' filelsymbolic linkssocketD(Solaris only) "door"- man find
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files by Size
- InhaltsvorschauYou want to do a little housecleaning, and to get the most out of your effort you are going to start by finding your largest files and deciding if you need to keep them around. But how do you find your largest files?Use the -size predicate in the find command to select files above, below, or exactly a certain size. For example:
find . -size +3000k -print
Like the numeric argument to-mtime, the-sizepredicate's numeric argument can be preceded by a minus sign, plus sign, or no sign at all to indicate less than, greater than, or exactly equal to the numeric argument. So we've indicated, in our example, that we're looking for files that are greater than the size indicated.The size indicated includes a unit ofkfor kilobytes. If you usecfor the unit, that means just bytes (or characters). If you use b, or don't put any unit, that indicates a size in blocks. (The block is a 512-byte block, historically a common unit in Unix systems.) So we're looking for files that are greater than 3 MB in size.- man find
- man du
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Files by Content
- InhaltsvorschauHow do you find a file of some known content? Let's say that you had written an important letter and saved it as a text file, putting .txt on the end of the filename. Beyond that, the only thing you remember about the content of the letter is that you had used the word "portend."If you are in the vicinity of that file, say within the current directory, you can start with a simple grep:
grep -i portend *.txt
With the-ioption, grep will ignore upper-and lowercase difference. This command may not be sufficient to find what you're looking for, but start simply. Of course, if you think the file might be in one of your many subdirectories, you can try to reach all the files that are in subdirectories of the current directory with this command:grep -i portend */*.txt
Let's face it, though, that's not a very thorough search.If that doesn't do it, let's use a more complete solution: the find command. Use the-execoption on find so that if the predicates are true up to that point, it will execute a command for each file it finds. You can invoke grep or other utilities like this:find . -name '*.txt' -exec grep -Hi portend '{}' \;We use the-name '*.txt'construct to help narrow down the search. Any such test will help, since having to run a separate executable for each file that it finds is costly in time and CPU horsepower. Maybe you have a rough idea of how old the file is (e.g.,-mdate -5or some such).The '{}' is where the filename is put when executing the command. The \; indicates the end of the command, in case you want to continue with more predicates. Both the braces and the semicolon need to be escaped, so we quote one and use the backslash for the other. It doesn't matter which way we escape them, only that we do escape them, so that bash doesn't misinterpret them.On some systems, the-Hoption will print the name of the file if grep finds something. Normally, with only one filename on the command, grep won't bother to name the file, it just prints out the matching line that it finds. Since we're searching through many files, we need to know which file was grepped.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Existing Files and Content Fast
- InhaltsvorschauYou'd like to be able to find files without having to wait for a long find command to complete, or you need to find a file with some specific content.If your system has locate, slocate, Beagle, Spotlight or some other indexer, you are already set. If not, look into them.As we discussed in , "Finding and Running Commands", locate and slocate consult database files about the system (usually compiled and updated by a cron job) to find file or command names almost instantly. The location of the actual database files, what is indexed therein, and how often, may vary from system to system. Consult your system's manpages for details.
$ locate apropos /usr/bin/apropos /usr/share/man/de/man1/apropos.1.gz /usr/share/man/es/man1/apropos.1.gz /usr/share/man/it/man1/apropos.1.gz /usr/share/man/ja/man1/apropos.1.gz /usr/share/man/man1/apropos.1.gz
locate and slocate don't index content though, so see , "Finding Files by Content" for that.Beagle and Spotlight are examples of a fairly recent technology known as desktop search engines or indexers. Google Desktop Search and Copernic Desktop Search are two examples from the Microsoft Windows world. Desktop search tools use some kind of indexer to crawl, parse, and index the names and contents of all of the files (and usually email messages) in your personal file space; i.e., your home directory on a Unix or Linux system. This information is then almost instantly available to you when you look for it. These tools are usually very configurable, graphical, operate on a per-user basis, and index the contents of your files.slocate stores permission information (in addition to filenames and paths) so that it will not list programs to which the user does not have access. On most Linux systems locate is a symbolic link to slocate; other systems may have separate programs, or may not have slocate at all. Both of these are command-line tools that crawl and index the entire filesystem, more or less, but they only contain filenames and locations.- man locate
- man slocate
- , "Finding and Running Commands"
- , "Finding Files by Content"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding a File Using a List of Possible Locations
- InhaltsvorschauYou need to execute, source, or read a file, but it may be located in a number of different places in or outside of the
$PATH.If you are going to source the file and it's located somewhere on the$PATH, just source it. bash's built-in source command (also known by the shorter-to-type but harder-to-read POSIX name ".") will search the$PATHif thesourcepathshell option is set, which it is by default:$ source myfile
If you want to execute a file only if you know it exists in the$PATHand is executable, and you have bash version 2.05b or higher, usetype -Pto search the $PATH. Unlike the which command,type -Ponly produces output when it finds the file, which makes it much easier to use in this case:LS=$(type -P ls) [ -x $LS ] && $LS # --OR-- LS=$(type -P ls) if [ -x $LS ]; then : commands involving $LS here fi
If you need to look in a variety of locations, possibly including the$PATH, use a for loop. To search the$PATH, use the variable substitution operator${variable/pattern/ replacement}to replace the : separator with a space, and then use for as usual. To search the$PATHand other possible locations, just list them:for path in ${PATH//:/ }; do [ -x "$path/ls" ] && $path/ls done # --OR-- for path in ${PATH//:/ } /opt/foo/bin /opt/bar/bin; do [ -x "$path/ls" ] && $path/ls doneIf the file is not in the$PATH, but could be in a list of locations, possibly even under different names, list the entire path and name:for file in /usr/local/bin/inputrc /etc/inputrc ~/.inputrc; do [ -f "$file" ] && bind -f "$file" && break # Use the first one found done
Perform any additional tests as needed. For example, you may wish to use screen when logging in if it's present on the system:for path in ${PATH//:/ }; do if [ -x "$path/screen" ]; then # If screen(1) exists and is executable: for file in /opt/bin/settings/run_screen ~/settings/run_screen; do [ -x "$file" ] && $file && break # Execute the first one found done fi doneSee , "Getting Started with a Custom Configuration" for more details on this code fragment.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 10: Additional Features for Scripting
- InhaltsvorschauMany scripts are written as simple one-off scripts that are only used by their author, consisting of only a few lines, perhaps only a single loop, if that. But some scripts are heavy-duty scripts that will see a lot of use from a variety of users. Such scripts will often need to take advantage of features that allow for better sharing and reuse of code. These advanced scripting tips and techniques can be useful for many kinds of scripts, and are often found in larger systems of scripts such as the /etc/init.d scripts on many Linux systems. You don't have to be a system administrator to appreciate and use these techniques. They will prove themselves on any large scripting effort.Sometimes you want a script to run as a daemon, i.e., in the background and never ending. To do this properly you need to be able to detach your script from its controlling tty, that is from the terminal session used to start the daemon. Simply putting an ampersand on the command isn't enough. If you start your daemon script on a remote system via an SSH (or similar) session, you'll notice that when you log out, the SSH session doesn't end and your window is hung until that script ends (which, being a daemon, it won't).Use the following to invoke your script, run it in the background, and still allow yourself to log out:
nohup mydaemonscript 0<&-1>/dev/null 2>&1 &
or:nohup mydaemonscript >>/var/log/myadmin.log 2>&1 <&- &
You need to close the controlling tty, which is connected in three ways to your (or any) job: standard input (STDIN), standard output (STDOUT), and standard error (STDERR). We can close STDOUT and STDERR by pointing them at another file— typically either a log file, so that you can retrieve their output at a later time, or at the file /dev/null to throw away all their output. We use the redirecting operator > to do this.But what about STDIN? The cleanest way to deal with STDIN is to close the file descriptor. The bash syntax to do that is like a redirect, but with a dash for the file-name (0<&-or <&-).We use the nohup command so that the script is run without being interrupted by a hangup signal when we log off.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - "Daemon-izing" Your Script
- InhaltsvorschauSometimes you want a script to run as a daemon, i.e., in the background and never ending. To do this properly you need to be able to detach your script from its controlling tty, that is from the terminal session used to start the daemon. Simply putting an ampersand on the command isn't enough. If you start your daemon script on a remote system via an SSH (or similar) session, you'll notice that when you log out, the SSH session doesn't end and your window is hung until that script ends (which, being a daemon, it won't).Use the following to invoke your script, run it in the background, and still allow yourself to log out:
nohup mydaemonscript 0<&-1>/dev/null 2>&1 &
or:nohup mydaemonscript >>/var/log/myadmin.log 2>&1 <&- &
You need to close the controlling tty, which is connected in three ways to your (or any) job: standard input (STDIN), standard output (STDOUT), and standard error (STDERR). We can close STDOUT and STDERR by pointing them at another file— typically either a log file, so that you can retrieve their output at a later time, or at the file /dev/null to throw away all their output. We use the redirecting operator > to do this.But what about STDIN? The cleanest way to deal with STDIN is to close the file descriptor. The bash syntax to do that is like a redirect, but with a dash for the file-name (0<&-or <&-).We use the nohup command so that the script is run without being interrupted by a hangup signal when we log off.In the first example, we use the file descriptor numbers (i.e., 0, 1, 2) explicitly in all three redirections. They are optional in the case of STDIN and STDOUT, so in our second example we don't use them explicitly. We also put the input redirect at the end of the second command rather than at the beginning, since the order here is not important. (However, the order is important and the file descriptor numbers are necessary in redirecting STDERR.)- and for more on redirecting input and redirecting output
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Reusing Code with Includes and Sourcing
- InhaltsvorschauThere are a set of shell variable assignments that you would like to have common across a set of scripts that you are writing. You tried putting this configuration information in its own script. But when you run that script from within another script, the values don't stick; e.g., your configuration is running in another shell, and when that shell exits, so do your values. Is there some way to run that configuration script within the current shell?Use the bash shell's source command or POSIX single period (.) to read in the contents of that configuration file. The lines of that file will be processed as if encountered in the current script.Here's an example of some configuration data:
$ cat myprefs.cfg SCRATCH_DIR=/var/tmp IMG_FMT=png SND_FMT=ogg $
It is just a simple script consisting of three assignments. Here's another script, one that will use these values:# # use the user prefs # source $HOME/myprefs.cfg cd ${SCRATCH_DIR:-/tmp} echo You prefer $IMG_FMT image files echo You prefer $SND_FMT sound filesand so forth.The script that is going to use the configuration file uses the source command to read in the file. It can also use a dot (.) in place of the word source. A dot is easy and quick to type, but hard to notice in a script or screen shot:. $HOME/myprefs.cfg
You wouldn't be the first person to look right past the dot and think that the script was just being executed.bash also has a third syntax, one that comes from the input processor readline, a topic we will not get into here. We'll just say that an equivalent action can occur with this syntax:$include $HOME/myprefs.cfg
provided that the file is in your search path (or else specify an explicit path) and that the file has execute permissions and, of course, read permission, too. That dollar sign is not the command prompt, but part of the directive$include.Sourcing is both a powerful and a dangerous feature of bash scripting. It gives you a way to create a configuration file and then share that file among several scripts. With that mechanism, you can change your configuration by editing one file, not several scripts.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Configuration Files in a Script
- InhaltsvorschauYou want to use one or more external configuration files for one or more scripts.You could write a lot of code to parse some special configuration file format. Do yourself a favor and don't do that. Just make the config file a shell script and use the solution in , "Reusing Code with Includes and Sourcing."This is just a specific application of sourcing a file. However, it's worth noting that you may need to give a little thought as to how you can reduce all of your configuration needs to bash-legal syntax. In particular, you can make use of Boolean flags, and optional variables (see and , "Getting Input from Another Machine").
# In config file VERBOSE=0 # '' for off, 1 for on SSH_USER='jbagadonutz@' # Note trailing @, set to '' to use the current user # In script [ "$VERBOSE" ] || echo "Verbose msg from $) goes to STDERR" >&2 [...] ssh $SSH_USER$REMOTE_HOST [...]Of course, depending on the user to get the configuration file correct can be chancy, so instead of requiring the user to read the comment and add the trailing @, we could do it in the script:# If $SSH_USER is set and doesn't have a trailing @ add it: [ -n "$SSH_USER" -a "$SSH_USER" = "${SSH_USER%@}" ] && SSH_USER="$SSH_USER@"Or just use:ssh ${SSH_USER:+${SSH_USER}@}${REMOTE_HOST} [...]to make that same substitution right in place. The bash variable operator :+ will do the following: if$SSH_USERhas a value, it will return the value to the right of the :+ (in this case we specified the variable itself along with an extra @); otherwise, if unset or empty, it will return nothing.- , "Reusing Code with Includes and Sourcing"
- , "Getting Input from Another Machine"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Defining Functions
- InhaltsvorschauThere are several places in your shell script where you would like to give the user a usage message (a message describing the proper syntax for the command), but you don't want to keep repeating the code for the same echo statement. Isn't there a way to do this just once and have several references to it? If you could make the usage message its own script, then you could just invoke it anywhere in your original script—but that requires two scripts, not one. Besides, it seems odd to have the message for how to use one script be the output of a different script. Isn't there a better way to do this?You need a bash function. At the beginning of your script put something like this:
function usage () { printf "usage: %s [ -a | - b ] file1 ... filen\n" $0 > &2 }Then later in your script you can write code like this:if [ $# -lt 1] then usage fi
Functions may be defined in several ways([ function ]name () compound-command[ redirections ]). We could write a function definition any of these ways:function usage () { printf "usage: %s [ -a | - b ] file1 ... filen\n" $0 > &2 } function usage { printf "usage: %s [ -a | - b ] file1 ... filen\n" $0 > &2 } usage ( ) { printf "usage: %s [ -a | - b ] file1 ... filen\n" $0 > &2 } usage ( ) { printf "usage: %s [ -a | - b ] file1 ... filen\n" $0 > &2 }Either the reserved wordfunctionor the trailing () must be present. Iffunctionis used, the () are optional. We like using the word function because it is very clear and readable, and it is easy to grep for; e.g., grep '^function'script will list the functions in your script.This function definition should go at the front of your shell script, or at least some-where before you need to invoke it. The definition is, in a sense, just another bash statement. But once it has been executed, then the function is defined. If you invoke the function before it is defined you will get a "command not found" error. That's why we always put our function definitions first before any other commands in our script.Our function does very little; it is just a printfEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Functions: Parameters and Return Values
- InhaltsvorschauYou want to use a function and you need to get some values into the function. How do you pass in parameters? How do you get values back?You don't put parentheses around the arguments like you might expect from some programming languages. Put any parameters for a bash function right after the function's name, separated by whitespace, just like you were invoking any shell script or command. Don't forget to quote them if necessary!
# define the function: function max () { ... } # # call the function: # max 128 $SIM max $VAR $CNTYou have two ways to get values back from a function. You can assign values to variables inside the body of your function. Those variables will be global to the whole script unless they are explicitly declaredlocalwithin the function:# cookbook filename: func_max.1 # define the function: function max () { local HIDN if [ $1 -gt $2 ] then BIGR=$1 else BIGR=$2 fi HIDN=5 }For example:# call the function: max 128 $SIM # use the result: echo $BIGR
The other way is to use echo or printf to send output to standard output. Then you must invoke the function inside a $(), capturing the output and using the result, or it will be wasted on the screen:# cookbook filename: func_max.2 # define the function: function max () { if [ $1 -gt $2 ] then echo $1 else echo $2 fi }For example:# call the function: BIGR=$(max 128 $SIM) # use the result echo $BIGR
Putting parameters on the invocation of the function is just like calling any shell script. The parameters are just the other words on the command line.Within the function, the parameters are referred to as if they were command-line arguments by using$1,$2, etc. However,$0is left alone. It remains the name by which the entire script was invoked. On return from the function,$1,$2, etc. are back to referring to the parameters with which the script was invoked.Also of interest is the $FUNCNAME array. $FUNCNAME all by itself references the zeroth element of the array, which is the name of the currently executing function. In other words, $FUNCNAME is to a function asEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Trapping Interrupts
- InhaltsvorschauYou are writing a script that needs to be able to trap signals and respond accordingly.Use the trap utility to set signal handlers. First, use
trap -l (or kill -l)to list the signals you may trap. They vary from system to system:# NetBSD $ trap -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGEMT 8) SIGFPE 9) SIGKILL 10) SIGBUS 11) SIGSEGV 12) SIGSYS 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGURG 17) SIGSTOP 18) SIGTSTP 19) SIGCONT 20) SIGCHLD 21) SIGTTIN 22) SIGTTOU 23) SIGIO 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGINFO 30) SIGUSR1 31) SIGUSR2 32) SIGPWR # Linux $ trap -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 33) SIGRTMIN 34) SIGRTMIN+1 35) SIGRTMIN+2 36) SIGRTMIN+3 37) SIGRTMIN+4 38) SIGRTMIN+5 39) SIGRTMIN+6 40) SIGRTMIN+7 41) SIGRTMIN+8 42) SIGRTMIN+9 43) SIGRTMIN+10 44) SIGRTMIN+11 45) SIGRTMIN+12 46) SIGRTMIN+13 47) SIGRTMIN+14 48) SIGRTMIN+15 49) SIGRTMAX-15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAX
Next, set your trap(s) and signal handlers. Note that the exit status of your script will be 128+signal number if the command was terminated by signal signal number. Here is a simple case where we only care that we got a signal and don't care what it was. If our trap had beentrap '' ABRT EXIT HUP INT TERM QUIT, this script would be rather hard to kill because any of those signals would just be ignored.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Redefining Commands with alias
- InhaltsvorschauYou'd like to slightly alter the definition of a command, perhaps so that you always use a particular option on a command (e.g., always using
-aon the ls command or-ion the rm command).Use the alias feature of bash for interactive shells (only). The alias command is smart enough not to go into an endless loop when you say something like:alias ls='ls -a'
In fact, just typealiaswith no other arguments and you can see a list of aliases that are already defined for you in your bash session. Some installations may already have several available for you.The alias mechanism is a straightforward text substitution. It occurs very early in the command-line processing, so other substitutions will occur after the alias. For example, if you want to define the single letter "h" to be the command that lists your home directory, you can do it like this:alias h='ls $HOME'
or like this:alias h='ls ~'
The use of single quotes is significant in the first instance, meaning that the variable$HOMEwill not be evaluated when the definition of the alias is made. Only when you run the command will the (string) substitution be made, and only then will the$HOMEvariable be evaluated. That way if you change the definition of$HOMEthe alias will move with it, so to speak.If, instead, you used double quotes, then the substitution of the variable's value would be made right away and the alias would be defined with the value of$HOMEsubstituted. You can see this by typingaliaswith no arguments so that bash lists all the alias definitions. You would see something like this:... alias h='ls /home/youracct' ...
If you don't like what your alias does and want to get rid of it, just use unalias and the name of the alias that you no longer want. For example:unalias h
will remove the definition that we just made above. If you get really messed up, you can useunalias -ato remove all the alias definitions in your current shell session. But what if someone has created an alias for unalias? Simple, if you prefix it with a backslash, alias expansion is not performed. So use\unalias-aEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Avoiding Aliases, Functions
- InhaltsvorschauYou've written an alias or function to override a real command, and now you want to execute the real command.Use the bash shell's builtin command to ignore shell functions and aliases to run the actual built-in command.Use the command command to ignore shell functions and aliases to run the actual external command.If you only want to avoid alias expansion, but still allow function definitions to be considered, then prefix the command with \ to just prevent alias expansion.Use the type command (also with -a) to figure out what you've got.Here are some examples:
$ alias echo='echo ~~~' $ echo test ~~~ test $ \echo test test $ builtin echo test test $ type echo echo is aliased to `echo ~~~' $ unalias echo $ type echo echo is a shell builtin $ type -a echo echo is a shell builtin echo is /bin/echo $ echo test test
Here is a function definition that we will discuss:function cd () { if [[ $1 = "..." ]] then builtin cd ../.. else builtin cd $1 fi }The alias command is smart enough not to go into an endless loop when you say something likealias ls='ls-a'oralias echo='echo ~~~', so in our first example we need to do nothing special on the righthand side of our alias definition to refer to the actual echo command.When we have echo defined as an alias, then the type command will tell us not only that this is an alias, but will show us the alias definition. Similarly with function definitions, we would be shown the actual body of the function.type -asome_command will show us all of the places (aliases, built-ins, functions, and external) that contain some_command (as long as you are not also using-p).In our last example, the function overrides the definition of cd so that we can add a simple shortcut. We want our function to understand thatcd…means to go up two directories; i.e.,cd ../..(see , "Creating a Better cd Command"). All other arguments will be treated as normal. Our function simply looks for a match with … and substitutes the real meaning. But how, within (or without) the function, do you invoke the underlyingEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 11: Working with Dates and Times
- InhaltsvorschauWorking with dates and times should be simple, but it's not. Regardless of whether you're writing a shell script or a much larger program, time keeping is full of complexities: different formats for displaying the time and date, Daylight Saving Time, leap years, leap seconds, and all of that. For example, imagine that you have a list of contracts and the dates on which they were signed. You'd like to compute expiration dates for all of those contracts. It's not a trivial problem: does a leap year get in the way? Is it the sort of contract where daylight saving time is likely to be a problem? And how do you format the output so that it's unambiguous? Does 7/4/07 mean July 4, 2007, or does it mean April 7?Dates and times permeate every aspect of computing. Sooner or later you are going to have to deal with them: in system, application, or transaction logs; in data processing scripts; in user or administrative tasks; and more. This chapter will help you deal with them as simply and cleanly as possible. Computers are very good at keeping time accurately, particularly if they are using the Network Time Protocol (NTP) to keep themselves synced with national and international time standards. They're also great at understanding the variations in Daylight Saving Time from locale to locale. To work with time in a shell script, you need the Unix date command (or even better, the GNU version of the date command, which is standard on Linux). date is capable of displaying dates in different formats and even doing date arithmetic correctly.Note that gawk (the GNU version of awk), has the same strftime formatting as the GNU date command. We're not going to cover gawk usage here except for one trivial example. We recommend sticking with GNU date because it's much easier to use and it has the very useful
-dargument. But keep gawk in mind should you ever encounter a system that has gawk but not GNU date.You need to format dates or time for output.Use the date command with a strftime format specification. See "Date and Time String Formatting with strftime" in or the strftimeEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Formatting Dates for Display
- InhaltsvorschauYou need to format dates or time for output.Use the date command with a strftime format specification. See "Date and Time String Formatting with strftime" in or the strftime manpage for the list of format specifications supported.
# Setting environment variables can be helpful in scripts: $ STRICT_ISO_8601='%Y-%m-%dT%H:%M:%S%z' # http://greenwichmeantime.com/info/iso.htm $ ISO_8601='%Y-%m-%d %H:%M:%S %Z' # Almost ISO-8601, but more human-readable $ ISO_8601_1='%Y-%m-%d %T %Z' # %T is the same as %H:%M:%S $ DATEFILE='%Y%m%d%H%M%S' # Suitable for use in a file name $ date "+$ISO_8601" 2006-05-08 14:36:51 CDT gawk "BEGIN {print strftime(\"$ISO_8601\")}" 2006-12-07 04:38:54 EST # Same as previous $ISO_8601 $ date '+%Y-%m-%d %H:%M:%S %Z' 2006-05-08 14:36:51 CDT $ date -d '2005-11-06' "+$ISO_8601" 2005-11-06 00:00:00 CST $ date "+Program starting at: $ISO_8601" Program starting at: 2006-05-08 14:36:51 CDT $ printf "%b" "Program starting at: $(date '+$ISO_8601')\n" Program starting at: $ISO_8601 $ echo "I can rename a file like this: mv file.log file_$(date +$DATEFILE).log" I can rename a file like this: mv file.log file_20060508143724.logYou may be tempted to place the + in the environment variable to simplify the later command. On some systems the date command is more picky about the existence and placement of the + than on others. Our advice is to explicitly add it to the date command itself.Many more formatting options are available, see the date manpage or the Cstrftime()function(man 3 strftime)on your system for a full list.Unless otherwise specified, the time zone is assumed to be local time as defined by your system. The%zformat is a nonstandard extension used by the GNU date command; it may not work on your system.ISO 8601 is the recommended standard for displaying dates and times and should be used if at all possible. It offers a number of advantages over other display formats:- It is a recognized standard
- It is unambiguous
- It is easy to read while still being easy to parse programmatically (e.g., using
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Supplying a Default Date
- InhaltsvorschauYou want your script to provide a useful default date, and perhaps prompt the user to verify it.Using the GNU date command, assign the most likely date to a variable, then allow the user to change it:
#!/usr/bin/env bash # cookbook filename: default_date # Use Noon time to prevent a script running around midnight and a clock a # few seconds off from causing off by one day errors. START_DATE=$(date -d 'last week Monday 12:00:00' '+%Y-%m-%d') while [ 1 ]; do printf "%b" "The starting date is $START_DATE, is that correct? (Y/new date)" read answer # Anything other than ENTER, "Y" or "y" is validated as a new date # could use "[Yy]*" to allow the user to spell out "yes"... # validate the new date format as: CCYY-MM-DD case "$answer" in [Yy]) break ;; [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]) printf "%b" "Overriding $START_DATE with $answer\n" START_DATE="$answer" ;; *) printf "%b" "Invalid date, please try again...\n" ;; esac done END_DATE=$(date -d "$START_DATE +7 days" '+%Y-%m-%d') echo "START_DATE: $START_DATE" echo "END_DATE: $END_DATE"
Not all date commands support the-doption, but the GNU version does. Our advice is to obtain and use the GNU date command if at all possible.Leave out the user verification code if your script is running unattended or at a known time (e.g., from cron).See , "Formatting Dates for Display" for information about how to format the dates and times.We use code like this in scripts that generate SQL queries. The script runs at a given time and creates a SQL query for a specific date range to generate a report.- man date
- , "Formatting Dates for Display"
- , "Automating Date Ranges"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Automating Date Ranges
- InhaltsvorschauYou have one date (perhaps from , "Supplying a Default Date") and you would like to generate the other automatically.The GNU date command is very powerful and flexible, but the power of
-disn't documented well. Your system may document this under getdate (try the getdate manpage). Here are some examples:$ date '+%Y-%m-%d %H:%M:%S %z' 2005-11-05 01:03:00 -0500 $ date -d 'today' '+%Y-%m-%d %H:%M:%S %z' 2005-11-05 01:04:39 -0500 $ date -d 'yesterday' '+%Y-%m-%d %H:%M:%S %z' 2005-11-04 01:04:48 -0500 $ date -d 'tomorrow' '+%Y-%m-%d %H:%M:%S %z' 2005-11-06 01:04:55 -0500 $ date -d 'Monday' '+%Y-%m-%d %H:%M:%S %z' 2005-11-07 00:00:00 -0500 $ date -d 'this Monday' '+%Y-%m-%d %H:%M:%S %z' 2005-11-07 00:00:00 -0500 $ date -d 'last Monday' '+%Y-%m-%d %H:%M:%S %z' 2005-10-31 00:00:00 -0500 $ date -d 'next Monday' '+%Y-%m-%d %H:%M:%S %z' 2005-11-07 00:00:00 -0500 $ date -d 'last week' '+%Y-%m-%d %H:%M:%S %z' 2005-10-29 01:05:24 -0400 $ date -d 'next week' '+%Y-%m-%d %H:%M:%S %z' 2005-11-12 01:05:29 -0500 $ date -d '2 weeks' '+%Y-%m-%d %H:%M:%S %z' 2005-11-19 01:05:42 -0500 $ date -d '-2 weeks' '+%Y-%m-%d %H:%M:%S %z' 2005-10-22 01:05:47 -0400 $ date -d '2 weeks ago' '+%Y-%m-%d %H:%M:%S %z' 2005-10-22 01:06:00 -0400 $ date -d '+4 days' '+%Y-%m-%d %H:%M:%S %z' 2005-11-09 01:06:23 -0500 $ date -d '-6 days' '+%Y-%m-%d %H:%M:%S %z' 2005-10-30 01:06:30 -0400 $ date -d '2000-01-01 +12 days' '+%Y-%m-%d %H:%M:%S %z' 2000-01-13 00:00:00 -0500 $ date -d '3 months 1 day' '+%Y-%m-%d %H:%M:%S %z' 2006-02-06 01:03:00 -0500
The-doption allows you to specify a specific date instead of using now, but not all date commands support it. The GNU version supports it and our advice is to obtain and use that version if at all possible.Using-dcan be tricky. These arguments work as expected:$ date '+%a %Y-%m-%d' Sat 2005-11-05 $ date -d 'today' '+%a %Y-%m-%d' Sat 2005-11-05 $ date -d 'Saturday' '+%a %Y-%m-%d' Sat 2005-11-05 $ date -d 'last Saturday' '+%a %Y-%m-%d' Sat 2005-10-29 $ date -d 'this Saturday' '+%a %Y-%m-%d' Sat 2005-11-05
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Converting Dates and Times to Epoch Seconds
- InhaltsvorschauYou want to convert a date and time to Epoch seconds to make it easier to do date and time arithmetic.Use the GNU date command with the nonstandard
-doption and a standard%sformat:# "Now" is easy $ date '+%s' 1131172934 # Some other time needs the non-standard -d $ date -d '2005-11-05 12:00:00 +0000' '+%s' 1131192000
If you do not have the GNU date command available, this is a harder problem to solve. Our advice is to obtain and use the GNU date command if at all possible. If that is not possible you might be able to use Perl. Here are three ways to print the time right now in Epoch seconds:$ perl -e 'print time, qq(\n);' 1154158997 # Same as above $ perl -e 'use Time::Local; print timelocal(localtime()) . qq(\n);' 1154158997 $ perl -e 'use POSIX qw(strftime); print strftime("%s", localtime()) . qq(\n);' 1154159097Using Perl to convert a specific day and time instead of right now is even harder due to Perl's date/time data structure. Years start at 1900 and months (but not days) start at zero instead of one. The format of the command is:timelocal(sec, min, hour, day, month-1, year-1900). So to convert2005-11-05 06:59:49to Epoch seconds:# The given time is in local time $ perl -e 'use Time::Local; print timelocal("49", "59", "06", "05", "10", "105") . .qq(\n);' 1131191989 # The given time is in UTC time $ perl -e 'use Time::Local; print timegm("49", "59", "06", "05", "10", "105") . qq(\n);' 1131173989
- man date
- , "Converting Epoch Seconds to Dates and Times"
- "Date and Time String Formatting with strftime" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Converting Epoch Seconds to Dates and Times
- InhaltsvorschauYou need to convert Epoch seconds to a human-readable date and time.Use the GNU date command with your desired format from , "Formatting Dates for Display":
EPOCH='1131173989' $ date -d "1970-01-01 UTC $EPOCH seconds" +"%Y-%m-%d %T %z" 2005-11-05 01:59:49 -0500 $ date --utc --date "1970-01-01 $EPOCH seconds" +"%Y-%m-%d %T %z" 2005-11-05 06:59:49 +0000
Since Epoch seconds are simply the number of seconds since the Epoch (which is Midnight on January 1, 1970, also known as 1970-01-01T00:00:00), this command starts at the Epoch, adds the Epoch seconds, and displays the date and time as you wish.If you don't have GNU date on your system you can try one of these Perl one-liners:EPOCH='1131173989' $ perl -e "print scalar(gmtime($EPOCH)), qq(\n);" # UTC Sat Nov 5 06:59:49 2005 $ perl -e "print scalar(localtime($EPOCH)), qq(\n);" # Your local time Sat Nov 5 01:59:49 2005 $ perl -e "use POSIX qw(strftime); print strftime('%Y-%m-%d %H:%M:%S', localtime($EPOCH)), qq(\n);" 2005-11-05 01:59:49- man date
- , "Formatting Dates for Display"
- , "Converting Dates and Times to Epoch Seconds"
- "Date and Time String Formatting with strftime" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Yesterday or Tomorrow with Perl
- InhaltsvorschauYou need to get yesterday or tomorrow's date, and you have Perl but not GNU date on your system.Use this Perl one-liner, adjusting the number of seconds added to or subtracted from
time:# Yesterday at this same time (note subtraction) $ perl -e "use POSIX qw(strftime); print strftime('%Y-%m-%d', localtime(time - 86400)), qq(\n);" # Tomorrow at this same time (note addition) $ perl -e "use POSIX qw(strftime); print strftime('%Y-%m-%d', localtime(time + 86400)), qq(\n);"This is really just a specific application of the recipes above, but is so common that it's worth talking about by itself. See , "Figuring Out Date and Time Arithmetic" for a handy table of values that may be of use.- , "Supplying a Default Date"
- , "Automating Date Ranges"
- , "Converting Dates and Times to Epoch Seconds"
- , "Converting Epoch Seconds to Dates and Times"
- , "Figuring Out Date and Time Arithmetic"
- "Date and Time String Formatting with strftime" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Figuring Out Date and Time Arithmetic
- InhaltsvorschauYou need to do some kind of arithmetic with dates and times.If you can't get the answer you need using the date command (see , "Automating Date Ranges"), convert your existing dates and times to Epoch seconds using , "Converting Dates and Times to Epoch Seconds," perform your calculations, then convert the resulting Epoch seconds back to your desired format using , "Converting Epoch Seconds to Dates and Times."If you don't have GNU date, you may find the shell functions presented in "Shell Corner: Date-Related Shell Functions" in the September 2005 issue of Unix Review to be very useful. See , "Automating Date Ranges."For example, suppose you have log data from a machine where the time was badly off. Everyone should already be using the Network Time Protocol (NTP) so this doesn't happen, but just suppose:
CORRECTION='172800' # 2 days worth of seconds # Code to extract the date portion from the data # into $bad_date go here # Suppose it's this: bad_date='Jan 2 05:13:05' # syslog formated date # Convert to Epoch using GNU date bad_epoch=$(date -d "$bad_date" '+%s') # Apply correction good_epoch=$(( bad_epoch + $CORRECTION )) # Make corrected date human-readable good_date=$(date -d "1970-01-01 UTC $good_epoch seconds") # GNU Date good_date_iso=$(date -d "1970-01-01 UTC $good_epoch seconds" +'%Y-%m-%d %T') # GNU Date echo "bad_date: $bad_date" echo "bad_epoch: $bad_epoch" echo "Correction: +$CORRECTION" echo "good_epoch: $good_epoch" echo "good_date: $good_date" echo "good_date_iso: $good_date_iso" # Code to insert the $good_date back into the data goes here
Watch out for years! Some Unix commands like ls and syslog try to be easy to read and omit the year under certain conditions. You may need to take that into account when calculating your correction factor. If you have data from a large range of dates or from different time zones, you will have to find some way to break it into separate files and process them individually.Dealing with any kind of date arithmetic is much easier using Epoch seconds than any other format of which we are aware. You don't have to worry about hours, days, weeks, or years, you just do some simple addition or subtraction and you're all set. Using Epoch seconds also avoids all the convoluted rules about leap years and seconds, and if you standardize on one time zone (usually UTC, which used to be called GMT) you can even avoid time zones.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Handling Time Zones, Daylight Saving Time, and Leap Years
- InhaltsvorschauYou need to account for time zones, Daylight Saving Time, and leap years or seconds.Don't. This is a lot trickier than it sounds. Leave it to code that's already been in use and debugged for years, and just use a tool that can handle your needs. Odds are high that one of the other recipes in this chapter has covered what you need, probably using GNU date. If not, there is almost certainly another tool out there that can do the job. For example, there are a number of excellent Perl modules that deal with dates and times.Really, we aren't kidding. This is a real nightmare to get right. Save yourself a lot of agony and just use a tool.
- , "Formatting Dates for Display"
- , "Automating Date Ranges"
- , "Converting Dates and Times to Epoch Seconds"
- , "Converting Epoch Seconds to Dates and Times"
- , "Figuring Out Date and Time Arithmetic"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using date and cron to Run a Script on the Nth Day
- InhaltsvorschauYou need to run a script on the Nth weekday of the month (e.g., the second Wednesday), and most crons will not allow that.Use a bit of shell code in the command to be run. In your Linux Vixie Cron crontab adapt one of the following lines. If you are using another cron program, you may need to convert the day of the week names to numbers according to the schedule your cron uses (0–6 or 1–7) and use
+%w(day of week as number) in place of+%a(locale's abbreviated weekday name):# Vixie Cron # Min Hour DoM Mnth DoW Program # 0-59 0-23 1-31 1-12 0-7 # Run the first Wednesday @ 23:00 00 23 1-7 * Wed [ "$(date '+%a')" == "Wed" ] && /path/to/command args to command # Run the second Thursday @ 23:00 00 23 8-14 * Thu [ "$(date '+%a')" == "Thu" ] && /path/to/command # Run the third Friday @ 23:00 00 23 15-21 * Fri [ "$(date '+%a')" == "Fri" ] && /path/to/command # Run the fourth Saturday @ 23:00 00 23 22-27 * Sat [ "$(date '+%a')" == "Sat" ] && /path/to/command # Run the fifth Sunday @ 23:00 00 23 28-31 * Sun [ "$(date '+%a')" == "Sun" ] && /path/to/command
Note that any given day of the week doesn't always happen five times during one month, so be sure you really know what you are asking for if you schedule something for the fifth week of the month.Most versions of cron (including Linux's Vixie Cron) do not allow you to schedule a job on the Nth day of the month. To get around that, we schedule the job to run during the range of days when the Nth day we need occurs, then check to see if it is the correct day on which to run. The "second Wednesday of the month" must occur somewhere in the range of the 8th to 14th day of the month. So we simply run every day and see if it's Wednesday. If so, we execute our command.shows the ranges noted above.Table 11-2: Day ranges for each week of a month WeekDay rangeFirst1 to 7Second8 to 14Third15 to 21Fourth22 to 27Fifth (see previous warning note)28 to 31We know this almost seems too simplistic; check a calendar if you don't believe us:$ cal 10 2006 October 2006 S M Tu W Th F S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 12: End-User Tasks As Shell Scripts
- InhaltsvorschauYou have seen a lot of smaller scripts and syntax up to now. Our examples have, of necessity, been small in scale and scope. Now we would like to show you a few larger (though not large) examples. They are meant to give you useful, real world examples of actual uses of shell scripts beyond just system administration tasks. We hope you find them useful or usable. More than that, we hope you learn something about bash by reading through them and maybe trying them yourself or even tweaking them for your own use.To print a line of dashes with a simple command might sound easy—and it is. But as soon as you think you've got a simple script, it begins to grow. What about varying the length of the line of dashes? What about changing the character from a dash to a user-supplied character? Do you see how easily feature creep occurs? Can we write a simple script that takes those extensions into account without getting too complex?Consider this script:
1 #!/usr/bin/env bash 2 # cookbook filename: dash 3 # dash - print a line of dashes 4 # options: # how many (default 72) 5 # -c X use char X instead of dashes 6 # 7 function usagexit ( ) 8 { 9 printf "usage: %s [-c X] [#]\n" $(basename $0) 10 exit 2 11 } >&2 12 LEN=72 13 CHAR='-' 14 while (( $# > 0 )) 15 do 16 case $1 in 17 [0-9]*) LEN=$1;; 18 -c) shift 19 CHAR=$1;; 20 *) usagexit;; 21 esac 22 shift 23 done 24 if (( LEN > 4096 )) 25 then 26 echo "too large" >&2 27 exit 3 28 fi 29 # build the string to the exact length 30 DASHES="" 31 for ((i=0; i<LEN; i++)) 32 do 33 DASHES="${DASHES}${CHAR}" 34 done 35 printf "%s\n" "$DASHES"The basic task is accomplished by building a string of the required number of dashes (or an alternate character) and then printing that string to standard output (STD-OUT). That takes only the six lines from 30–35. Lines 12 and 13 set the default values. All the other lines are spent on argument parsing, error checking, user messages, and comments.You will find that it's pretty typical for a robust, end-user script. Less than 20 percent of the code does more than 80 percent of the work. But that 80 percent of the code is what makes it usable and "friendly" for your users.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Starting Simple by Printing Dashes
- InhaltsvorschauTo print a line of dashes with a simple command might sound easy—and it is. But as soon as you think you've got a simple script, it begins to grow. What about varying the length of the line of dashes? What about changing the character from a dash to a user-supplied character? Do you see how easily feature creep occurs? Can we write a simple script that takes those extensions into account without getting too complex?Consider this script:
1 #!/usr/bin/env bash 2 # cookbook filename: dash 3 # dash - print a line of dashes 4 # options: # how many (default 72) 5 # -c X use char X instead of dashes 6 # 7 function usagexit ( ) 8 { 9 printf "usage: %s [-c X] [#]\n" $(basename $0) 10 exit 2 11 } >&2 12 LEN=72 13 CHAR='-' 14 while (( $# > 0 )) 15 do 16 case $1 in 17 [0-9]*) LEN=$1;; 18 -c) shift 19 CHAR=$1;; 20 *) usagexit;; 21 esac 22 shift 23 done 24 if (( LEN > 4096 )) 25 then 26 echo "too large" >&2 27 exit 3 28 fi 29 # build the string to the exact length 30 DASHES="" 31 for ((i=0; i<LEN; i++)) 32 do 33 DASHES="${DASHES}${CHAR}" 34 done 35 printf "%s\n" "$DASHES"The basic task is accomplished by building a string of the required number of dashes (or an alternate character) and then printing that string to standard output (STD-OUT). That takes only the six lines from 30–35. Lines 12 and 13 set the default values. All the other lines are spent on argument parsing, error checking, user messages, and comments.You will find that it's pretty typical for a robust, end-user script. Less than 20 percent of the code does more than 80 percent of the work. But that 80 percent of the code is what makes it usable and "friendly" for your users.In line 9 we usebasenameto trim off any leading pathname characters when displaying this script's name. That way no matter how the user invokes the script (for example, ./dashes, /home/username/bin/dashes, or even ../../over/there/dashes), it will still be referred to as just dashes in the usage message.The argument parsing is done while there are some arguments to parse (line 14). As arguments are handled, eachEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Viewing Photos in an Album
- InhaltsvorschauYou have a directory full of images like the ones you just downloaded from your digital camera. You want a quick and easy way to view them all, so that you can pick out the good ones.Write a shell script that will generate a set of html pages so that you can view your photos with a browser. Call it mkalbum and put it somewhere like your ~/bin directory.On the command line, cd into the directory where you want your album created (typically where your photos are located). Then run some command that will generate the list of photos that you want included in this album (e.g.,
ls *.jpg, but see also , "Finding Files Irrespective of Case"), and pipe this output into the mkalbum shell script, which we will explain later. You need to put the name of the album (i.e., the name of a directory that will be created by the script) on the command line as the only argument to the shell script. It might look something like this:$ ls *.jpg | mkalbum rugbymatch
shows a sample of the generated web page.
Figure 12-1: Sample mkalbum web pageThe large title is the photo (i.e., the filename); there are hyperlinks to other pages for first, last, next, and previous.The fpllowing is the shell script (mkalbum) that will generate a set of html pages, one page per image (the line numbers are not part of the script, but are put here to make it easier to discuss):1 #!/usr/bin/env bash 2 # cookbook filename: mkalbum 3 # mkalbum - make an html "album" of a pile of photo files. 4 # ver. 0.2 5 # 6 # An album is a directory of html pages. 7 # It will be created in the current directory. 8 # 9 # An album page is the html to display one photo, with 10 # a title that is the filename of the photo, along with 11 # hyperlinks to the first, previous, next, and last photos. 12 # 13 # ERROUT 14 ERROUT( ) 15 { 16 printf "%b" "$@" 17 } >&2 18 19 # 20 # USAGE 21 USAGE( ) 22 { 23 ERROUT "usage: %s <newdir>\n" $(basename $0) 24 } 25 26 # EMIT(thisph, startph, prevph, nextph, lastph) 27 EMIT( ) 28 { 29 THISPH="../$1" 30 STRTPH="${2%.*}.html" 31 PREVPH="${3%.*}.html" 32 NEXTPH="${4%.*}.html" 33 LASTPH="${5%.*}.html" 34 if [ -z "$3" ] 35 then 36 PREVLINE='<TD> Prev </TD>' 37 else 38 PREVLINE='<TD> <A HREF="'$PREVPH'"> Prev </A> </TD>' 39 fi 40 if [ -z "$4" ] 41 then 42 NEXTLINE='<TD> Next </TD>' 43 else 44 NEXTLINE='<TD> <A HREF="'$NEXTPH'"> Next </A> </TD>' 45 fi 46 cat <<EOF 47 <HTML> 48 <HEAD><TITLE>$THISPH</TITLE></HEAD> 49 <BODY> 50 <H2>$THISPH</H2> 51 <TABLE WIDTH="25%"> 52 <TR> 53 <TD> <A HREF="$STRTPH"> First </A> </TD> 54 $PREVLINE 55 $NEXTLINE 56 <TD> <A HREF="$LASTPH"> Last </A> </TD> 57 </TR> 58 </TABLE> 59 <IMG SRC="$THISPH" alt="$THISPH" 60 BORDER="1" VSPACE="4" HSPACE="4" 61 WIDTH="800" HEIGHT="600"/> 62 </BODY> 63 </HTML> 64 EOF 65 } 66 67 if (( $# != 1 )) 68 then 69 USAGE 70 exit -1 71 fi 72 ALBUM="$1" 73 if [ -d "${ALBUM}" ] 74 then 75 ERROUT "Directory [%s] already exists.\n" ${ALBUM} 76 USAGE 77 exit -2 78 else 79 mkdir "$ALBUM" 80 fi 81 cd "$ALBUM" 82 83 PREV="" 84 FIRST="" 85 LAST="last" 86 87 while read PHOTO 88 do 89 # prime the pump 90 if [ -z "${CURRENT}" ] 91 then 92 CURRENT="$PHOTO" 93 FIRST="$PHOTO" 94 continue 95 fi 96 97 PHILE=$(basename "${CURRENT}") 98 EMIT "$CURRENT" "$FIRST" "$PREV" "$PHOTO" "$LAST" > "${PHILE%.*}.html" 99 100 1 # set up for next iteration 101 PREV="$CURRENT" 102 CURRENT="$PHOTO" 103 104 done 105 106 PHILE=$(basename ${CURRENT}) 107 EMIT "$CURRENT" "$FIRST" "$PREV" "" "$LAST" > "${PHILE%.*}.html" 108 109 # make the symlink for "last" 110 ln -s "${PHILE%.*}.html" ./last.html 111 112 # make a link for index.html 113 ln -s "${FIRST%.*}.html" ./index.html 114Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Loading Your MP3 Player
- InhaltsvorschauYou have a collection of MP3 files that you would like to put in your MP3 player. But you have more music than can fit on your MP3 player. How can you load your player with music without having to baby-sit it by dragging and dropping files until it is full?Use a shell script to keep track of the available space as it copies files onto the MP3 player, quitting when it is full.
1 #!/usr/bin/env bash 2 # cookbook filename: load_mp3 3 # Fill up my mp3 player with as many songs as will fit. 4 # N.B.: This assumes that the mp3 player is mounted on /media/mp3 5 # 6 7 # 8 # determine the size of a file 9 # 10 function FILESIZE ( ) 11 { 12 FN=${1:-/dev/null} 13 if [[ -e $FN ]] 14 then 15 # FZ=$(ls -s $FN | cut -d ' ' -f 1) 16 set -- $(ls -s "$FN") 17 FZ=$1 18 fi 19 } 20 21 # 22 # compute the freespace on the mp3 player 23 # 24 function FREESPACE 25 { 26 # FREE=$(df /media/mp3 | awk '/^\/dev/ {print $4}') 27 set -- $(df /media/mp3 | grep '^/dev/') 28 FREE=$4 29 } 30 31 # subtract the (given) filesize from the (global) freespace 32 function REDUCE ( ) 33 (( FREE-=${1:-0})) 34 35 # 36 # main: 37 # 38 let SUM=0 39 let COUNT=0 40 export FZ 41 export FREE 42 FREESPACE 43 find . -name '*.mp3' -print | \ 44 (while read PATHNM 45 do 46 FILESIZE "$PATHNM" 47 if ((FZ <= FREE)) 48 then 49 echo loading $PATHNM 50 cp "$PATHNM" /media/mp3 51 if (( $? == 0 )) 52 then 53 let SUM+=FZ 54 let COUNT++ 55 REDUCE $FZ 56 else 57 echo "bad copy of $PATHNM to /media/mp3" 58 rm -f /media/mp3/$(basename "$PATHNM") 59 # recompute because we don't know how far it got 60 FREESPACE 61 fi 62 # any reason to go on? 63 if (( FREE <= 0 )) 64 then 65 break 66 fi 67 else 68 echo skipping $PATHNM 69 fi 70 done 71 printf "loaded %d songs (%d blocks)" $COUNT $SUM 72 printf " onto /media/mp3 (%d blocks free)\n" $FREE 73 ) 74 # end of scriptInvoke this script and it will copy any MP3 file that it finds from the current directory on down (toward the leaf nodes of the tree) onto an MP3 player (or other device) mounted onEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Burning a CD
- InhaltsvorschauYou have a directory full of files on your Linux system that you would like to burn to a CD. Do you need an expensive CD burning program, or can you do it with the shell and some open source programs?You can do it with two open source programs called mkisofs and cdrecord, and a bash script to help you keep all the options straight.Start by putting all the files that you want to copy to CD into a directory structure. The script will take that directory, make an ISO filesystem image from those files, then burn the ISO image. All it takes is a bunch of disk space and a bit of time—but you can get up and wander while the bash script runs.This script may not work on your system. We include it here as an example of shell scripting, not as a workable CD recording and backup mechanism.
1 #!/usr/bin/env bash 2 # cookbook filename: cdscript 3 # cdscript - prep and burn a CD from a dir. 4 # 5 # usage: cdscript dir [ cddev ] 6 # 7 if [[ $# < 1 || $# > 2 ]] 8 then 9 echo 'usage: cdscript dir [ cddev ]' 10 exit 2 11 fi 12 13 # set the defaults 14 RCDIR=$1 15 # your device might be "ATAPI:0,0,0" or other digits 16 CDDEV=${2:-"ATAPI:0,0,0"} 17 ISOIMAGE=/tmp/cd$$.iso 18 19 echo "building ISO image..." 20 # 21 # make the ISO fs image 22 # 23 mkisofs $ISOPTS -A "$(cat ~/.cdAnnotation)" \ 24 -p "$(hostname)" -V "$(basename $SRCDIR)" \ 25 -r -o "$ISOIMAGE" $SRCDIR 26 STATUS=$? 27 if [ $STATUS -ne 0 ] 28 then 29 echo "Error. ISO image failed." 30 echo "Investigate then remove $ISOIMAGE" 31 exit $STATUS 32 fi 33 34 echo "ISO image built; burning to cd..." 35 exit 36 37 # burn the CD 38 SPD=8 39 OPTS="-eject -v fs=64M driveropts=burnproof" 40 cdrecord $OPTS -speed=$SPD dev=${CDDEV} $ISOImage 41 STATUS=$? 42 if [ $STATUS -ne 0 ] 43 then 44 echo "Error. CD Burn failed." 45 echo "Investigate then remove $ISOIMAGE" 46 exit $STATUS 47 fi 48 49 rm -f $ISOIMAGE 50 echo "Done."Here is a quick look at some of the odder constructs in this script.At line 17:17 ISOIMAGE=/tmp/cd$$.iso
we construct a temporary filename by using the $$ variable, which gives us our process number. As long as this script is running, it will be the one and only process of that number, so it gives us a name that is unique among all other running processes. (See , "Using Secure Temporary Files" for a better way.)Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Comparing Two Documents
- InhaltsvorschauIt is easy to compare two text files (see , "Using diff and patch"). But what about documents produced by your suite of office applications? They are not stored as text, so how can you compare them? If you have two versions of the same document, and you need to know what the content changes are (if any) between the two versions, is there anything you can do besides printing them out and comparing page after page?First, use an office suite that will let you save your documents in Open Document Format (ODF). This is the case for packages like OpenOffice.org while other commercial packages have promised to add support soon. Once you have your files in ODF, you can use a shell script to compare just the content of the files. We stress the word content here because the formatting differences are another issue, and it is (usually) the content that is the most important determinant of which version is new or more important to the end user.Here is a bash script that can be used to compare two OpenOffice.org files, which are saved in ODF (but use the conventional suffix odt to indicate a text-oriented document, as opposed to a spreadsheet or a presentation file).
1 #!/usr/bin/env bash 2 # cookbook filename: oodiff 3 # oodiff -- diff the CONTENTS of two OpenOffice.org files 4 # works only on .odt files 5 # 6 function usagexit ( ) 7 { 8 echo "usage: $0 file1 file2" 9 echo "where both files must be .odt files" 10 exit $1 11 } >&2 12 13 # assure two readable arg filenames which end in .odt 14 if (( $# != 2 )) 15 then 16 usagexit 1 17 fi 18 if [[ $1 != *.odt || $2 != *.odt ]] 19 then 20 usagexit 2 21 fi 22 if [[ ! -r $1 || ! -r $2 ]] 23 then 24 usagexit 3 25 fi 26 27 BAS1=$(basename "$1" .odt) 28 BAS2=$(basename "$2" .odt) 29 30 # unzip them someplace private 31 PRIV1="/tmp/${BAS1}.$$_1" 32 PRIV2="/tmp/${BAS2}.$$_2" 33 34 # make absolute 35 HERE=$(pwd) 36 if [[ ${1:0:1} == '/' ]] 37 then 38 FULL1="${1}" 39 else 40 FULL1="${HERE}/${1}" 41 fi 42 43 # make absolute 44 if [[ ${2:0:1} == '/' ]] 45 then 46 FULL2="${2}" 47 else 48 FULL2="${HERE}/${2}" 49 fi 50 51 # mkdir scratch areas and check for failure 52 # N.B. must have whitespace around the { and } and 53 # must have the trailing ; in the {} lists 54 mkdir "$PRIV1" || { echo Unable to mkdir $PRIV1 ; exit 4; } 55 mkdir "$PRIV2" || { echo Unable to mkdir $PRIV2 ; exit 5; } 56 57 cd "$PRIV1" 58 unzip -q "$FULL1" 59 sed -e 's/>/>\ 60 /g' -e 's/</\ 61 </g' content.xml > contentwnl.xml 62 63 cd "$PRIV2" 64 unzip -q "$FULL2" 65 sed -e 's/>/>\ 66 /g' -e 's/</\ 67 </g' content.xml > contentwnl.xml 68 69 cd $HERE 70 71 diff "${PRIV1}/contentwnl.xml" "${PRIV2}/contentwnl.xml" 72 73 rm -rf $PRIV1 $PRIV2Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 13: Parsing and Similar Tasks
- InhaltsvorschauThis is a chapter of tasks that programmers might recognize. It's not necessarily more advanced than other bash script recipes in the book, but if you are not a programmer, these tasks might seem obscure or irrelevant to your use of bash. We won't do much explaining of the reasons why you'd find yourself in these situations (as a programmer, you'll recognize some if not all of them). Even if you don't recognize the situation, you should read them for what you can learn about bash.Some of the recipes in this chapter include the parsing of command-line arguments. Recall that the typical way to specify options on a shell script is to have a leading minus sign and a single letter. For example, an option for your script to give fewer messages might use
-qas a flag to mean quiet mode. Sometimes an option might take an argument. For example, a user option where you need to specify a username might use-ufollowed by the username. This distinction will be made clear in this chapter's first recipe.You want to have some options on your shell script, some flags that you can use to alter its behavior. You could do the parsing directly, using ${#} to tell you how many arguments have been supplied, and testing${1:0:1}to test the first character of the first argument to see if it is a minus sign. You would need someif/thenorcaselogic to identify which option it is and whether it takes an argument. What if the user doesn't supply a required argument? What if the user calls your script with two options combined (e.g.,-ab)? Will you also parse for that? The need to parse options for a shell script is a common situation. Lots of scripts have options. Isn't there a more standard way to do this?Use bash's built-in getopts command to help parse options.Here is an example, based largely on the example in the manpage for getopts:#!/usr/bin/env bash # cookbook filename: getopts_example # # using getopts # aflag= bflag= while getopts 'ab:' OPTION do case $OPTION in a) aflag=1 ;; b) bflag=1 bval="$OPTARG" ;; ?) printf "Usage: %s: [-a] [-b value] args\n" $(basename $0) >&2 exit 2 ;; esac done shift $(($OPTIND - 1)) if [ "$aflag" ] then printf "Option -a specified\n" fi if [ "$bflag" ] then printf 'Option -b "%s" specified\n' "$bval" fi printf "Remaining arguments are: %s\n" "$*"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Arguments for Your Shell Script
- InhaltsvorschauYou want to have some options on your shell script, some flags that you can use to alter its behavior. You could do the parsing directly, using ${#} to tell you how many arguments have been supplied, and testing
${1:0:1}to test the first character of the first argument to see if it is a minus sign. You would need someif/thenorcaselogic to identify which option it is and whether it takes an argument. What if the user doesn't supply a required argument? What if the user calls your script with two options combined (e.g.,-ab)? Will you also parse for that? The need to parse options for a shell script is a common situation. Lots of scripts have options. Isn't there a more standard way to do this?Use bash's built-in getopts command to help parse options.Here is an example, based largely on the example in the manpage for getopts:#!/usr/bin/env bash # cookbook filename: getopts_example # # using getopts # aflag= bflag= while getopts 'ab:' OPTION do case $OPTION in a) aflag=1 ;; b) bflag=1 bval="$OPTARG" ;; ?) printf "Usage: %s: [-a] [-b value] args\n" $(basename $0) >&2 exit 2 ;; esac done shift $(($OPTIND - 1)) if [ "$aflag" ] then printf "Option -a specified\n" fi if [ "$bflag" ] then printf 'Option -b "%s" specified\n' "$bval" fi printf "Remaining arguments are: %s\n" "$*"
There are two kinds of options supported here. The first and simpler kind is an option that stands alone. It typically represents a flag to modify a command's behavior. An example of this sort of option is the-loption on the ls command. The second kind of option requires an argument. An example of this is the mysql command's-uoption, which requires that a username be supplied, as inmysql -u sysadmin. Let's look at how getopts supports the parsing of both kinds.The use of getopts has two arguments.getopts 'ab:' OPTION
The first is a list of option letters. The second is the name of a shell variable. In our example, we are defining-aand-bas the only two valid options, so the first argument ingetoptshas just those two letters…and a colon. What does the colon signify? It indicates thatEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Arguments with Your Own Error Messages
- InhaltsvorschauYou are using getopts to parse your options for your shell script. But you don't like the error messages that it writes when it encounters bad input. Can you still use getopts but write your own error handling?If you just want getopts to be quiet and not report any errors at all, just assign
$OPTERR=0before you begin parsing. But if you want getopts to give you more information without the error messages, then just begin the option list with a colon. (Thev---in the comments below is meant to be an arrow pointing to a particular place in the line below it, to show that special colon.)#!/usr/bin/env bash # cookbook filename: getopts_custom # # using getopts - with custom error messages # aflag= bflag= # since we don't want getopts to generate error # messages, but want this script to issue its # own messages, we will put, in the option list, a # leading ':' v---here to silence getopts. while getopts :ab: FOUND do case $FOUND in a) aflag=1 ;; b) bflag=1 bval="$OPTARG" ;; \:) printf "argument missing from -%s option\n" $OPTARG printf "Usage: %s: [-a] [-b value] args\n" $(basename $0) exit 2 ;; \?) printf "unknown option: -%s\n" $OPTARG printf "Usage: %s: [-a] [-b value] args\n" $(basename $0) exit 2 ;; esac >&2 done shift $(($OPTIND - 1)) if [ "$aflag" ] then printf "Option -a specified\n" fi if [ "$bflag" ] then printf 'Option -b "%s" specified\n' "$bval" fi printf "Remaining arguments are: %s\n" "$*"The script is very much the same as the recipe , "Parsing Arguments for Your Shell Script." See that discussion for more background. One difference here is that getopts may now return a colon. It does so when an option is missing (e.g., you invoke the script with-bbut without an argument for it). In that case, it puts the option letter into$OPTARGso that you know what option it was that was missing its argument.Similarly, if an unsupported option is given (e.g., if you tried-dwhen invoking our example) getopts returns a question mark as the value forEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Some HTML
- InhaltsvorschauYou want to pull the strings out of some HTML. For example, you'd like to get at the
href="urlstringstuff" type strings from the<a>tags within a chunk of HTML.For a quick and easy shell parse of HTML, provided it doesn't have to be foolproof, you might want to try something like this:cat $1 | sed -e 's/>/>\ /g' | grep '<a' | while IFS='"' read a b c ; do echo $b; done
Parsing HTML from bash is pretty tricky, mostly because bash tends to be very line oriented whereas HTML was designed to treat newlines like whitespace. So it's not uncommon to see tags split across two or more lines as in:<a href="blah...blah...blah other stuff >
There are also two ways to write<a>tags, one with a separate ending</a>tag, and one without, where instead the singular<a>tag itself ends with a/>. So, with multiple tags on a line and the last tag split across lines, it's a bit messy to parse, and our simple bash technique for this is often not foolproof.Here are the steps involved in our solution. First, break the multiple tags on one line into at most one line per tag:cat file | sed -e 's/>/>\ /g'
Yes, that's a newline right after the backslash so that it substitutes each end-of-tag character (i.e., the>) with that same character and then a newline. That will put tags on separate lines with maybe a few extra blank lines. The trailinggtellssedto do the search and replace globally, i.e., multiple times on a line if need be.Then you can pipe that output into grep to grab just the<atag lines or maybe just lines with double quotes:cat file | sed -e 's/>/>\ /g' | grep '<a'
or:cat file | sed -e 's/>/>\ /g' | grep '".*"'
(that's g r e p ' ". * " '). The single quotes tell the shell to take the inner characters literally and not do any shell expansion on them; the rest is a regular expression to match a double quote followed by any character (.) any number of times (*) followed by another double quote. (This won't work if the string itself is split across lines.)To parse out the contents of what's inside the double quotes, one trick is to use the shell's Internal Field SeparatorEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Output into an Array
- InhaltsvorschauYou want the output of some program or script to be put into an array.
#!/usr/bin/env bash # cookbook filename: parseViaArray # # find the file size # use an array to parse the ls -l output into words LSL=$(ls -ld $1) declare -a MYRA MYRA=($LSL) echo the file $1 is ${MYRA[4]} bytes.In our example, we take the output from thels -lcommand and parse the words by putting them into an array. Then we can just refer to each array element to get at each word. The typical output from thels -lcommand looks like this (yours may vary due to locale):-rw-r--r--1 albing users 113 2006-10-10 23:33 mystuff.txt
Arrays are easy to initialize if you know the values as you write the script. The format is simple. We begin by declaring the variable to be an array, and then we assign it values:declare -a MYRA MYRA=(first second third home)
The same can be done by using a variable inside those parentheses. Just be sure not to use quotes around the variable. WritingMYRA=$("$LSL")will put the entire string into the first argument, since it is all contained as one quoted string. Then${MYRA[0]}will be the only array element, and it will contain the entire string, which is not what you wanted.We also could have shortened this script by combining the steps, and it would look like this:declare -a MYRA MYRA=($(ls -ld $1))
If you want to know how many elements you have in your new array, just reference the variable${#MYRA[*]}or${#MYRA[@]}, either of which is a lot of special characters to type.- , "Using Array Variables"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Output with a Function Call
- InhaltsvorschauYou want to parse the output of some program into various variables to be used else-where in your program. Arrays are great when you are looping through the values, but not very readable if you want to refer to each separately, rather than by an index.Use a function call to parse the words:
#!/usr/bin/env bash # cookbook filename: parseViaFunc # # parse ls -l via function call # an example of output from ls -l follows: # -rw-r--r--1 albing users 126 2006-10-10 22:50 fnsize function lsparts () { PERMS=$1 LCOUNT=$2 OWNER=$3 GROUP=$4 SIZE=$5 CRDATE=$6 CRTIME=$7 FILE=$8 } lsparts $(ls -l "$1") echo $FILE has $LCOUNT 'link(s)' and is $SIZE bytes long.Here's what it looks like when it runs:$ ./fnsize fnsize fnsize has 1 link(s) and is 311 bytes long. $
We can let bash do the work of parsing by putting the text to be parsed on a function call. Calling a function is much like calling a shell script. bash parses the words into separate variables and assigns them to$1,$2, etc. Our function can just assign each positional parameter to a separate variable. If the variables are not declared locally then they are available outside as well as inside the function.We put quotes around the reference to$1in the ls command in case the filename supplied has spaces in its name. The quotes keep it all together so that ls sees it as a single filename and not as a series of separate filenames.We use quotes in the expression'link(s)'to avoid special treatment of the parentheses by bash. Alternatively, we could have put the entire phrase (except for theechoitself) inside of double quotes—double, not single, quotes so that the variable substitution (for$FILE, etc.) still occurs.- , "Defining Functions"
- , "Using Functions: Parameters and Return Values"
- , "Getting Your Plurals Right"
- , "Clearing the Screen When You Log Out"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing Text with a read Statement
- InhaltsvorschauThe are many ways to parse text with bash. What if I don't want to use a function? Is there another way?Use the
readstatement.#!/usr/bin/env bash # cookbook filename: parseViaRead # # parse ls -l with a read statement # an example of output from ls -l follows: # -rw-r--r--1 albing users 126 2006-10-10 22:50 fnsize ls -l "$1" | { read PERMS LCOUNT OWNER GROUP SIZE CRDATE CRTIME FILE ; echo $FILE has $LCOUNT 'link(s)' and is $SIZE bytes long. ; }Here we letreaddo all the parsing. It will break apart the input into words, where words are separated by whitespace, and assign each word to the variables named on thereadcommand. Actually, you can even change the separator, by setting the bash variable$IFS(which means Internal Field Separator) to whatever character you want for parsing; just remember to set it back!As you can see from the sample output ofls -l, we have tried to choose names that get at the meaning of each word in that output. SinceFILEis the last word, any extra fields will also be part of that variable. That way if the name has whitespace in it like "Beethoven Fifth Symphony" then all three words will end up in$FILE.- , "Saving or Grouping Output from Several Commands"
- , "Forgetting That Pipelines Make Subshells"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing with read into an Array
- InhaltsvorschauYou've got a varying number of words on each line of input, so you can't just assign each word to a predetermined variable.Use the
-aoption on the read statement, and the words will be read into an array variable.read -a MYRAY
Whether coming from user input or a pipeline,readwill parse the input into words, putting each word in its own array element. The variable does not need to be declared as an array—using it in this fashion is enough to make it into an array. Each element can be referenced with the bash array syntax, which is a zero-based array. So the second word on a line of input will be put into${MYRAY[1]}in our example. The number of words will determine the size of the array. In our example, the size of the array is${#MYRAY[@]}.- , "Getting User Input"
- , "Parsing Text with a read Statement"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Your Plurals Right
- InhaltsvorschauYou want to use a plural noun when you have more than one of an object. But you don't want to scatter
ifstatements all through your code.#!/usr/bin/env bash # cookbook filename: pluralize # # A function to make words plural by adding an s # when the value ($2) is != 1 or -1 # It only adds an 's'; it is not very smart. # function plural () { if [ $2 -eq 1 -o $2 -eq -1 ] then echo ${1} else echo ${1}s fi } while read num name do echo $num $(plural "$name" $num) doneThe function, though only set to handle the simple addition of ans, will do fine for many nouns. The function doesn't do any error checking of the number or contents of the arguments. If you wanted to use this script in a serious application, you might want to add those kinds of checks.We put the name in quotes when we call the plural function in case there are embedded blanks in the name. It did, after all, come from thereadstatement, and the last variable on areadstatement gets all the remaining text from the input line. You can see that in the following example.We put the solution script into a file named pluralize and ran it against the following data:$ cat input.file 1 hen 2 duck 3 squawking goose 4 limerick oyster 5 corpulent porpoise $ ./pluralize < input.file 1 hen 2 ducks 3 squawking gooses 4 limerick oysters 5 corpulent porpoises $
"Gooses" isn't correct English, but the script did what was intended. If you like the C-like syntax better, you could write theifstatement like this:if (( $2 == 1 || $2 == -1 ))
The square bracket (i.e., thetestbuilt-in) is the older form, more common across the various versions of bash, but either should work. Use whichever form's syntax is easiest for you to remember.We don't expect you would keep a file like pluralize around, but thepluralfunction might be handy to have as part of a larger scripting project. Then whenever you report on the count of something you could use thepluralfunction as part of the reference, as shown in the while loop above.- , "Looping with a read"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Taking It One Character at a Time
- InhaltsvorschauYou have some parsing to do and for whatever reason nothing else will do—you need to take your strings apart one character at a time.The substring function for variables will let you take things apart and another feature tells you how long a string is:
#!/usr/bin/env bash # cookbook filename: onebyone # # parsing input one character at a time while read ALINE do for ((i=0; i < ${#ALINE}; i++)) do ACHAR=${ALINE:i:1} # do something here, e.g. echo $ACHAR done doneThereadwill take input from standard in and put it, a line at a time, into the variable$ALINE. Since there are no other variables on thereadcommand, it takes the entire line and doesn't divvy it up.Theforloop will loop once for each character in the$ALINEvariable. We can compute how many times to loop by using${#ALINE}, which returns the length of the contents of$ALINE.Each time through the loop we assignACHARthe value of the one-character substring ofALINEthat begins at the ith position. That's simple enough. Now, what was it you needed to parse this way?- Check out the other parsing techniques in this chapter to see if you can avoid working at this low level
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Cleaning Up an SVN Source Tree
- InhaltsvorschauSubversion's
svn statuscommand shows all the files that have been modified, but if you have scratch files or other garbage lying around in your source tree, svn will list those, too. It would be useful to have a way to clean up your source tree, removing those files unknown to Subversion.Subversion won't know about new files unless and until you do ansvn addcommand. Don't run this script until you've added any new source files, or they'll be gone for good.svn status src | grep '^\?' | cut -c8- | \ while read fn; do echo "$fn"; rm -rf "$fn"; done
Thesvn statusoutput lists one file per line. It puts anMas the first character of a line for files that have been modified, anAfor newly added (but not yet committed) files, and a question mark for those about which it knows nothing. We just grep for those lines beginning with a question mark and cut off the leading eight columns of each line of output so that we are left with just the filename on each line. We read those filenames with areadstatement in awhile loop. The echo isn't strictly necessary, but it's useful to see what's being removed, just in case there is a mistake or an error. You can at least see that it's gone for good. When we do the remove, we use the-rfoptions in case the file is a directory, but mostly just to keep the remove quiet. Problems encountered with permissions and such are squelched by the-foption. It just removes the file as best as your permissions allow. We put the reference to the file-name in quotes"$fn"in case there are special characters (like spaces) in the filename.- , "Looping with a read"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting Up a Database with MySQL
- InhaltsvorschauYou want to create and initialize several databases using MySQL. You want them all to be initialized using the same SQL commands. Each database needs its own name, but each database will have the same contents, at least at initialization. You may need to do this setup over and over, as in the case where these databases are used as part of a test suite that needs to be reset when tests are rerun.A simple bash script can help with this administrative task:
#!/usr/bin/env bash # cookbook filename: dbiniter # # initialize databases from a standard file # creating databases as needed. DBLIST=$(mysql -e "SHOW DATABASES;" | tail +2) select DB in $DBLIST "new..." do if [[ $DB == "new..." ]] then printf "%b" "name for new db: " read DB rest echo creating new database $DB mysql -e "CREATE DATABASE IF NOT EXISTS $DB;" fi if [ "$DB" ] then echo Initializing database: $DB mysql $DB < ourInit.sql fi doneThetail+2is added to remove the heading from the list of databases (see , "Skipping a Header in a File").Theselectcreates the menus showing the existing databases. We added the literal"new…"as an additional choice (see , "Selecting from a List of Options" and , "Creating Simple Menus").When the user wants to create a new database, we prompt for and read a new name, but we use two fields on thereadcommand as a bit of error handling. If the user types more than one name on the line, we only use the first name—it gets put into the variable$DBwhile the rest of the input is put into$restand ignored. (We could add an error check to see if$restis null.)Whether created anew or chosen from the list of extant databases, if the$DBvariable is not empty, it will invoke mysql one more time to feed it the set of SQL statements that we've put into the file ourInit.sql as our standardized initialization sequence.If you're going to use a script like this, you might need to add parameters to your mysql command, such as-uand-pto prompt for username and password. It will depend on how your database and its permissions are configured or whether you have a file namedEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Isolating Specific Fields in Data
- InhaltsvorschauYou need to extract one or more fields from each line of output.Use cut if there are delimiters you can easily pick out, even if they are different for the beginning and end of the field you need:
# Here's an easy one, what users, home directories and shells do # we have on this NetBSD system $ cut -d':' -f1,6,7 /etc/passwd root:/root:/bin/csh toor:/root:/bin/sh daemon:/:/sbin/nologin operator:/usr/guest/operator:/sbin/nologin bin:/:/sbin/nologin games:/usr/games:/sbin/nologin postfix:/var/spool/postfix:/sbin/nologin named:/var/chroot/named:/sbin/nologin ntpd:/var/chroot/ntpd:/sbin/nologin sshd:/var/chroot/sshd:/sbin/nologin smmsp:/nonexistent:/sbin/nologin uucp:/var/spool/uucppublic:/usr/libexec/uucp/uucico nobody:/nonexistent:/sbin/nologin jp:/home/jp:/usr/pkg/bin/bash # What is the most popular shell on the system? $ cut -d':' -f7 /etc/passwd | sort | uniq -c | sort -rn 10 /sbin/nologin 2 /usr/pkg/bin/bash 1 /bin/csh 1 /bin/sh 1 /usr/libexec/uucp/uucico # Now let's see the first two directory levels $ cut -d':' -f6 /etc/passwd | cut -d'/' -f1-3 | sort -u / /home/jp /nonexistent /root /usr/games /usr/guest /var/chroot /var/spool
Use awk to split on multiples of whitespace, or if you need to rearrange the order of the output fields. Note the → denotes a tab character in the output. The default is space but you can change that using$OFS:# Users, home directories and shells, but swap the last two # and use a tab delimiter $ awk 'BEGIN {FS=":"; OFS="\t"; } { print $1,$7,$6; }' /etc/passwd root → /bin/csh → /root toor → /bin/sh → /root daemon → /sbin/nologin → / operator → /sbin/nologin → /usr/guest/operator bin → /sbin/nologin → / games → /sbin/nologin → /usr/games postfix → /sbin/nologin → /var/spool/postfix named → /sbin/nologin → /var/chroot/named ntpd → /sbin/nologin → /var/chroot/ntpd sshd → /sbin/nologin → /var/chroot/sshd smmsp → /sbin/nologin → /nonexistent uucp → /usr/libexec/uucp/uucico → /var/spool/uucppublic nobody → /sbin/nologin → /nonexistent jp → /usr/pkg/bin/bash → /home/jp # Multiples of whitespace and swapped, first field removed $ grep '^# [1-9]' /etc/hosts | awk '{print $3,$2}' 10.255.255.255 10.0.0.0 172.31.255.255 172.16.0.0 192.168.255.255 192.168.0.0Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Updating Specific Fields in Data Files
- InhaltsvorschauYou need to extract certain parts (fields) of a line (record) and update them.In the simple case, you want to extract a single field from a line, then perform some operation on it. For that, you can use cut or awk. See , "Isolating Specific Fields in Data" for details.For the more complicated case, you need to modify a field in a data file without extracting it. If it's a simple search and replace, use sed.For example, let's switch everyone from csh to sh on this NetBSD system.
$ grep csh /etc/passwd root:*:0:0:Charlie &:/root:/bin/csh $ sed 's/csh$/sh/' /etc/passwd | grep '^root' root:*:0:0:Charlie &:/root:/bin/sh
You can use awk if you need to do arithmetic on a field or modify a string only in a certain field:$ cat data_file Line 1 ends Line 2 ends Line 3 ends Line 4 ends Line 5 ends $ awk '{print $1, $2+5, $3}' data_file Line 6 ends Line 7 ends Line 8 ends Line 9 ends Line 10 ends # If the second field contains '3', change it to '8' and mark it $ awk '{ if ($2 == "3") print $1, $2+5, $3, "Tweaked" ; else print $0; }' data_file Line 1 ends Line 2 ends Line 8 ends Tweaked Line 4 ends Line 5 endsThe possibilities here are as endless as your data, but hopefully the examples above will give you enough of a start to easily modify your data.- man awk
- man sed
- , "Figuring Out Date and Time Arithmetic"
- , "Isolating Specific Fields in Data"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Trimming Whitespace
- InhaltsvorschauYou need to trim leading and/or trailing whitespace from lines for fields of data.These solutions rely on a bash- specific treatment of read and
$REPLY. See the end of the discussion for an alternate solution.First, we'll show a file with some leading and trailing whitespace. Note we add~~to show the whitespace. Note the → denotes a literal tab character in the output:# Show the whitespace in our sample file $ while read; do echo ~~"$REPLY"~~; done < whitespace ~~ This line has leading spaces.~~ ~~This line has trailing spaces. ~~ ~~ This line has both leading and trailing spaces. ~~ ~~ → Leading tab.~~ ~~Trailing tab. → ~~ ~~ → Leading and trailing tab. → ~~ ~~ → Leading mixed whitespace.~~ ~~Trailing mixed whitespace. → ~~ ~~ → Leading and trailing mixed whitespace. → ~~
To trim both leading and trailing whitespace use$IFSadd the built-inREPLYvariable (see the discussion for why this works):$ while read REPLY; do echo ~~"$REPLY"~~; done < whitespace ~~This line has leading spaces.~~ ~~This line has trailing spaces.~~ ~~This line has both leading and trailing spaces.~~ ~~Leading tab.~~ ~~Trailing tab.~~ ~~Leading and trailing tab.~~ ~~Leading mixed whitespace.~~ ~~Trailing mixed whitespace.~~ ~~Leading and trailing mixed whitespace.~~To trim only leading or only trailing spaces, use a simple pattern match:# Leading spaces only $ while read; do echo "~~${REPLY## }~~"; done < whitespace ~~This line has leading spaces.~~ ~~This line has trailing spaces. ~~ ~~This line has both leading and trailing spaces. ~~ ~~ → Leading tab.~~ ~~Trailing tab. ~~ ~~ → Leading and trailing tab. → ~~ ~~ → Leading mixed whitespace.~~ ~~Trailing mixed whitespace. → ~~ ~~ → Leading and trailing mixed whitespace. → ~~ # Trailing spaces only $ while read; do echo "~~${REPLY%% }~~"; done < whitespace ~~ This line has leading spaces.~~ ~~This line has trailing spaces.~~ ~~ This line has both leading and trailing spaces.~~ ~~ → Leading tab.~~ ~~Trailing tab. ~~ ~~ → Leading and trailing tab. → ~~ ~~ → Leading mixed whitespace.~~ ~~Trailing mixed whitespace. → ~~ ~~ → Leading and trailing mixed whitespace. → ~~Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Compressing Whitespace
- InhaltsvorschauYou have runs of whitespace in a file (perhaps it is fixed length, space padded) and you need to compress the spaces down to a single character or delimiter.Use tr or awk as appropriate.If you are trying to compress runs of whitespace down to a single character, you can use tr, but be aware that you may damage the file if it is not well formed. For example, if fields are delimited by multiple whitespace characters but internally have spaces, compressing multiple spaces down to one space will remove that distinction. Imagine if
the _ charactersin the following example were spaces instead. Note the → denotes a literal tab character in the output.$ cat data_file Header1 Header2 Header3 Rec1_Field1 Rec1_Field2 Rec1_Field3 Rec2_Field1 Rec2_Field2 Rec2_Field3 Rec3_Field1 Rec3_Field2 Rec3_Field3 $ cat data_file | tr -s ' ' '\t' Header1 → Header2 → Header3 Rec1_Field1 → Rec1_Field2 → Rec1_Field3 Rec2_Field1 → Rec2_Field2 → Rec2_Field3 Rec3_Field1 → Rec3_Field2 → Rec3_Field3
If your field delimiter is more than a single character, tr won't work since it translates single characters from its first set into the matching single character in the second set. You can use awk to combine or convert field separators. awk's internal field separatorFSaccepts regular expressions, so you can separate on pretty much anything. There is a handy trick to this as well. An assignment to any field causes awk to reassemble the record using the output field separatorOFS. So assigning field one to itself and then printing the record has the effect of translatingFStoOFSwithout you having to worry about how many records there are in the data.In this example, multiple spaces delimit fields, but fields also have internal spaces, so the more simple case ofawk 'BEGIN { OFS = "\t"} {$1=$1; print }' data_file1won't work. Here is a data file:$ cat data_file1 Header1 Header2 Header3 Rec1 Field1 Rec1 Field2 Rec1 Field3 Rec2 Field1 Rec2 Field2 Rec2 Field3 Rec3 Field1 Rec3 Field2 Rec3 Field
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Processing Fixed-Length Records
- InhaltsvorschauYou need to read and process data that is in a fixed-length (also called fixed-width) form.Use Perl or gawk 2.13 or greater. Given a file like:
$ cat fixed-length_file Header1-----------Header2-------------------------Header3--------- Rec1 Field1 Rec1 Field2 Rec1 Field3 Rec2 Field1 Rec2 Field2 Rec2 Field3 Rec3 Field1 Rec3 Field2 Rec3 Field3
You can process it using GNU's gawk, by settingFIELDWIDTHSto the correct field lengths, settingOFSas desired, and making an assignment so gawk rebuilds the record (see the awk trick in , "Trimming Whitespace"). However, gawk does not remove the spaces used in padding the original record, so we use two gsubs to do that, one for all the internal fields and the other for the last field in each record. Finally, we just print. Note the → denotes a literal tab character in the output. The output is a little hard to read, so there is a hex dump as well. Recall that ASCII tab is09while ASCII space is20.$ gawk ' BEGIN { FIELDWIDTHS = "18 32 16"; OFS = "\t" } { $1 = $1; gsub(/ +\t/, "\ t"); gsub(/ +$/, ""); print }' fixed-length_file Header1----------- → Header2------------------------- → Header3--------- Rec1 Field1 → Rec1 Field2 → Rec1 Field3 Rec2 Field1 → Rec2 Field2 → Rec2 Field3 Rec3 Field1 → Rec3 Field2 → Rec3 Field3 $ gawk ' BEGIN { FIELDWIDTHS = "18 32 16"; OFS = "\t" } { $1 = $1; gsub(/ +\t/, "\ t"); gsub(/ +$/, ""); print }' fixed-length_file | hexdump -C 00000000 48 65 61 64 65 72 31 2d 2d 2d 2d 2d 2d 2d 2d 2d |Header1---------| 00000010 2d 2d 09 48 65 61 64 65 72 32 2d 2d 2d 2d 2d 2d |--.Header2------| 00000020 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d |----------------| 00000030 2d 2d 2d 09 48 65 61 64 65 72 33 2d 2d 2d 2d 2d |---.Header3-----| 00000040 2d 2d 2d 2d 0a 52 65 63 31 20 46 69 65 6c 64 31 |----.Rec1 Field1| 00000050 09 52 65 63 31 20 46 69 65 6c 64 32 09 52 65 63 |.Rec1 Field2.Rec| 00000060 31 20 46 69 65 6c 64 33 0a 52 65 63 32 20 46 69 |1 Field3.Rec2 Fi| 00000070 65 6c 64 31 09 52 65 63 32 20 46 69 65 6c 64 32 |eld1.Rec2 Field2| 00000080 09 52 65 63 32 20 46 69 65 6c 64 33 0a 52 65 63 |.Rec2 Field3.Rec| 00000090 33 20 46 69 65 6c 64 31 09 52 65 63 33 20 46 69 |3 Field1.Rec3 Fi| 000000a0 65 6c 64 32 09 52 65 63 33 20 46 69 65 6c 64 33 |eld2.Rec3 Field3| 000000b0 0a |.| 000000b1Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Processing Files with No Line Breaks
- InhaltsvorschauYou have a large file with no line breaks, and you need to process it.Pre-process the file and add line breaks in appropriate places. For example, Open- Office.org's Open Document Format (ODF) files are basically zipped XML files. It is possible to unzip them and grep the XML, which we did a lot while writing this book. See , "Comparing Two Documents" for a more comprehensive treatment of ODF files. In this example, we insert a newline after every closing angle bracket (>). That makes it much easier to process the file using grep or other textutils. Note that we must enter a backslash followed immediately by the Enter key to embed an escaped newline in the sed script:
$ wc -l content.xml 1 content.xml $ sed -e 's/>/>\ /g' content.xml | wc -l 1687
If you have fixed-length records with no newlines, do this instead, where 48 is the length of the record.$ cat fixed-length Line_1_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_2_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_3_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_4_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_5_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_6_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_7_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_8_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_9_ _ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_10_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_11_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZLine_12_ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ $ wc -l fixed-length 1 fixed-length $ sed 's/.\{48\}/&\ /g;' fixed-length Line_1_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_2_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_3_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_4_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_5_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_6_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_7_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_8_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_9_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_10_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_11_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_12_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ $ perl -pe 's/(.{48})/$1\n/g;' fixed-length Line_1_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_2_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_3_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_4_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_5_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_6_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_7_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_8_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_9_ _aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_10_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_11_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZ Line_12_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaZZZEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Converting a Data File to CSV
- InhaltsvorschauYou have a data file that you need to convert to a Comma Separated Values (CSV) file.Use awk to convert the data into CSV format:
$ awk 'BEGIN { FS="\t"; OFS="\",\"" } { gsub(/"/, "\"\""); $1 = $1; printf "\"%s\"\ n", $0}' tab_delimited "Line 1","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 2","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 3","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 4","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes"You can do the same thing in Perl also:$ perl -naF'\t' -e 'chomp @F; s/"/""/g for @F; print q(").join(q(","), @F).qq("\n);' tab_delimited "Line 1","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 2","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 3","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes" "Line 4","Field 2","Field 3","Field 4","Field 5 with ""internal"" double-quotes"First of all, it's tricky to define exactly what CSV really means. There is no formal specification, and various vendors have implemented various versions. Our version here is very simple, and should hopefully work just about anywhere. We place double quotes around all fields (some implementations only quote strings, or strings with internal commas), and we double internal double quotes.To do that, we have awk split up the input fields using a tab as the field separator, and set the output field separator (OFS) to ",". We then globally replace any double quotes with two double quotes, make an assignment so awk rebuilds the record (see the awk trick in , "Trimming Whitespace") and print out the record with leading and trailing double quotes. We have to escape double quotes in several places, which looks a little cluttered, but otherwise this is very straightforward.- awk FAQ
- , "Trimming Whitespace"
- , "Parsing a CSV Data File"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Parsing a CSV Data File
- InhaltsvorschauYou have a Comma Separated Values (CSV) data file that you need to parse.Unlike the previous recipe for converting to CSV, there is no easy way to do this, since it's tricky to define exactly what CSV really means.Possible solutions for you to explore are:
- Perl: Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly) has a regex to do this
- Perl: See the CPAN (http://www.cpan.org/) for various modules
- Load the CSV file into a spreadsheet (OpenOffice.org's Calc and Microsoft's Excel both work), then copy and paste into a text editor and you should get tab delimited output that you can now use easily
As noted in , "Converting a Data File to CSV," there is no formal specification for CSV, and that fact, combined with data variations, makes this task much harder than it sounds.- , "Converting a Data File to CSV"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 14: Writing Secure Shell Scripts
- InhaltsvorschauWriting secure shell scripts?! How can shell scripts be secure when you can read the source code?Any system that depends on concealing implementation details is attempting to use security by obscurity, and that is no security at all. Just ask the major software manufacturers whose source code is a closely guarded trade secret, yet whose products are incessantly vulnerable to exploits written by people who have never seen that source code. Contrast that with the code from OpenSSH and OpenBSD, which is totally open, yet very secure.Security by obscurity will never work for long, though some forms of it can be a useful additional layer of security. For example, having daemons assigned to listen on nonstandard port numbers will keep a lot of the so-called script-kiddies away. But security by obscurity must never be the only layer of security because sooner or later, someone is going to discover whatever you've hidden.As Bruce Schneier says, security is a process. It's not a product, object, or technique, and it is never finished. As technology, networks, attacks and defenses evolve, so must your security process. So what does it mean to write secure shell scripts?Secure shell scripts will reliably do what they are supposed to do, and only what they are supposed to do. They won't lend themselves to being exploited to gain root access, they won't accidentally
rm -rf /, and they won't leak information, such as passwords. They will be robust, but will fail gracefully. They will tolerate in advertent user mistakes and sanitize all user input. They will be as simple as possible, and contain only clear, readable code and documentation so that the intention of each line is unambiguous.That sounds a lot like any well-designed, robust program, doesn't it? Security should be part of any good design process from the start—it shouldn't be tacked on at the end. In this chapter we've highlighted the most common security weaknesses and questions, and shown you how to tackle them.A lot has been written about security over the years. If you're interested, Practical UNIX & Internet SecurityEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Avoiding Common Security Problems
- InhaltsvorschauYou want to avoid common security problems in your scripting.Validate all external input, including interactive input and that from configuration files and interactive use. In particular, never
evalinput that you have not checked very thoroughly.Use secure temporary files, ideally in secure temporary directories.Make sure you are using trusted external executables.In a way, this recipe barely scratches the surface of scripting and system security. Yet it also covers the most common security problems you'll find.Data validation, or rather the lack of it, is a huge deal in computer security right now. This is the problem that leads to buffer overflows, which are by far the most common class of exploit going around. bash doesn't suffer from this issue in the same way that C does, but the concepts are the same. In the bash world it's more likely that unvalidated input will contain something like;rm-rf/than a buffer over-flow; however, neither is welcome. Validate your data!Race conditions are another big issue, closely tied to the problem of an attacker gaining an ability to write over unexpected files. A race condition exists when two or more separate events must occur in the correct order at the correct time without external interference. They often result in providing an unprivileged user with read and/or write access to files they shouldn't be able to access, which in turn can result in so-called privilege escalation, where an ordinary user can gain root access. Insecure use of temporary files is a very common factor in this kind of attack. Using secure temporary files, especially inside secure temporary directories, will eliminate this attack vector.Another common attack vector is trojaned utilities. Like the Trojan horse, these appear to be one thing while they are in fact something else. The canonical example here is the trojaned ls command that works just like the real ls command except when run by root. In that case it creates a new user called r00t, with a default password known to the attacker and deletes itself. Using a secure$PATHis about the best you can do from the scripting side. From the systems side there are many tools such as Tripwire and AIDE to help you assure system integrity.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Avoiding Interpreter Spoofing
- InhaltsvorschauYou want to avoid certain kinds of setuid root spoofing attacks.Pass a single trailing dash to the shell, as in:
#!/bin/bash -
The first line of a script is a magic line (often called the shebang line) that tells the kernel what interpreter to use to process the rest of the file. The kernel will also look for a single option to the specified interpreter. There are some attacks that take advantage of this fact, but if you pass an argument along, they are avoided. See http://www.faqs.org/faqs/unix-faq/faq/part4/section-7.html for details.However, hard-coding the path to bash may present a portability issue. See , "Finding bash Portably for #!" for details.- , "Writing setuid or setgid Scripts"
- , "Finding bash Portably for #!"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting a Secure $PATH
- InhaltsvorschauYou want to make sure you are using a secure path.Set
$PATHto a known good state at the beginning of every script:# Set a sane/secure path PATH='/usr/local/bin:/bin:/usr/bin' # It's almost certainly already marked for export, but make sure export PATH
Or use the getconf utility to get a path guaranteed by POSIX to find all of the standard utilities:export PATH=$(getconf PATH)
There are two portability problems with the example above. First, `` is more portable (but less readable) than $(). Second, having theexportcommand on the same line as the variable assignment won't always work.var='foo'; export varis more portable thanexport var='foo'. Also note that theexportcommand need only be used once to flag a variable to be exported to child processes.If you don't use getconf, our example is a good default path for starters, though you may need to adjust it for your particular environment or needs. You might also use the less portable version:export PATH='/usr/local/bin:/bin:/usr/bin'
Depending on your security risk and needs, you should also consider using absolute paths. This tends to be cumbersome and can be an issue where portability is concerned, as different operating systems put tools in different places. One way to mitigate these issues to some extent is to use variables. If you do this, sort them so you don't end up with the same command three times because you missed it scanning the unsorted list.One other advantage of this method is that it makes it very easy to see exactly what tools your script depends on, and you can even add a simple function to make sure that each tool is available and executable before your script really gets going.#!/usr/bin/env bash # cookbook filename: finding_tools # export may or may not also be needed, depending on what you are doing # These are fairly safe bets _cp='/bin/cp' _mv='/bin/mv' _rm='/bin/rm' # These are a little trickier case $(/bin/uname) in 'Linux') _cut='/bin/cut' _nice='/bin/nice' # [...] ;; 'SunOS') _cut='/usr/bin/cut' _nice='/usr/bin/nice' # [...] ;; # [...] esac
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Clearing All Aliases
- InhaltsvorschauYou need to make sure that there are no malicious aliases in your environment for security reasons.Use the
\unalias-acommand to unalias any existing aliases.If an attacker can trick root or even another user into running a command, they will be able to gain access to data or privileges they shouldn't have. One way to trick another user into running a malicious program is to create an alias to some other common program (e.g., ls).The leading \, which suppresses alias expansion, is very important because without it you can do evil things like this:$ alias unalias=echo $ alias builtin=ls $ builtin unalias vi ls: unalias: No such file or directory ls: vi: No such file or directory $ unalias -a -a
- , "Redefining Commands with alias"
- , "Avoiding Aliases, Functions"
- , "Shortening or Changing Command Names"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Clearing the Command Hash
- InhaltsvorschauYou need to make sure that your command hash has not been subverted.Use the
hash -rcommand to clear entries from the command hash.On execution, bash "remembers" the location of most commands found in the$PATHto speed up subsequent invocations.If an attacker can trick root or even another user into running a command, they will be able to gain access to data or privileges they shouldn't have. One way to trick another user into running a malicious program is to poison the hash so that the wrong program may be run.- , "Finding World-Writable Directories in Your $PATH"
- , "Adding the Current Directory to the $PATH"
- , "Setting a POSIX $PATH"
- , "Change Your $PATH Permanently"
- , "Change Your $PATH Temporarily"
- , "Forgetting That the Current Directory Is Not in the $PATH"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Preventing Core Dumps
- InhaltsvorschauYou want to prevent your script from dumping core in the case of an unrecoverable error, since core dumps may contain sensitive data from memory such as passwords.Use the bash built-in ulimit to set the core file size limit to 0, typically in your .bashrc file:
ulimit -H -c 0 --
Core dumps are intended for debugging and contain an image of the memory used by the process at the time it failed. As such, the file will contain anything the process had stored in memory (e.g., user-entered passwords).Set this in a system-level file such as /etc/profile or /etc/bashrc to which users have no write access if you don't want them to be able to change it.- help ulimit
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting a Secure $IFS
- InhaltsvorschauYou want to make sure your Internal Field Separator environment variable is clean.Set it to a known good state at the beginning of every script using this clear (but not POSIX-compliant) syntax:
# Set a sane/secure IFS (note this is bash & ksh93 syntax only--not portable!) IFS=$' \t\n'
As noted, this syntax is not portable. However, the canonical portable syntax is unreliable because it may easily be inadvertently stripped by editors that trim whitespace. The values are traditionally space, tab, newline—and the order is important. $*, which returns all positional parameters, the special${!prefix@}and${!prefix*}parameter expansions, and programmable completion, all use the first value of$IFSas their separator.The typical method for writing that leaves a trailing space and tab on the first line:1 IFS=' • → ¶ 2 '
Newline, space, tab is less likely to be trimmed, but changes the default order, which may result in unexpected results from some commands.1 IFS='• ¶ 2 • → '
- , "Trimming Whitespace"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting a Secure umask
- InhaltsvorschauYou want to make sure you are using a secure
umask.Use the bash built-in umask to set a known good state at the beginning of every script:# Set a sane/secure umask variable and use it # Note this does not affect files already redirected on the command line # 002 results in 0774 perms, 077 results in 0700 perms, etc... UMASK=002 umask $UMASK
We set the$UMASKvariable in case we need to use different masks elsewhere in the program. You could just as easily do without it; it's not a big deal.umask 002
Remember thatumaskis a mask that specifies the bits to be taken away from the default permissions of777for directories and 666 for files. When in doubt, test it out:# Run a new shell so you don't affect your current environment /tmp$ bash # Check the current settings /tmp$ touch um_current # Check some other settings /tmp$ umask 000 ; touch um_000 /tmp$ umask 022 ; touch um_022 /tmp$ umask 077 ; touch um_077 /tmp$ ls -l um_* -rw-rw-rw- 1 jp jp 0 Jul 22 06:05 um000 -rw-r--r-- 1 jp jp 0 Jul 22 06:05 um022 -rw------- 1 jp jp 0 Jul 22 06:05 um077 -rw-rw-r-- 1 jp jp 0 Jul 22 06:05 umcurrent # Clean up and exit the sub-shell /tmp$ rm um_* /tmp$ exit
- help umask
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding World-Writable Directories in Your $PATH
- InhaltsvorschauYou want to make sure that there are no world-writable directories in root's
$PATH. To see why, read , "Adding the Current Directory to the $PATH."Use this simple script to check your$PATH. Use it in conjunction withsu -orsudo to check paths for other users:#!/usr/bin/env bash # cookbook filename: chkpath.1 # Check your $PATH for world-writable or missing directories exit_code=0 for dir in ${PATH//:/ }; do [ -L "$dir" ] && printf "%b" "symlink, " if [ ! -d "$dir" ]; then printf "%b" "missing\t\t" (( exit_code++ )) elif [ "$(ls -lLd $dir | grep '^d.......w. ')" ]; then printf "%b" "world writable\t" (( exit_code++ )) else printf "%b" "ok\t\t" fi printf "%b" "$dir\n" done exit $exit_codeFor example:# ./chkpath ok /usr/local/sbin ok /usr/local/bin ok /sbinok /bin ok /usr/sbin ok /usr/bin ok /usr/X11R6/bin ok /root/bin missing /does_not_exist world writable /tmp symlink, world writable /tmp/bin symlink, ok /root/sbin
We convert the$PATHto a space-delimited list using the technique from , "Finding a File Using a List of Possible Locations," test for symbolic links(-L), and make sure the directory actually exists(-d). Then we get a long directory listing(-l), dereferencing symbolic links(-L), and listing the directory name only(-d), not the directory's contents. Then we finally get to grep for world-writable directories.As you can see, we spaced out theokdirectories, while directories with a problem may get a little cluttered. We also broke the usual rule of Unix tools being quiet unless there's a problem, because we felt it was a useful opportunity to see exactly what is in your path and give it a once-over in addition to the automated check.We also provide an exit code of zero on success with no problems detected in the$PATH, or the count of errors found. With a little more tweaking, we can add the file's mode, owner, and group into the output, which might be even more valuable to check:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adding the Current Directory to the $PATH
- InhaltsvorschauHaving to
type ./scriptis tedious and you'd rather just add . (or an empty directory, meaning a leading or trailing : ora ::in the middle) to your$PATH.We advise against doing this for any user, but we strongly advise against doing this for root. If you absolutely must do this, make sure . comes last. Never do it as root.As you know, the shell searches the directories listed in$PATHwhen you enter a command name without a path. The reason not to add . is the same reason not to allow world-writable directories in your$PATH.Say you are in /tmp and have . as the first thing in your$PATH. If you type ls and there happens to be a file called /tmp/ls, you will run that file instead of the /bin/ls you meant to run. Now what? Well, it depends. It's possible (even likely given the name) that /tmp/ls is a malicious script, and if you have just run it as root there is no telling what it could do, up to and including deleting itself when it's finished to remove the evidence.So what if you put it last? Well, have you ever typed mc instead of mv? We have. So unless Midnight Commander is installed on your system, you could accidentally run ./mc when you meant /bin/mv, with the same results as above.Just say no to dot!- Section 2.13 of http://www.faqs.org/faqs/unix-faq/faq/part2/
- , "Finding a File Using a List of Possible Locations"
- , "Setting a Secure $PATH"
- , "Finding World-Writable Directories in Your $PATH"
- , "Setting a POSIX $PATH"
- , "Change Your $PATH Permanently"
- , "Change Your $PATH Temporarily"
- , "Forgetting That the Current Directory Is Not in the $PATH"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Secure Temporary Files
- InhaltsvorschauYou need to create a temporary file or directory, but are aware of the security implications of using a predictable name.The easy and "usually good enough" solution is to just use
$RANDOMinline in your script. For example:# Make sure $TMP is set to something [ -n "$TMP" ] || TMP='/tmp' # Make a "good enough" random temp directory until [ -n "$temp_dir" -a ! -d "$temp_dir" ]; do temp_dir="/tmp/meaningful_prefix.${RANDOM}${RANDOM}${RANDOM}" done mkdir -p -m 0700 $temp_dir || { echo "FATAL: Failed to create temp dir '$temp_dir': $?"; exit 100 } # Make a "good enough" random temp file until [ -n "$temp_file" -a ! -e "$temp_file" ]; do temp_file="/tmp/meaningful_prefix.${RANDOM}${RANDOM}${RANDOM}" done touch $temp_file && chmod 0600 $temp_file || { echo "FATAL: Failed to create temp file '$temp_file': $?"; exit 101 }Even better, use both a random temporary directory and a random filename!# cookbook filename: make_temp # Make a "good enough" random temp directory until [ -n "$temp_dir" -a ! -d "$temp_dir" ]; do temp_dir="/tmp/meaningful_prefix.${RANDOM}${RANDOM}${RANDOM}" done mkdir -p -m 0700 $temp_dir \ || { echo "FATAL: Failed to create temp dir '$temp_dir': $?"; exit 100 } # Make a "good enough" random temp file in the temp dir temp_file="$temp_dir/meaningful_prefix.${RANDOM}${RANDOM}${RANDOM}" touch $temp_file && chmod 0600 $temp_file \ || { echo "FATAL: Failed to create temp file '$temp_file': $?"; exit 101 }No matter how you do it, don't forget to set a trap to clean up. As noted,$temp_dirmust be set before this trap is declared, and its value must not change. If those things aren't true, rewrite the logic to account for your needs.# cookbook filename: clean_temp # Do our best to clean up temp files no matter what # Note $temp_dir must be set before this, and must not change! cleanup="rm -rf $temp_dir" trap "$cleanup" ABRT EXIT HUP INT QUIT
$RANDOMhas been available since at least bash-2.0, and using it is probably good enough. Simple code is better and easier to secure than complicated code, so usingEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Validating Input
- InhaltsvorschauYou've asked for input (e.g., from a user or a program) and to ensure security or data integrity you need to make sure you got what you asked for.There are various ways to validate your input, depending on what the input is and how strict you need to be.Use pattern matching for simple "it matches or it doesn't" situations (see , "Testing for Equal," , "Testing with Pattern Matches," and , "Testing with Regular Expressions").
[[ "$raw_input" == *.jpg ]] && echo "Got a JPEG file."
Use a case statement when there are various things that might be valid (see , "Branching Many Ways" and , "Parsing Command-Line Arguments").# cookbook filename: validate_using_case case $raw_input in *.company.com ) # Probably a local hostname ;; *.jpg ) # Probably a JPEG file ;; *.[jJ][pP][gG] ) # Probably a JPEG file, case insensitive ;; foo | bar ) # entered 'foo' or 'bar ;; [0-9][0-9][0-9] ) # A 3 digit number ;; [a-z][a-z][a-z][a-z] ) # A 4 lower-case char word ;; * ) # None of the above ;; esac
Use a regular expression when pattern matching isn't specific enough and you have bash version 3.0+ (see , "Testing with Regular Expressions"). This example is looking for a three to six alphanumeric character filename with a.jpgextension (case sensitive):[[ "$raw_input" =~ [[:alpha:]]{3,6}\.jpg ]] && echo "Got a JPEG file."For a larger and more detailed example, see the examples/scripts/shprompt in a recent bash tarball. Note this was written by Chet Ramey, who maintains bash:# shprompt -- give a prompt and get an answer satisfying certain criteria # # shprompt [-dDfFsy] prompt # s = prompt for string # f = prompt for filename # F = prompt for full pathname to a file or directory # d = prompt for a directory name # D = prompt for a full pathname to a directory # y = prompt for y or n answer # # Chet Ramey # chet@ins.CWRU.Edu
For a similar example, see examples/scripts.noah/y_or_n_p.bash written circa 1993 by Noah Friedman and later converted toEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting Permissions
- InhaltsvorschauYou want to set permissions in a secure manner.If you need to set exact permissions for security reasons (or you are sure that you don't care what is already there, you just need to change it), use chmod with 4-digit octal modes.
$ chmod 0755 some_scriptIf you only want to add or remove permissions, but need to leave other existing permissions unchanged, use the + and - operations in symbolic mode.$ chmod +x some_scriptIf you try to recursively set permissions on all the files in a directory structure using something likechmod -R0644some_directory then you'll regret it because you've now rendered any subdirectories non-executable, which means you won't be able to access their content, cd into them, or traverse below them. Use find and xargs with chmod to set the files and directories individually.$ find some_directory -type f | xargs chmod 0644 # File perms $ find some_directory -type d | xargs chmod 0755 # Dir. perms
Of course, if you only want to set permissions on the files in a single directory (non-recursive), just cd in there and set them.When creating a directory, usemkdir -mmode new_directory since you not only accomplish two tasks with one command, but you avoid any possible race condition between creating the directory and setting the permissions.Many people are in the habit of using three-digit octal modes, but we like to use all four possible digits to be explicit about what we mean to do with all attributes. We also prefer using octal mode when possible because it's very clear what permissions you are going to end up with. You may also use the absolute operation (=) in symbolic mode if you like, but we're traditionalists who like the old octal method best.Ensuring the final permissions when using the symbolic mode and the + or - operations is trickier since they are relative and not absolute. Unfortunately, there are many cases where you can't simply arbitrarily replace the existing permissions usingoctalmode. In such cases you have no choice but to use symbolic mode, often using + to add a permission while not disturbing other existing permissions. Consult your specific system'sEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Leaking Passwords into the Process List
- Inhaltsvorschaups may show passwords entered on the command line in the clear. For example:
$ ./cheesy_app -u user -p password & [1] 13301 $ ps PID TT STAT TIME COMMAND 5280 p0 S 0:00.08 -bash 9784 p0 R+ 0:00.00 ps 13301 p0 S 0:00.01 /bin/sh ./cheesy_app -u user -p password
Try really hard not to use passwords on the command line.Really. Don't do that.Many applications that provide a-por similar switch will also prompt you if a password required and you do not provide it on the command line. That's great for interactive use, but not so great in scripts. You may be tempted to write a trivial "wrapper" script or an alias to try and encapsulate the password on the command line. Unfortunately, that won't work since the command is eventually run and so ends up in the process list anyway. If the command can accept the password on STDIN, you may be able to pass it in that way. That creates other problems, but at least avoids displaying the password in the process list.$ ./bad_app ~.hidden/bad_apps_password
If that won't work, you'll need to either find a new app, patch the one you are using, or just live with it.- , "Prompting for a Password"
- , "Using Passwords in Scripts"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Writing setuid or setgid Scripts
- InhaltsvorschauYou have a problem you think you can solve by using the setuid or setgid bit on a shell script.Use Unix groups and file permissions and/or sudo to grant the appropriate users the least privilege they need to accomplish their task.Using the setuid or setgid bit on a shell script will create more problems—especially security problems—than it solves. Some systems (such as Linux) don't even honor the setuid bit on shell scripts, so creating setuid shell scripts creates an unnecessary portability problem in addition to the security risks.setuid root scripts are especially dangerous, so don't even think about it. Use sudo.setuid and setgid have a different meaning when applied to directories than they do when applied to executable files. When one of these is set on a directory it causes any newly created files or subdirectories to be owned by the directory's owner or group, respectively.Note you can check a file to see if it is setuid by using test
-uor setgid by usingtest -g.$ mkdir suid_dir sgid_dir $ touch suid_file sgid_file $ ls -l total 4 drwxr-xr-x 2 jp users 512 Dec 9 03:45 sgid_dir -rw-r--r-- 1 jp users 0 Dec 9 03:45 sgid_file drwxr-xr-x 2 jp users 512 Dec 9 03:45 suid_dir -rw-r--r-- 1 jp users 0 Dec 9 03:45 suid_file $ chmod 4755 suid_dir suid_file $ chmod 2755 sgid_dir sgid_file $ ls -l total 4 drwxr-sr-x 2 jp users 512 Dec 9 03:45 sgid_dir -rwxr-sr-x 1 jp users 0 Dec 9 03:45 sgid_file drwsr-xr-x 2 jp users 512 Dec 9 03:45 suid_dir -rwsr-xr-x 1 jp users 0 Dec 9 03:45 suid_file $ [ -u suid_dir ] && echo 'Yup, suid' || echo 'Nope, not suid' Yup, suid $ [ -u sgid_dir ] && echo 'Yup, suid' || echo 'Nope, not suid' Nope, not suid $ [ -g sgid_file ] && echo 'Yup, sgid' || echo 'Nope, not sgid' Yup, sgid $ [ -g suid_file ] && echo 'Yup, sgid' || echo 'Nope, not sgid' Nope, not sgid
- man chmod
- , "Running As a Non-root User"
- , "Using sudo More Securely"
- , "Using Passwords in Scripts"
- , "Using sudo on a Group of Commands"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Restricting Guest Users
- InhaltsvorschauThe material concerning the restricted shell in this recipe also appears in Learning the bash Shell by Cameron New man (O'Reilly).You need to allow some guest users on your system and need to restrict what they can do.Avoid using shared accounts if possible, since you lose accountability and create logistical headaches when users leave and you need to change the password and inform the other users. Create separate accounts with the least possible permissions necessary to do whatever is needed. Consider using:
- A chroot jail, as discussed in , "Using chroot Jails"
- SSH to allow non-interactive access to commands or resources, as discussed in , "Using SSH Without a Password"
- bash's restricted shell
The restricted shell is designed to put the user into an environment where their ability to move around and write files is severely limited. It's usually used for guest accounts. You can make a user's login shell restricted by putting rbash in the user's /etc/passwd entry if this option was included when bash was compiled.The specific constraints imposed by the restricted shell disallow the user from doing the following:- Changing working directories: cd is inoperative. If you try to use it, you will get the error message from bash
cd:restricted. - Redirecting output to a file: the redirectors >, >|, <>, and >> are not allowed.
- Assigning a new value to the environment variables
$ENV,$BASH_ENV,$SHELL, or$PATH. - Specifying any commands with slashes (/) in them. The shell will treat files outside of the current directory as "not found."
- Using the exec built-in.
- Specifying a filename containing a / as an argument to the .(source) built-in command.
- Importing function definitions from the shell environment at startup.
- Adding or deleting built-in commands with the
-fand-doptions to the enable built-in command. - Specifying the
-poption to the command built-in command. - Turning off restricted mode with set+r.
These restrictions go into effect after the user's .bash_profile and environment files are run. In addition, it is wise to change the owner of the users' .bash_profile and .bashrcEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using chroot Jails
- InhaltsvorschauYou have to use a script or application that you don't trust.Consider placing it in a so-called chroot jail. The chroot command changes the root directory of the current process to the directory you specify, then returns a shell or exec's a given command. That has the effect of placing the process, and thus the program, into a jail from which it theoretically can't escape to the parent directory. So if that application is compromised or otherwise does something malicious, it can only affect the small portion of the file system you restricted it to. In conjunction with running as a user with very limited rights, this is a very useful layer of security to add.Unfortunately, covering all the details of chroot is beyond the scope of this recipe, since it would probably require a whole separate book. We present it here to promote awareness of the functionality.So why doesn't everything run in chroot jails? Because many applications need to interact with other applications, files, directories, or sockets all over the file system. That's the tricky part about using chroot jails; the application can't see outside of its walls, so everything it needs must be inside those walls. The more complicated the application, the more difficult it is to run in a jail.Some applications that must inherently be exposed to the Internet, such as DNS (e.g., BIND), web, and mail (e.g., Postfix) servers, may be configured to run in chroot jails with varying degrees of difficulty. See the documentation for the distribution and specific applications you are running for details.Another interesting use of chroot is during system recovery. Once you have booted from a Live CD and mounted the root file system on your hard drive, you may need to run a tool such as Lilo or Grub which, depending on your configuration, might need to believe it's really running onto the damaged system. If the Live CD and the installed system are not too different, you can usually chroot into the mount point of the damaged system and fix it. That works because all the tools, libraries, configuration files, and devices already exist in the jail, since they really are a complete (if not quite working) system. You might have to experiment with yourEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Running As a Non-root User
- InhaltsvorschauYou'd like to run your scripts as a non-root user, but are afraid you won't be able to do the things you need to do.Run your scripts under non-root user IDs, either as you or as dedicated users, and run interactively as non-root, but configure sudo to handle any tasks that require elevated privileges.sudo may be used in a script as easily as it may be used interactively. See the sudoers
NOPASSWDoption especially. See , "Using sudo More Securely."- man sudo
- man sudoers
- , "Writing setuid or setgid Scripts"
- , "Using sudo More Securely"
- , "Using Passwords in Scripts"
- , "Using sudo on a Group of Commands"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using sudo More Securely
- InhaltsvorschauYou want to use sudo but are worried about granting too many people too many privileges.Good! You should be worrying about security. While using sudo is much more secure than not using it, the default settings may be greatly improved.Take the time to learn a bit about sudo itself and the /etc/sudoers file. In particular, learn that in most cases you should not be using the
ALL=(ALL) ALLspecification! Yes, that will work, but it's not even remotely secure. The only difference between that and just giving everyone the root password is that they don't know the root password. They can still do everything root can do. sudo logs the commands it runs, but that's trivial to avoid by usingsudo bash.Second, give your needs some serious thought. Just as you shouldn't be using theALL=(ALL) ALLspecification, you probably shouldn't be managing users one by one either. The sudoers utility allows for very granular management and we strongly recommend using it.man sudoersprovides a wealth of material and examples, especially the section on preventing shell escapes.sudoers allows for four kinds of aliases: user, runas, host, and command. Judicious use of them as roles or groups will significantly reduce the maintenance burden. For instance, you can set up aUser_Aliasfor BUILD_USERS, then define the machines those users need to run on withHost_Aliasand the commands they need to run withCmnd_Alias. If you set a policy to only edit /etc/sudoers on one machine and copy it around to all relevant machines periodically using scp with public-key authentication, you can set up a very secure yet usable system of least privilege.When sudo asks for your password, it's really asking for your password. As in, your user account. Not root. For some reason people often get confused by this at first.Unfortunately, sudo is not installed by default on every system. It is usually installed on Linux and OpenBSD; other systems will vary. You should consult your system documentation and install it if it's not already there.You should always use visudo to edit your /etc/sudoers file. Like vipwEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Passwords in Scripts
- InhaltsvorschauYou need to hardcode a password in a script.This is obviously a bad idea and should be avoided whenever possible. Unfortunately, sometimes it isn't possible to avoid it.The first way to try to avoid doing this is to see if you can use sudo with the
NOPASSWDoption to avoid having to hardcode a password anywhere. This obviously has its own risks, but is worth checking out. See , "Using sudo More Securely" for more details.Another alternative may be to use SSH with public keys and ideally restricted commands. See , "Using SSH Without a Password."If there is no other way around it, about the best you can do is put the user ID and password in a separate file that is readable only by the user who needs it, then source that file when necessary (, "Using Configuration Files in a Script"). Leave that file out of revision control, of course.Accessing data on remote machines in a secure manner is relatively easy using SSH (see , "Using SSH Without a Password" and , "Getting Input from Another Machine"). It may even be possible to use that SSH method to access other data on the same host, but it's probably much more efficient to use sudo for that. But what about accessing data in a remote database, perhaps using some SQL command? There is not much you can do in that case.Yes, you say, but what about crypt or the other password hashes? The problem is that the secure methods for storing passwords all involve using what's known as a one-way hash. The password checks in, but it can't check out. In other words, given the hash, there is theoretically no way to get the plain-text password back out. And that plain-text password is the point—we need it to access our database or whatever. So secure storage is out.That leaves insecure storage, but the problem here is that it may actually be worse than plain text because it might give you a false sense of security. If it really makes you feel better, and you promise not to get a false sense of security, go ahead and use ROT13 or something to obfuscate the password. ROT13 only handles ASCII letters, so you could also use ROT47 to handle some punctuation as well.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using SSH Without a Password
- InhaltsvorschauYou need to use SSH or scp in a script and would like to do so without using a password. Or you're using them in a cron job and can't have a password.SSH1 (the protocol) and SSH1 (the executables) are deprecated and considered less secure than the newer SSH2 protocol as implemented by OpenSSH and SSH Communications Security. We strongly recommend using SSH2 with OpenSSH and will not cover SSH1 here.There are two ways to use SSH without a password, the wrong way and the right way. The wrong way is to use a public-key that is not encrypted by a passphrase. The right way is to use a passphrase protected public-key with ssh-agent or keychain.We assume you are using OpenSSH; if not, consult your documentation (the commands and files will be similar).First, you need to create a key pair if you don't already have one. Only one key pair is necessary to authenticate you to as many machines as you configure, but you may decide to use more than one key pair, perhaps for personal and work reasons. The pair consists of a private key that you should protect at all costs, and a public key (*.pub) that you can post on a billboard if you like. The two are related in a complex mathematical way such that they can identify each other, but you can't derive one from the other.Use ssh-keygen (might be ssh-keygen2 if you're not using OpenSSH) to create a key pair.
-tis mandatory and its arguments arersaordsa.-bis optional and specifies the number of bits in the new key (1024 is the default at the time of this writing).-Callows you to specify a comment, but it defaults to user@hostname if you omit it. We recommend at least using-t dsa -b 2048and we recommend strongly against using no passphrase. ssh-keygen also allows you to change your key file's passphrase or comment.$ ssh-keygen You must specify a key type (-t). Usage: ssh-keygen [options] Options: -b bits Number of bits in the key to create. -c Change comment in private and public key files. -e Convert OpenSSH to IETF SECSH key file. -f filename Filename of the key file. -g Use generic DNS resource record format. -i Convert IETF SECSH to OpenSSH key file. -l Show fingerprint of key file. -p Change passphrase of private key file. -q Quiet. -y Read private key file and print public key. -t type Specify type of key to create. -B Show bubblebabble digest of key file. -H Hash names in known_hosts file -F hostname Find hostname in known hosts file -C comment Provide new comment. -N phrase Provide new passphrase. -P phrase Provide old passphrase. -r hostname Print DNS resource record. -G file Generate candidates for DH-GEX moduli -T file Screen candidates for DH-GEX moduli $ ssh-keygen -t dsa -b 2048 -C 'This is my new key' Generating public/private dsa key pair. Enter file in which to save the key (/home/jp/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/jp/.ssh/id_dsa. Your public key has been saved in /home/jp/.ssh/id_dsa.pub. The key fingerprint is: 84:6f:45:fc:08:3b:ce:b2:4f:2e:f3:5e:b6:9f:65:63 This is my new key $ ls -l id_dsa* -rw------- 1 jp jp 1264 Dec 13 23:39 id_dsa -rw-r--r-- 1 jp jp 1120 Dec 13 23:39 id_dsa.pub $ cat id_dsa.pub ssh-dss AAAAB3NzaC1kc3MAAAEBANpgvvTslst2m0ZJA0ayhh1Mqa3aWwU3kfv0m9+myFZ9veFsxM7IVxIjWfAlQh3jp lY+Q78fMzCTiG+ZrGZYn8adZ9yg5/ wAC03KXm2vKt8LfTx6I+qkMR7v15NI7tZyhxGah5qHNehReFWLuk7JXCtRrzRvWMdsHc/ L2SA1Y4fJ9Y9FfVlBdE1Er+ZIuc5xIlO6D1HFjKjt3wjbAal+oJxwZJaupZ0Q7N47uwMslmc5ELQBRNDsaoqF RKlerZASPQ5P+AH/+Cxa/fCGYwsogXSJJ0H5S7+QJJHFze35YZI/ +A1D3BIa4JBf1KvtoaFr5bMdhVAkChdAdMjo96xhbdEAAAAVAJSKzCEsrUo3KAvyUO8KVD6e0B/NAAAA/3u/ Ax2TIB/M9MmPqjeH67Mh5Y5NaVWuMqwebDIXuvKQQDMUU4EPjRGmS89Hl8UKAN0Cq/C1T+OGzn4zrbE06CO/ Sm3SRMP24HyIbElhlWV49sfLR05Qmh9fRl1s7ZdcUrxkDkr2J6on5cMVB9M2nIl90IhRVLd5RxP01u81yqvhv E61ORdA6IMjzXcQ8ebuD2R733O37oGFD7e2O7DaabKKkHZIduL/zFbQkzMDK6uAMP8ylRJN0fUsqIhHhtc// 16OT2H6nMU09MccxZTFUfqF8xIOndElP6um4jXYk5Q30i/CtU3TZyvNeWVwyGwDi4wg2jeVe0YHU2Rh/ ZcZpwAAAQEAv2O86701U9sIuRijp8sO4h13eZrsE5rdn6aul/mkm+xAlO+WQeDXR/ ONm9BwVSrNEmIJB74tEJL3qQTMEFoCoN9Kp00Ya7Qt8n4gZ0vcZlI5u+cgyd1mKaggS2SnoorsRlb2Lh/ Hpe6mXus8pUTf5QT8apgXM3TgFsLDT+3rCt40IdGCZLaP+UDBuNUSKfFwCru6uGoXEwxaL08Nv1wZOc19qrc0 Yzp7i33m6i3a0Z9Pu+TPHqYC74QmBbWq8U9DAo+7yhRIhq/ fdJzk3vIKSLbCxg4PbMwx2Qfh4dLk+L7wOasKnl5//W+RWBUrOlaZ1ZP1/azsK0Ncygno/0F1ew== This is my new key
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Restricting SSH Commands
- InhaltsvorschauYou'd like to restrict what an incoming SSH user or script can do.Edit the ~/.ssh/authorized_keys file, use SSH forced commands, and optionally disable unnecessary SSH features. For example, suppose you want to allow an rsync process without also allowing interactive use.First, you need to figure out exactly what command is being run on the remote side. Create a key (, "Using SSH Without a Password") and add a forced command to tell you. Edit the ~/.ssh/authorized_keys file and add:
command="/bin/echo Command was: $SSH_ORIGINAL_COMMAND"
before the key. It will look something like this, all on one line:command="/bin/echo Command was: $SSH_ORIGINAL_COMMAND" ssh-dss AAAAB3NzaC1kc3MAAAEBANpgvvTslst2m0ZJA0ayhh1Mqa3aWwU3kfv0m9+myFZ9veFsxM7IVxIjWfAlQh3jp lY+Q78fMzCTiG+ZrGZYn8adZ9yg5/ wAC03KXm2vKt8LfTx6I+qkMR7v15NI7tZyhxGah5qHNehReFWLuk7JXCtRrzRvWMdsHc/ L2SA1Y4fJ9Y9FfVlBdE1Er+ZIuc5xIlO6D1HFjKjt3wjbAal+oJxwZJaupZ0Q7N47uwMslmc5ELQBRNDsaoqF RKlerZASPQ5P+AH/+Cxa/fCGYwsogXSJJ0H5S7+QJJHFze35YZI/ +A1D3BIa4JBf1KvtoaFr5bMdhVAkChdAdMjo96xhbdEAAAAVAJSKzCEsrUo3KAvyUO8KVD6e0B/NAAAA/3u/ Ax2TIB/M9MmPqjeH67Mh5Y5NaVWuMqwebDIXuvKQQDMUU4EPjRGmS89Hl8UKAN0Cq/C1T+OGzn4zrbE06CO/ Sm3SRMP24HyIbElhlWV49sfLR05Qmh9fRl1s7ZdcUrxkDkr2J6on5cMVB9M2nIl90IhRVLd5RxP01u81yqvhv E61ORdA6IMjzXcQ8ebuD2R733O37oGFD7e2O7DaabKKkHZIduL/zFbQkzMDK6uAMP8ylRJN0fUsqIhHhtc// 16OT2H6nMU09MccxZTFUfqF8xIOndElP6um4jXYk5Q30i/CtU3TZyvNeWVwyGwDi4wg2jeVe0YHU2Rh/ ZcZpwAAAQEAv2O86701U9sIuRijp8sO4h13eZrsE5rdn6aul/mkm+xAlO+WQeDXR/ ONm9BwVSrNEmIJB74tEJL3qQTMEFoCoN9Kp00Ya7Qt8n4gZ0vcZlI5u+cgyd1mKaggS2SnoorsRlb2Lh/ Hpe6mXus8pUTf5QT8apgXM3TgFsLDT+3rCt40IdGCZLaP+UDBuNUSKfFwCru6uGoXEwxaL08Nv1wZOc19qrc0 Yzp7i33m6i3a0Z9Pu+TPHqYC74QmBbWq8U9DAo+7yhRIhq/ fdJzk3vIKSLbCxg4PbMwx2Qfh4dLk+L7wOasKnl5//W+RWBUrOlaZ1ZP1/azsK0Ncygno/0F1ew== This is my new key
Now execute your command and see what the result is.$ ssh remote_host 'ls -l /etc' Command was: ls -l /etc
Now, the problem with this approach is that it will break a program like rsync that depends on having the STDOUT/STDIN channel all to itself.$ rsync -avzL -e ssh remote_host:/etc . protocol version mismatch -- is your shell clean? (see the rsync man page for an explanation) rsync error: protocol incompatibility (code 2) at compat.c(64)
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Disconnecting Inactive Sessions
- InhaltsvorschauYou'd like to be able to automatically log out inactive users, especially root.Set the
$TMOUTenvironment variable in /etc/bashrc or ~/.bashrc to the number of seconds of inactivity before ending the session. In interactive mode, once a prompt is issued, if the user does not enter a command in$TMOUTseconds, bash will exit.$TMOUTis also used in the read built-in and theselectcommand in scripts.Don't forget to set this as a read-only variable in a system-level file such as /etc/profile or /etc/bashrc to which users have no write access if you don't want them to be able to change it.declare -r TMOUT=3600 # Or: readonly TMOUT=3600
Since the user has control over their own environment, you cannot totally rely on$TMOUT, even if you set it as read-only, since the user could just run a different shell. Think of it as a helpful reminder to cooperative users, especially knowledgeable and interrupt-driven system administrators who may get distracted (constantly).- , "Creating Self-Contained, Portable RC Files"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 15: Advanced Scripting
- InhaltsvorschauUnix and POSIX have long promised compatibility and portability, and long struggled to deliver it; thus, one of the biggest problems for advanced scripters is writing scripts that are portable, i.e., that can work on any machine that has bash installed. Writing scripts that run well on a wide variety of platforms is much more difficult than we wish it were. There are many variations from one system to another that can get in the way; for example, bash itself isn't always installed in the same place, and many common Unix commands have slightly different options (or give slightly different output) depending on the operating system. In this chapter, we'll look at several of those problems, and show you how to solve them.Many of other things that are periodically needed are not as simple as we'd like them to be, either. So, we'll also cover solutions for additional advanced scripting tasks, such as automating processes using phases, sending email from your script, logging to syslog, using your network resources, and a few tricks for getting input and redirecting output.Although this chapter is about advanced scripting, we'd like to stress the need for clear code, written as simply as possible, and documented. Brian Kernighan, one of the first Unix developers, put it well:Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.It's easy to write very clever shell scripts that are very difficult, if not impossible, to understand. The more clever you think you're being now, as you solve the problem de jour, the more you'll regret it 6, 12, or 18 months from now when you (or worse yet, someone else) have to figure out what you did and why it broke. If you have to be clever, at least document how the script works (see , "Documenting Your Script")!You need to run a bash script on several machines, but bash is not always in the same place. See , "Getting bash for xBSD."Use the /usr/bin/env command in the shebang line, as in
#!/usr/bin/env bash. If your system doesn't haveEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding bash Portably for #!
- InhaltsvorschauYou need to run a bash script on several machines, but bash is not always in the same place. See , "Getting bash for xBSD."Use the /usr/bin/env command in the shebang line, as in
#!/usr/bin/env bash. If your system doesn't have env in /usr/bin, ask your system administrator to install it, move it, or create a symbolic link because this is the required location. For example, Red Hat inexplicably uses /bin/env, but they at least create a symlink to the correct location.You could also create symbolic links for bash itself, but using env is the canonical and correct solution.env's purpose is to "run a program in a modified environment," but since it will search the path for the command it is given to run, it works very well for this use.You may be tempted to use#!/bin/shinstead. Don't. If you are using bash-specific features in your script, they will not work on machines that do not use bash in Bourne shell mode for /bin/sh (e.g., BSD, Solaris, Ubuntu 6.10+). And even if you aren't using bash-specific features now, you may forget about that in the future. If you are committed to using only POSIX features, by all means use#!/bin/sh(and don't develop on Linux, see , "Developing Portable Shell Scripts"), but otherwise be specific.You may sometimes see a space between#!and/bin/whatever. Historically there were some systems that required the space, though in practice we haven't seen one in a long time. It's very unlikely any system running bash will require the space, and the lack of the space seems to be the most common usage now. But for the utmost historical compatibility, use the space.We have chosen to use#!/usr/bin/env bashin the longer scripts and functions we've made available to download (see the end of the Preface for details), because that will run unchanged on most systems. However, since env uses the$PATHto find bash, this is arguably a security issue (see , "Avoiding Interpreter Spoofing"), albeit a minor one in our opinion.Ironically, since we're trying to use env for portability, shebang line processing is not consistent across systems. Many, including Linux, allow only a single argument to the interpreter. ThusEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting a POSIX $PATH
- InhaltsvorschauYou are on a machine that provides older or proprietary tools (e.g., Solaris) and you need to set your PATH so that you get POSIX-compliant tools.Use the getconf utility:
PATH=$(PATH=/bin:/usr/bin getconf PATH)
Here are some default and POSIX paths on several systems:# Red Hat Enterprise Linux (RHEL) 4.3 $ echo $PATH /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/home/$USER/bin $ getconf PATH /bin:/usr/bin # Debian Sarge $ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games $ getconf PATH /bin:/usr/bin # Solaris 10 $ echo $PATH /usr/bin: $ getconf PATH /usr/xpg4/bin:/usr/ccs/bin:/usr/bin:/opt/SUNWspro/bin # OpenBSD 3.7 $ echo $PATH /home/$USER/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/ local/sbin:/usr/games $ getconf PATH /usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin
getconf reports various system configuration variables, so you can use it to set a default path. However, unless getconf itself is a built-in, you will need a minimal path to find it, hence thePATH=/bin:/usr/binpart of the solution.In theory, the variable you use should beCS_PATH. In practice,PATHworked every-where we tested whileCS_PATHfailed on the BSDs.- , "Finding a File Using a List of Possible Locations"
- , "Setting a Secure $PATH"
- , "Finding World-Writable Directories in Your $PATH"
- , "Adding the Current Directory to the $PATH"
- , "Change Your $PATH Permanently"
- , "Change Your $PATH Temporarily"
- , "Forgetting That the Current Directory Is Not in the $PATH"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Developing Portable Shell Scripts
- InhaltsvorschauYou are writing a shell script that will need to run on multiple versions of multiple Unix or POSIX operating systems.First, try using the command built-in with its
-poption to find the POSIX version of program, e.g., in /usr/xpg4 or /usr/xpg6 on Solaris:$ command -p program argsThen, if possible, find the oldest or least capable Unix machine and develop the script on that platform. If you aren't sure what the least capable platform is, use a BSD variant or Solaris (and the older a version you can find, the better).command -puses a default path that is guaranteed to find all of the POSIX-standard utilities. If you're sure your script will only ever run on Linux (famous last words), then don't worry about it;otherwise, avoid developing cross-platform scripts on Linux or Windows (e.g., via Cygwin).The problems with writing cross-platform shell scripts on Linux are:- /bin/sh is not the Bourne shell, it's really /bin/bash in Bourne mode, except when it's /bin/dash (for example Ubuntu 6.10). Both are very good, but not perfect, and none of the three work exactly the same, which can be very confusing. In particular, the behavior of echo can change.
- Linux uses the GNU tools instead of the original Unix tools.
Don't get us wrong, we love Linux and use it every day. But it isn't really Unix: it does some things differently, and it has the GNU tools. The GNU tools are great, and that's the problem. They have a lot of switches and features that aren't present on other platforms, and your script will break in odd ways no matter how careful you are about that. Conversely, Linux is so compatible with everything that scripts written for any other Unix-like systems will almost always run on it. They may not be perfect (e.g., echo's default behavior is to display\ninstead of printing a newline), but are often good enough.There is an ironic Catch-22 here— the more shell features you use, the less you have to depend on external programs that may or may not be there or work as expected. While bash is far more capable than sh, it's also one of the tools that may or may not be there. Some form ofEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing Scripts in VMware
- InhaltsvorschauYou need to develop cross-platform scripts but do not have the appropriate systems or hardware.If the target platforms run on the x86 architecture, download the free VMware Server and build your own. Or search for prebuilt virtual machines on the VMware site, the OS vendor or distributor's site, or the Internet.The flaw in this solution is the systems such as AIX and HP-UX that don't run on an x86 architecture, and thus don't run under VMware. Again, if you have these systems, use them. If not, see the recipe , "Getting bash Without Getting bash."Testing shell scripts is usually not very resource intensive, so even moderate hard-ware capable of running VMware or a similar virtualization package should be fine. We mention VMware specifically because the Server and Player products are without cost, they run on Linux and Windows (and soon the Mac), and are very easy to use; but there are certainly other alternatives available.If you install VMware Server on a Linux server, you don't even need the overhead of a GUI on the host machine—you can use the VNC-based VMware Console from another Linux or Windows machine with a GUI. Minimal virtual machines with 128 MB of RAM, or sometimes even less, should be more than enough for a shell environment for testing. Set up an NFS share to store your test scripts and data, and then simply telnet or ideally SSH to the test system.To get you started, here's a trivial example using VMware player:
- Get the free VMware Player for Windows or Linux from http://www.vmware.com/player/.
- Get a pre-built virtual machine image:
- Ubuntu Linux 5.10 (Debian derivative), Firefox 1.0.7, and Gnome 2.12.1 form the basis for VMware's "Browser Appliance v1.0.0" (258M at http://www.vmware.com/vmtn/appliances/directory/browserapp.html).
- PC-BSD is a BSD and KDE-based desktop distribution (609M at http://www.pcbsd.org/?p=download#vmware).
- Unzip whichever one you selected and open it in Player, creating a new VMware UUID if prompted.
Once you boot, which takes a while, you will have either an Ubuntu 5.10 Gnome-based desktop with bash 3.0 or a BSD and KDE-based GUI desktop complete withEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using for Loops Portably
- InhaltsvorschauYou need to do a for loop but want it to work on older versions of bash.This method is portable back to bash-2.04+:
$ for ((i=0; i<10; i++)); do echo $i; done 0 1 2 3 4 5 6 7 8 9
There are nicer ways of writing this loop in newer versions of bash, but they are not backwards compatible. As of bash-3.0+ you can use the syntaxfor {x..y}, as in:$ for i in {1..10}; do echo $i; done 1 2 3 4 5 6 7 8 9 10If your system has the seq command, you could also do this:$ for i in $(seq 1 10); do echo $i; done 1 2 3 4 5 6 7 8 9 10
- help for
- man seq
- , "Looping with a Count"
- , "Looping with Floating-Point Values"
- , "Writing Sequences"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using echo Portably
- InhaltsvorschauYou are writing a script that will run on multiple versions of Unix and Linux and you need echo to behave consistently even if it is not running on bash.Use
printf "%b"whatever, or test for the system and setxpg_echousingshopt -s xpg_echoas needed.If you omit the"%b"format string (for example,printfwhatever), then printf will try to interpret any % characters in whatever, which is probably not what you want. The"%b"format is an addition to the standard printf format that will prevent that misinterpretation and also expand backslash escape sequences in whatever.Settingxpg_echois less consistent since it only works on bash. It can be effective if you are sure that you'll only every run under bash, and not under sh or another similar shell that doesn't usexpg_echo.Using printf requires changes to how you writeechostatements, but it's defined by POSIX and should be consistent across any POSIX shell anywhere. Specifically, you have to writeprintf "%b"instead of justecho.If you automatically type$binstead of%byou will be unhappy because that will print a blank line, since you have specified a null format. That is unless$bis actually defined, in which case the results depend on the value of$b. Either way, this can be a very difficult bug to find since$band%blook very similar:$ printf "%b" "Works" Works $ printf "$b" "Broken" $
In some shells, built-in echo behaves differently than the external echo used on other systems. This is not always obvious when running on Linux since /bin/sh is actually bash (usually;it could also be dash on Ubuntu 6.10+), and there are similar circumstances on some BSDs. The difference is in how echo does or does not expand back-slash-escape sequences. Shell built-in versions tend not to expand, while external versions (e.g., /bin/echo and /usr/bin/echo) tend to expand;but again, that can change from system to system.Typical Linux (/bin/bash):$ type -a echo echo is a shell builtin echo is /bin/echo $ builtin echo "one\ttwo\nthree" one\ttwo\nthree\n $ /bin/echo "one\ttwo\nthree" one\ttwo\nthree\n $ echo -e "one\ttwo\nthree" one → two three $ /bin/echo -e "one\ttwo\nthree" one → two three
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Splitting Output Only When Necessary
- InhaltsvorschauYou want to split output only if the input exceeds your limit, but the split command always creates at least one new file.
# cookbook filename: func_split #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # Output fixed-size pieces of input ONLY if the limit is exceeded # Called like: Split <file> <prefix> <limit option> <limit argument> # e.g. Split $output ${output}_ --lines 100 # See split(1) and wc(1) for option details function Split { local file=$1 local prefix=$2 local limit_type=$3 local limit_size=$4 local wc_option # Sanity Checks if [ -z "$file" ]; then printf "%b" "Split: requires a file name!\n" return 1 fi if [ -z "$prefix" ]; then printf "%b" "Split: requires an output file prefix!\n" return 1 fi if [ -z "$limit_type" ]; then printf "%b" "Split: requires a limit option (e.g. --lines), see 'man split'!\ n" return 1 fi if [ -z "$limit_size" ]; then printf "%b" "Split: requires a limit size (e.g. 100), see 'man split'!\n" return 1 fi # Convert split options to wc options. Sigh. # Not all options supported by all wc/split on all systems case $limit_type in -b|--bytes) wc_option='-c';; -C|--line-bytes) wc_option='-L';; -l|--lines) wc_option='-l';; esac # If whatever limit is exceeded if [ "$(wc $wc_option $file | awk '{print $1}')" -gt $limit_size ]; then # actually do something split --verbose $limit_type $limit_size $file $prefix fi } # end of function SplitDepending on your system, some options (e.g.,-C) may not be available in split or wc.- , "Counting Lines, Words, or Characters in a File"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Viewing Output in Hex
- InhaltsvorschauYou need to see output in hex mode to verify that a certain whitespace or unprintable character is as expected.Pipe the output though hexdump using the
-Coption for canonical output:$ hexdump -C filename 00000000 4c 69 6e 65 20 31 0a 4c 69 6e 65 20 32 0a 0a 4c |Line 1.Line 2..L| 00000010 69 6e 65 20 34 0a 4c 69 6e 65 20 35 0a 0a |ine 4.Line 5..| 0000001e
For example, nl uses spaces (ASCII 20), then the line number, then a tab (ASCII 09) in its output:$ nl -ba filename | hexdump -C 00000000 20 20 20 20 20 31 09 4c 69 6e 65 20 31 0a 20 20 | 1.Line 1. | 00000010 20 20 20 32 09 4c 69 6e 65 20 32 0a 20 20 20 20 | 2.Line 2. | 00000020 20 33 09 0a 20 20 20 20 20 34 09 4c 69 6e 65 20 | 3.. 4.Line | 00000030 34 0a 20 20 20 20 20 35 09 4c 69 6e 65 20 35 0a |4. 5.Line 5.| 00000040 20 20 20 20 20 36 09 0a | 6..| 00000048
hexdump is a BSD utility that also comes with many Linux distributions. Other systems, notably Solaris, do not have it by default. You can use the octal dump command od, but it's a lot harder to read:$ nl -ba filename | od -x 0000000 2020 2020 3120 4c09 6e69 2065 0a31 2020 0000020 2020 3220 4c09 6e69 2065 0a32 2020 2020 0000040 3320 0a09 2020 2020 3420 4c09 6e69 2065 0000060 0a34 2020 2020 3520 4c09 6e69 2065 0a35 0000100 2020 2020 3620 0a09 0000110 $ nl -ba filename | od -tx1 0000000 20 20 20 20 20 31 09 4c 69 6e 65 20 31 0a 20 20 0000020 20 20 20 32 09 4c 69 6e 65 20 32 0a 20 20 20 20 0000040 20 33 09 0a 20 20 20 20 20 34 09 4c 69 6e 65 20 0000060 34 0a 20 20 20 20 20 35 09 4c 69 6e 65 20 35 0a 0000100 20 20 20 20 20 36 09 0a 0000110
There is also a simple Perl script available at http://www.khngai.com/perl/bin/hexdump.txt that might work:$ ./hexdump.pl filename /0 /1 /2 /3 /4 /5 /6 /7 /8 /9/ A /B /C /D /E /F 0123456789ABCDEF 0000 : 4C 69 6E 65 20 31 0A 4C 69 6E 65 20 32 0A 0A 4C Line 1.Line 2..L 0010 : 69 6E 65 20 34 0A 4C 69 6E 65 20 35 0A 0A ine 4.Line 5..
- man hexdump
- man od
- "Table of ASCII Values" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using bash Net-Redirection
- InhaltsvorschauYou need to send or receive very simple network traffic but you do not have a tool such as netcat installed.If you have bash version 2.04+ compiled with
--enable-net-redirections(it isn't compiled this way in Debian and derivatives), you can use bash itself. The following example is also used in , "Finding My IP Address":$ exec 3<> /dev/tcp/www.ippages.com/80 $ echo -e "GET /simple/?se=1 HTTP/1.0\n" >&3 $ cat <&3 HTTP/1.1 200 OK Date: Tue, 28 Nov 2006 08:13:08 GMT Server: Apache/2.0.52 (Red Hat) X-Powered-By: PHP/4.3.9 Set-Cookie: smipcomID=6670614; expires=Sun, 27-Nov-2011 08:13:09 GMT; path=/ Pragma: no-cache Cache-Control: no-cache, must-revalidate Content-Length: 125 Connection: close Content-Type: text/plain; charset=ISO-8859-1 72.NN.NN.225 (US-United States) http://www..com Tue, 28 Nov 2006 08:13:09 UTC/GMT flagged User Agent - reduced functionality
As noted, this recipe will probably not work under Debian and derivatives such as Ubuntu since they expressly do not compile bash with--enable-net-redirections.As noted in , "Redirecting Output for the Life of a Script," it is possible to use exec to permanently redirect file handles within the current shell session, so the first command sets up input and output on file handle3. The second line sends a trivial command to a path on the web server defined in the first command. Note that the user agent will appear as "-" on the web server side, which is what is causing the "flagged User Agent" warning. The third command simply displays the results.Both TCP and UDP are supported. Here is a trivial way to send syslog messages to a remote server (although in production we recommend using the logger utility, which is much more user friendly and robust):echo "<133>$0[$$]: Test syslog message from bash" > /dev/udp/loghost.example.com/514
Since UDP is connectionless, this is actually much easier to use than the previous TCP example.<133>is the syslog priority value for local0.notice, calculated according to RFC 3164. See the RFC "4.1.1 PRI Part" and logger manpage for details.$0is the name and$$Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding My IP Address
- InhaltsvorschauYou need to know the IP address of the machine you are running on.There is no good way to do this that will work on all systems in all situations, so we will present several possible solutions.First, you can parse output from ifconfig to look for IP addresses. These examples will either return the first IP address that is not a loopback or nothing if there are no interfaces configured or up.
# cookbook filename: finding_ipas # IPv4 Using awk, cut and head $ /sbin/ifconfig -a | awk '/(cast)/ { print $2 }' | cut -d':' -f2 | head -1 # IPv4 Using Perl, just for fun $ /sbin/ifconfig -a | perl -ne 'if ( m/^\s*inet (?:addr:)?([\d.]+).*?cast/ ) { print qq($1\n); exit 0; }' # IPv6 Using awk, cut and head $ /sbin/ifconfig -a | egrep 'inet6 addr: |address: ' | cut -d':' -f2-| cut -d'/' -f1 | head -1 | tr -d ' ' # IPv6 Using Perl, just for fun $ /sbin/ifconfig -a | perl -ne 'if ( m/^\s*(?:inet6)? \s*addr(?:ess)?: ([0-9A-Fa-f: ]+)/ ) { print qq($1\n); exit 0; }'Second, you can get your hostname and resolve back to an IP address. This is often unreliable because today's systems (especially workstations) might have incomplete or incorrect hostnames and/or might be on a dynamic network that lacks proper reverse lookup. Use at your own risk and test well.$ host $(hostname)
Third, you may be more interested in your host's external, routable address than its internal RFC 1918 address. In that case you can use an external host such as http://www.ippages.com/ or "FollowMeIP" (see below) to learn the address of your firewall or NAT device. The catch here is that non-Linux systems often have no command-line tool like wget installed by default. lynx or curl will also work, but they aren't usually installed by default either (although Mac OS X 10.4 has curl). Note the IP address is deliberately obscured in the following examples:$ wget -qO - http://www.ippages.com/simple/ 72.NN.NN.225 (US-United States) http://www.ippages.com Mon, 27 Nov 2006 21:02:23 UTC/ GMT (5 of 199 allowed today) alternate access in XML format at: http://www.ippages.com/xml alternate access via SOAP at: http://www.ippages.com/soap/server.php alternate access via RSS feed at: http://www.ippages.com/rss.php alternate access in VoiceXML format at: http://www.ippages.com/voicexml $ wget -qO - http://www.ippages.com/simple/?se=1 72.NN.NN.225 (US-United States) http://www.ippages.com Tue, 28 Nov 2006 08:11:36 UTC/ GMT $ wget -qO - http://www.ippages.com/simple/?se=1 | cut -d' ' -f1 72.NN.NN.225 $ lynx -dump http://www.ippages.com/simple/?se=1 | cut -d' ' -f1 72.NN.NN.225 $ curl -s http://www.ippages.com/simple/?se=1 | cut -d' ' -f1 72.NN.NN.225
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Input from Another Machine
- InhaltsvorschauYour script needs to get input from another machine, perhaps to check if a file exists or a process is running.Use SSH with public keys and command substitution. To do this, set up SSH so that you do not need a password, as described in , "Using SSH Without a Password." Next, tailor the command that SSH runs to output exactly what your script needs as input. Then simply use command substitution.
#!/usr/bin/env bash # cookbook filename: command_substitution REMOTE_HOST='host.example.com' # Required REMOTE_FILE='/etc/passwd' # Required SSH_USER='user@' # Optional, set to '' to not use# SSH_ID='-i ~/.ssh/foo.id' # Optional, set to '' to not use SSH_ID='' result=$( ssh $SSH_ID $SSH_USER$REMOTE_HOST \ "[ -r $REMOTE_FILE ] && echo 1 || echo 0" ) || { echo "SSH command failed!" >&2; exit 1; } if [ $result = 1 ]; then echo "$REMOTE_FILE present on $REMOTE_HOST" else echo "$REMOTE_FILE not present on $REMOTE_HOST" fiWe do a few interesting things here. First, notice how both$SSH_USERand$SSH_IDwork. They have an effect when they have a value, but when they are empty they interpolate to the empty set and are ignored. This allows us to abstract the values in the code, which lends itself to putting those values in a configuration file, putting the code into a function, or both.# Interpolated line of the variables have values: ssh -i ~/.ssh/foo.id user@host.example.com [...] # No values: ssh host.example.com [...]
Next, we set up the command that SSH runs so that there is always output (0 or 1), then check that$resultis not empty. That's one way to make sure that the SSH command runs (see also , "Telling If a Command Succeeded or Not"). If$resultis empty, we group commands using a {} code block to issue an error message and exit. But since we're always getting output from the SSH command, we have to test the value; we can't just useif[$result]; then.If we didn't use the code block, we'd only issue the warning if the SSH command returned an empty$result, but we'd always exit. Read the code again until you understand why, because this is an easy way to get bitten. Likewise, if we'd tried to use a () subshell instead of the {} code block, our intent would fail because theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Redirecting Output for the Life of a Script
- InhaltsvorschauYou'd like to redirect output for an entire script and you'd rather not have to edit every echo or printf statement.Use a little known feature of the exec command to redirect
STDOUTorSTDERR:# Optional, save the "old" STDERR exec 3>&2 # Redirect any output to STDERR to an error log file instead exec 2> /path/to/error_log # script with "globally" redirected STDERR goes here # Turn off redirect by reverting STDERR and closing FH3 exec 2>&3-
Usually exec replaces the running shell with the command supplied in its arguments, destroying the original shell. However, if no command is given, it can manipulate redirection in the current shell. You are not limited to redirecting STDOUT or STDERR, but they are the most common targets for redirection in this case.- help exec
- , "Using bash Net-Redirection"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Working Around "argument list too long" Errors
- InhaltsvorschauYou get an "argument list too long" error while trying to do an operation involving shell wildcard expansion.Use the xargs command, possibly in conjunction with find, to break up your argument list.For simple cases, just use a for loop or find instead of ls:
$ ls /path/with/many/many/files/*e* -/bin/bash: /bin/ls: Argument list too long # Short demo, surrounding ~ are for illustration only $ for i in ./some_files/*e*; do echo "~$i~"; done ~./some_files/A file with (parens)~ ~./some_files/A file with [brackets]~ ~./some_files/File with embedded newline~ ~./some_files/file with = sign~ ~./some_files/file with spaces~ ~./some_files/file with |~ ~./some_files/file with:~ ~./some_files/file with;~ ~./some_files/regular_file~ $ find ./some_files -name '*e*' -exec echo ~{}~ \; ~./some_files~ ~./some_files/A file with [brackets]~ ~./some_files/A file with (parens)~ ~./some_files/regular_file~ ~./some_files/file with spaces~ ~./some_files/file with = sign~ ~./some_files/File with embedded newline~ ~./some_files/file with;~ ~./some_files/file with:~ ~./some_files/file with |~ $ for i in /path/with/many/many/files/*e*; do echo "$i"; done [This works, but the output is too long to list] $ find /path/with/many/many/files/ -name '*e*' [This works, but the output is too long to list]The example above works correctly with the echo command, but when you feed that"$i"into other programs, especially other shell constructs,$IFSand other parsing may come into play. The GNU find and xargs take that into account withfind - print0andxargs -0. (No, we don't know why it's-print0and-0instead of being consistent.) These arguments cause find to use the null character (which can't appear in a filename) instead of whitespace as an output record separator, and xargs to use null as its input record separator. That will correctly parse files containing odd characters.$ find /path/with/many/many/files/ -name '*e*' -print0 | xargs -0 proggy
Note that the default behavior of bash (and sh) is to return unmatched patterns unchanged. That means you could end up with your for loop settingEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Logging to syslog from Your Script
- InhaltsvorschauYou'd like your script to be able to log to syslog.Use logger, Netcat, or bash's built-in network redirection features.logger is installed by default on most systems and is an easy way to send messages to the local syslog service. However, it does not send syslog to remote hosts by itself. If you need to do that, you can use bash or Netcat.
$ logger -p local0.notice -t $0[$$] test message
Netcat is known as the "TCP/IP Swiss Army knife" and is usually not installed by default. It may also be prohibited as a hacking tool by some security policies, though bash's net-redirection features do pretty much the same thing. See the discussion in , "Using bash Net-Redirection" for details on the<133>$0[$$]part.# Netcat $ echo "<133>$0[$$]: Test syslog message from Netcat" | nc -w1 -u loghost 514 # bash $ echo "<133>$0[$$]: Test syslog message from bash" \ > /dev/udp/loghost.example.com/514
logger and Netcat have many more features than we include here. See the respective manpages for details.- man logger
- man nc
- , "Using bash Net-Redirection"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sending Email from Your Script
- InhaltsvorschauYou'd like your script to be able to send email, optionally with attachments.These solutions depend on a compatible mailer such as mail, mailx, or mailto, an Message Transfer Agent (MTA) being installed and running, and proper configuration of your email environment. Unfortunately, you can't always count on all of that, so these solutions must be well tested in your intended environment.The first way to send mail from your script is to write some code to generate and send a message, as follows:
# Simple cat email_body | mail -s "Message subject" recipient1@example.com recipient2@example. com
or:# Attachment only $ uuencode /path/to/attachment_file attachment_name | mail -s "Message Subject" recipient1@example.com recipient2@example.com
or:# Attachment and body $ (cat email_body ; uuencode /path/to/attachment_file attachment_name) | mail -s "Message Subject" recipient1@example.com recipient2@example.com
In practice, it's not always that easy. For one thing, while uuencode will probably be there, mail and friends may or may not, or their capabilities may vary. In some cases mail and mailx are even the same program, hard-or soft-linked together. In production, you will want to use some abstraction to allow for portability. For example, mail works on Linux and the BSDs, but mailx is required for Solaris since its mail lacks support for-s. mailx works on some Linux distributions (e.g., Debian), but not others (e.g., Red Hat). We're choosing the mailer based on hostname here, but depending on your environment usinguname -omight make more sense.# cookbook filename: email_sample # Define some mail settings. Use a case statement with uname or hostname # to tweak settings as required for your environment. case $HOSTNAME in *.company.com ) MAILER='mail' ;; # Linux and BSD host1.* ) MAILER='mailx' ;; # Solaris, BSD and some Linux host2.* ) MAILER='mailto' ;; # Handy, if installed esac RECIPIENTS='recipient1@example.com recipient2@example.com' SUBJECT="Data from $0" [...] # Create the body as a file or variable using echo, printf, or a here-document # Create or modify $SUBJECT and/or $RECIPIENTS as needed [...] ( echo $email_body ; uuencode $attachment $(basename $attachment) ) \ | $MAILER -s "$SUBJECT" "$RECIPIENTS"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Automating a Process Using Phases
- InhaltsvorschauYou have a long job or process you need to automate, but it may require manual intervention and you need to be able to restart at various points in the progress. You might use a
GOTOto jump around, but bash doesn't have that.Use a case statement to break your script up into sections or phases.First, we'll define a standardized way to get answers from the user:# cookbook filename: func_choice function choice { # Let the user make a choice about something and return a standardized # answer. How the default is handled and what happens next is up to # the if/then after the choice in main local answer printf "%b" "\a" # Ring the bell read -p "$*" answer case "$answer" in [yY1] ) choice='y';; [nN0] ) choice='n';; * ) choice="$answer";; esac } # end of function choiceThen, we'll set up our phases:# cookbook filename: using_phases # Main Loop until [ "$phase" = "Finished." ]; do case $phase in phase0 ) ThisPhase=0 NextPhase="$(( $ThisPhase + 1 ))" echo '############################################' echo "Phase$ThisPhase = Initialization of FooBarBaz build" # Things that should only be initialized at the beginning of a # new build cycle go here # ... echo "Phase${ThisPhase}=Ending" phase="phase$NextPhase" ;; # ... phase20 ) ThisPhase=20 NextPhase="$(( $ThisPhase + 1 ))" echo '############################################' echo "Phase$ThisPhase = Main processing for FooBarBaz build" # ... choice "[P$ThisPhase] Do we need to stop and fix anything? [y/N]: " if [ "$choice" = "y" ]; then echo "Re-run '$MYNAME phase${ThisPhase}' after handling this." exit $ThisPhase fi echo "Phase${ThisPhase}=Ending" phase="phase$NextPhase" ;; # ... * ) echo "What the heck?!? We should never get HERE! Gonna croak!" echo "Try $0 -h" exit 99 phase="Finished." ;; esac printf "%b" "\a" # Ring the bell doneEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 16: Configuring and Customizing bash
- InhaltsvorschauWould you want to work in an environment where you couldn't adjust things to your liking? Imagine not being able to adjust the height of your chair, or being forced to walk the long way to the lunchroom, just because someone else thought that was the "right way." That sort of inflexibility wouldn't be acceptable for long; however, that's what most users expect, and accept, from their computing environments. But if you're used to thinking of your user interface as something inflexible and unchangeable, relax—the user interface is not carved in stone. bash lets you customize it so that it works with you, rather than against you.bash gives you a very powerful and flexible environment. Part of that flexibility is the extent to which it can be customized. If you're a casual Unix user, or if you're used to a less flexible environment, you might not be aware of what's possible. This chapter shows you how to configure bash to suit your individual needs and style. If you think the Unix cat command has a ridiculous name (most non-Unix people would agree), you can define an alias that renames it. If you use a few commands all the time, you can assign abbreviations to them, too—or even misspellings that correspond to your favorite typing errors (e.g., "mroe" for the more command). You can create your own commands, which can be used the same way as standard Unix commands. You can alter the prompt so that it contains useful information (like the cur-rent directory). And you can alter the way bash behaves; for example, you can make it case-insensitive, so that it doesn't care about the difference between upper-and lowercase. You will be surprised and pleased at how much you can improve your productivity with a few simple bash tweaks, especially to readline.For more information about customizing and configuring bash, see of Learning the bash Shell by Cameron Newham (O'Reilly).You'd like to understand the various options you can use when starting bash, but
bash --helpis not helping you.In addition tobash--help, trybash -c "help set"andbash-c help, or justhelpsetandEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - bash Startup Options
- InhaltsvorschauYou'd like to understand the various options you can use when starting bash, but
bash --helpis not helping you.In addition tobash--help, trybash -c "help set"andbash-c help, or justhelpsetandhelpif you are already running in a bash shell.bash sometimes has several different ways to set the same option, and this is an example of that. You can set options on startup (for example,bash -x), then later turn the same option off interactively usingset+x.- , "Testing bash Script Syntax"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Customizing Your Prompt
- InhaltsvorschauThe default bash prompt is usually something uninformative that ends with $ and doesn't tell you much, so you would like to customize it to show information you find useful.Customize the
$PS1and$PS2variables as you desire.The default prompt varies depending on your system. bash itself will show its major and minor version (\s-\v\$), for example,bash-3.00$. However, your operating system may have its own default, such as [user@host~]$ ([\u@\h\W]\$) for Fedora Core 5. This solution presents eight basic prompts and three fancier prompts.Section 16.2.2.1: Basic prompts
Here are eight examples of more useful prompts that will work with bash-1.14.7 or newer. The trailing \$ displays # if the effective UID is zero (i.e., you are root) and $ otherwise:- Username@hostname, the date and time, and the current working directory:
$ export PS1='[\u@\h \d \A] \w \$ ' [jp@freebsd Wed Dec 28 19:32] ~ $ cd /usr/local/bin/ [jp@freebsd Wed Dec 28 19:32] /usr/local/bin $
- Username@long-hostname, the date and time in ISO 8601 format, and the base-name of the current working directory
(\W):$ export PS1='[\u@\H \D{%Y-%m-%d %H:%M:%S%z}] \W \$ ' [jp@freebsd.jpsdomain.org 2005-12-28 19:33:03-0500] ~ $ cd /usr/local/bin/ [jp@freebsd.jpsdomain.org 2005-12-28 19:33:06-0500] bin $ - Username@hostname, bash version, and the current working directory
(\w):$ export PS1='[\u@\h \V \w] \$ ' [jp@freebsd 3.00.16] ~ $ cd /usr/local/bin/ [jp@freebsd 3.00.16] /usr/local/bin $
- New line, username@hostname, base PTY, shell level, history number, newline, and full working directory name
($PWD):$ export PS1='\n[\u@\h \l:$SHLVL:\!]\n$PWD\$ ' [jp@freebsd ttyp0:3:21] /home/jp$ cd /usr/local/bin/ [jp@freebsd ttyp0:3:22] /usr/local/bin$
PTY is the number of the pseudoterminal (in Linux terms) to which you are connected. This is useful when you have more than one session and are trying to keep track of which is which. Shell level is the depth of subshells you are in. When you first log in it's 1, and as you run subprocesses (for example, screen) it increments, so after running screen it would normally be 2. The history line is the number of the current command in the command history.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Change Your $PATH Permanently
- InhaltsvorschauYou need to permanently change your path.First you need to discover where the path is set, and then update it. For your local account, it's probably set in ~/.profile or ~/.bash_profile. Find the file with
grep -l PATH ~/.[^.]*and edit it with your favorite editor; then source the file to have the change take effect immediately.If you are root and you need to set the path for the entire system, the basic procedure is the same, but there are different files in /etc where the$PATHmay be set, depending on your operating system and version. The most likely file is /etc/profile, but /etc/bashrc, /etc/rc, /etc/default/login, ~/.ssh/environment, and the PAM /etc/ environment files are also possible.Thegrep -l PATH~/.[^.]*command is interesting because of the nature of shell wild-card expansion and the existence of the /. and /.. directories. See , "Showing All Hidden (dot) Files in the Current Directory," for details.The locations listed in the$PATHhave security implications, especially when you are root. If a world-writable directory is in root's path before the typical directories (i.e., /bin, /sbin), then a local user can create files that root might execute, doing arbitrary things to the system. This is the reason that the current directory (.) should not be in root's path either.To be aware of this issue and avoid it:- Make root's path as short as possible, and never use relative paths.
- Avoid having world-writable directories in root's path.
- Consider setting explicit paths in shell scripts run by root.
- Consider hardcoding absolute paths to utilities used in shell scripts run by root.
- Put user or application directories last in the
$PATH, and then only for unprivileged users.
- , "Showing All Hidden (dot) Files in the Current Directory"
- , "Running Any Executable"
- , "Setting a Secure $PATH"
- , "Finding World-Writable Directories in Your $PATH"
- , "Adding the Current Directory to the $PATH"
- , "Change Your $PATH Temporarily"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Change Your $PATH Temporarily
- InhaltsvorschauYou want to easily add or remove a directory to or from your
$PATHfor this session only.There are several ways to handle this problem.You can prepend or append the new directory, usingPATH="newdir:$PATH"orPATH="$PATH:newdir", though you should make sure the directory isn't already in the$PATH.If you need to edit something in the middle of the path, you can echo the path to the screen, then use your terminal's kill and yank (copy and paste) facility to duplicate it on a new line and edit it. Or, you can add the "Macros that are convenient for shell interaction" from the readline documentation at http://tiswww.tis.case.edu/php/chet/readline/readline.html#SEC12, specifically:# edit the path "\C-xp": "PATH=${PATH}\e\C-e\C-a\ef\C-f" # [...] # Edit variable on current line. "\M-\C-v": "\C-a\C-k$\C-y\M-\C-e\C-a\C-y="Then pressing Ctrl-X P will display the$PATHon the current line for you to edit, while typing any variable name and pressing Meta Ctrl-V will display that variable for editing. Very handy.For simple cases you can use this quick function (adapted slightly from Red Hat Linux's /etc/profile):# cookbook filename: func_pathmunge # Adapted from Red Hat Linux function pathmunge { if ! echo $PATH | /bin/egrep -q "(^|:)$1($|:)" ; then if [ "$2" = "after" ] ; then PATH="$PATH:$1" else PATH="$1:$PATH" fi fi }The egrep pattern looks for the value in$1between two : or (|) at the beginning (^) or end ($) of the$PATHstring. We chose to use acasestatement in our function, and to force a leading and trailing : to do the same thing. Ours is theoretically faster since it uses a shell built-in, but the Red Hat version is more concise. Our version is also an excellent illustration of the fact that theifcommand works on exit codes, so the firstifworks by using the exit code set by grep, while the second requires the use of the test operator ( [ ] ).For more complicated cases when you'd like a lot of error checking you can source and then use the following more generic functions:# cookbook filename: func_tweak_path #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # Add a directory to the beginning or end of your path as long as it's not # already present. Does not take into account symbolic links! # Returns: 1 or sets the new $PATH # Called like: add_to_path <directory> (pre|post) function add_to_path { local location=$1 local directory=$2 # Make sure we have something to work with if [ -z "$location" -o -z "$directory" ]; then echo "$0:$FUNCNAME: requires a location and a directory to add&" >&2 echo "e.g. add_to_path pre /bin" >&2 return 1 fi # Make sure the directory is not relative if [ $(echo $directory | grep '^/') ]; then :echo "$0:$FUNCNAME: '$directory' is absolute" >&2 else echo "$0:$FUNCNAME: can't add relative directory '$directory' to the \$PATH" >&2 return 1 fi # Make sure the directory to add actually exists if [ -d "$directory" ]; then : echo "$0:$FUNCNAME: directory exists" >&2 else echo "$0:$FUNCNAME: '$directory' does not exist--aborting" >&2 return 1 fi # Make sure it's not already in the PATH if [ $(contains "$PATH" "$directory") ]; then echo "$0:$FUNCNAME: '$directory' already in \$PATH--aborting" >&2 else :echo "$0:$FUNCNAME: adding directory to \$PATH" >&2 fi # Figure out what to do case $location in pre* ) PATH="$directory:$PATH" ;; post* ) PATH="$PATH:$directory" ;; * ) PATH="$PATH:$directory" ;; esac # Clean up the new path, then set it PATH=$(clean_path $PATH) } # end of function add_to_path #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #Remove a directory from your path, if present. # Returns: sets the new $PATH # Called like: rm_from_path <directory> function rm_from_path { local directory=$1 # Remove all instances of $directory from $PATH PATH=${PATH//$directory/} # Clean up the new path, then set it PATH=$(clean_path $PATH) } # end of function rm_from_path #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # Remove leading/trailing or duplicate ':', remove duplicate entries # Returns: echos the "cleaned up" path # Called like: cleaned_path=$(clean_path $PATH) function clean_path { local path=$1 local newpath local directory # Make sure we have something to work with [ -z "$path" ] && return 1 # Remove duplicate directories, if any for directory in ${path//:/ }; do contains "$newpath" "$directory" && newpath="${newpath}:${directory}" done # Remove any leading ':' separators # Remove any trailing ':' separators # Remove any duplicate ':' separators newpath=$(echo $newpath | sed 's/^:*//; s/:*$//; s/::/:/g') # Return the new path echo $newpath } # end of function clean_path #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # Determine if the path contains a given directory # Return 1 if target is contained within pattern, 0 otherwise # Called like: contains $PATH $dir function contains { local pattern=":$1:" local target=$2 # This will be a case-sensitive comparison unless nocasematch is set case $pattern in *:$target:* ) return 1;; * ) return 0;; esac } # end of function containsEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting Your $CDPATH
- InhaltsvorschauYou want to make it easier to switch between several directories in various locations.Set your
$CDPATHappropriately. Your commonly used directories will likely be unique, so for a contrived example, suppose you spend a lot of time working with init's rc directories:/home/jp$ cd rc3.d bash: cd: rc3.d: No such file or directory /home/jp$ export CDPATH='.:/etc' /home/jp$ cd rc3.d /etc/rc3.d /etc/rc3.d$ cd rc5.d /etc/rc5.d /etc/rc5.d$ /etc/rc5.d$ cd games bash: cd: games: No such file or directory /etc/rc5.d$ export CDPATH='.:/etc:/usr' /etc/rc5.d$ cd games /usr/games /usr/games$
According to the bash Reference,$CDPATHis "a colon-separated list of directories used as a search path for the cd built-in command." Think of it as$PATHfor cd. It's a little subtle, but can be very handy.If the argument to cd begins with a slash,$CDPATHwill not be used. If$CDPATHis used, the absolute pathname to the new directory is printed to STDOUT, as in the example above.Watch out when running bash in POSIX mode (e.g., as /bin/sh or with--posix). As the bash Reference notes:"If $CDPATH is set, the cd built-in will not implicitly append the current directory to it. This means that cd will fail if no valid directory name can be constructed from any of the entries in$CDPATH, even if a directory with the same name as the name given as an argument to cd exists in the current directory."To avoid this, explicitly include . in$CDPATH. However, if you do that, then another subtle point noted in the bash Reference comes into play:"If a nonempty directory name from$CDPATHis used, or if '-' is the first argument, and the directory change is successful, the absolute pathname of the new working directory is written to the standard output."In other words, pretty much every time you use cd it will echo the new path to STDOUT, which is not the standard behavior.Common directories to include in$CDPATHare:- .
- The current directory (see the warning above)
- ~/
- Your home directory
- ..
- The parent directory
- ../..
- The grandparent directory
- ~/.dirlinks
- A hidden directory containing nothing but symbolic links to other commonly used directories
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Shortening or Changing Command Names
- InhaltsvorschauYou'd like to shorten a long or complex command you use often, or you'd like to rename a command you can't remember or find awkward to type.Do not manually rename or move executable files, as many aspects of Unix and Linux depend on certain commands existing in certain places; instead, you should use aliases, functions, and possibly symbolic links.According to the bash Reference, "Aliases allow a string to be substituted for a word when it is used as the first word of a simple command. The shell maintains a list of aliases that may be set and unset with the alias and unalias built-in commands." This means that you can rename commands, or create a macro, by listing many commands in one alias. For example,
alias copy='cp'oralias ll.='ls -ld .*'.Aliases are only expanded once, so you can change how a command works, as withalias ls='ls -F', without going into an endless loop. In most cases only the first word of the command line is checked for alias expansion, and aliases are strictly text substitutions; they cannot use arguments to themselves. In other words, you can't doalias='mkdir $1 && cd $1' because that doesn't work.Functions are used in two different ways. First, they can be sourced into your interactive shell, where they become, in effect, shell scripts that are always held in memory. They are usually small, and are very fast since they are already in memory and are executed in the current process, not in a spawned subshell. Second, they may be used within a script as subroutines. Functions do allow arguments. For example:# cookbook filename: func_calc # Trivial command line calculator function calc { # INTEGER ONLY! --> echo The answer is: $(( $* )) # Floating point awk "BEGIN {print \"The answer is: \" $* }"; } # end of calcFor personal or system-wide use, you are probably better off using aliases or functions to rename or tweak commands, but symbolic links are very useful in allowing a command to be in more than one place at a time. For example, Linux systems almost always use /bin/bash while other systems may use /usr/bin/bash, /usr/local/bin/bashEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adjusting Shell Behavior and Environment
- InhaltsvorschauYou want to adjust your shell environment to account for the way you work, your physical location, your language, and more.See the table in the section "Adjusting Shell Behavior Using set, shopt, and Environment Variables" in .There are three ways to adjust various aspects of your environment. set is standardized in POSIX and uses one-letter options. shopt is specifically for bash shell options. And there are many environment variables in use for historical reasons, as well as for compatibility with many third-party applications. How you adjust what and where, can be be very confusing. The table in the section "Adjusting Shell Behavior Using set, shopt, and Environment Variables" in will help you sort it out, but it's too big to duplicate here.
- help set
- help shopt
- Bash Docs (http://www.bashcookbook.com)
- "Adjusting Shell Behavior Using set, shopt, and Environment Variables" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adjusting readline Behavior Using .inputrc
- InhaltsvorschauYou'd like to adjust the way bash handles input, especially command completion. For example, you'd like it to be case-insensitive.Edit or create a ~/.inputrc or /etc/inputrc file as appropriate. There are many parameters you can adjust to your liking. To have readline use your file when it initializes, set
$INPUTRC; for example,set INPUTRC='~/.inputrc'. To re-read the file and apply or test after making changes, usebind-ffilename.We recommend you explore the bind command and the readline documentation, especiallybind -v, bind -l, bind -s,andbind -p, though the last one is rather long and cryptic.Some useful settings for users from other environments, notably Windows, are (see the section "Readline Init File Syntax" in ):# settings/inputrc: # readline settings # To re-read (and implement changes to this file) use: # bind -f $SETTINGS/inputrc # First, include any systemwide bindings and variable # assignments from /etc/inputrc # (fails silently if file doesn't exist) $include /etc/inputrc $if Bash # Ignore case when doing completion set completion-ignore-case on # Completed dir names have a slash appended set mark-directories on # Completed names which are symlinks to dirs have a slash appended set mark-symlinked-directories on # List ls -F for completion set visible-stats on # Cycle through ambiguous completions instead of list "\C-i": menu-complete # Set bell to audible set bell-style audible # List possible completions instead of ringing bell set show-all-if-ambiguous on # From the readline documentation at # http://tiswww.tis.case.edu/php/chet/readline/readline.html#SEC12 # Macros that are convenient for shell interaction # edit the path "\C-xp": "PATH=${PATH}\e\C-e\C-a\ef\C-f" # prepare to type a quoted word -- insert open and close double quotes # and move to just after the open quote "\C-x\"": "\"\"\C-b" # insert a backslash (testing backslash escapes in sequences and macros) "\C-x\\": "\\" # Quote the current or previous word "\C-xq": "\eb\"\ef\"" # Add a binding to refresh the line, which is unbound "\C-xr": redraw-current-line # Edit variable on current line. #"\M-\C-v": "\C-a\C-k$\C-y\M-\C-e\C-a\C-y=" "\C-xe": "\C-a\C-k$\C-y\M-\C-e\C-a\C-y=" $endifEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping a Private Stash of Utilities by Adding ~/bin
- InhaltsvorschauYou have a stash of personal utilities you like to use, but you are not root on the system and can't place them into the normal locations like /bin or /usr/local/bin, or there is some other reason to separate them.Create a ~/bin directory, place your utilities in it and add it to your path:
$ PATH="$PATH:~/bin"
You'll want to make this change in one of your shell initialization files, such as ~/.bashrc. Some systems already add$HOME/binas the last directory in a nonprivileged user account by default, so check first.As a fully qualified shell user (well, you bought this book), you'll certainly be creating lots of scripts. It's inconvenient to invoke scripts with their full pathname. By collecting your scripts in a ~/bin directory, you can make your scripts look like regular Unix programs—at least to you.For security reasons, don't put your bin directory at the start of your path. Starting your path with ~/bin makes it easy to override system commands—which is inconvenient, if it happens accidentally (we've all done it), and dangerous if it's done maliciously.- , "Finding World-Writable Directories in Your $PATH"
- , "Adding the Current Directory to the $PATH"
- , "Change Your $PATH Permanently"
- , "Change Your $PATH Temporarily"
- , "Shortening or Changing Command Names"
- , "Naming Your Script Test"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Secondary Prompts: $PS2, $PS3, $PS4
- InhaltsvorschauYou'd like to understand what the
$PS2, PS3, andPS4prompts do.$PS2is called the secondary prompt string and is used when you are interactively entering a command that you have not completed yet. It is usually set to "> " but you can redefine it. For example:[jp@freebsd jobs:0] /home/jp$ export PS2='Secondary: ' [jp@freebsd jobs:0] /home/jp$ for i in $(ls) Secondary: do Secondary: echo $i Secondary: done colors deepdir trunc_PWD
$PS3is the select prompt, and is used by the select statement to prompt the user for a value. It defaults to #?, which isn't very intuitive. You should change it before using the select command; for example:[jp@freebsd jobs:0] /home/jp$ select i in $(ls) Secondary: do Secondary: echo $i Secondary: done 1) colors 2) deepdir 3) trunc_PWD #? 1 colors #? ^C [jp@freebsd jobs:0] /home/jp$ export PS3='Choose a directory to echo: ' [jp@freebsd jobs:0] /home/jp$ select i in $(ls); do echo $i; done 1) colors 2) deepdir 3) trunc_PWD Choose a directory to echo: 2 deepdir Choose a directory to echo: ^C
$PS4is displayed during trace output. Its first character is shown as many times as necessary to denote the nesting depth. The default is "+ ". For example:[jp@freebsd jobs:0] /home/jp$ cat demo #!/usr/bin/env bash set -o xtrace alice=girl echo "$alice" ls -l $(type -path vi) echo line 10 ech0 line 11 echo line 12 [jp@freebsd jobs:0] /home/jp$ ./demo + alice=girl + echo girl girl ++ type -path vi + ls -l /usr/bin/vi -r-xr-xr-x 6 root wheel 285108 May 8 2005 /usr/bin/vi + echo line 10 line 10 + ech0 line 11 ./demo: line 11: ech0: command not found + echo line 12 line 12 [jp@freebsd jobs:0] /home/jp$ export PS4='+xtrace $LINENO: ' [jp@freebsd jobs:0] /home/jp$ ./demo +xtrace 5: alice=girl +xtrace 6: echo girl girl ++xtrace 8: type -path vi +xtrace 8: ls -l /usr/bin/vi -r-xr-xr-x 6 root wheel 285108 May 8 2005 /usr/bin/vi +xtrace 10: echo line 10 line 10 +xtrace 11: ech0 line 11 ./demo: line 11: ech0: command not found +xtrace 12: echo line 12 line 12
The$PS4prompt uses theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Synchronizing Shell History Between Sessions
- InhaltsvorschauYou run more than one bash session at a time and you would like to have a shared history between them. You'd also like to prevent the last session closed from clobbering the history from any other sessions.Use the history command to synchronize your history between sessions manually or automatically.Using default settings, the last shell to gracefully exit will overwrite your history file, so unless it is synchronized with any other shells you had open at the same time, it will clobber their histories. Using the shell option shown in , "Setting Shell History Options," to append rather than overwrite the history file helps, but keeping your history in sync across sessions may offer additional benefits.Manually synchronizing history involves writing an alias to append the current history to the history file, then re-reading anything new in that file into the current shell's history:
$ history -a $ history -n # OR, 'history sync' alias hs='history -a ; history -n'
The disadvantage to this approach is that you must manually run the commands in each shell when you want to synchronize your history.To automate that approach, you could use the$PROMPT_COMMANDvariable:PROMPT_COMMAND='history -a ; history -n'
The value of$PROMPT_COMMANDis interpreted as a command to execute each time the default interactive prompt$PS1is displayed. The disadvantage to that approach is that it runs those commands every time$PS1is displayed. That is very often, and on a heavily loaded or slower system that can cause it significant slowdown in your shell, especially if you have a large history.- help history
- , "Setting Shell History Options"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Setting Shell History Options
- InhaltsvorschauYou'd like more control over your command-line history.Set the
$HIST*variables and shell options as desired.The$HISTFILESIZEvariable sets the number of lines permitted in the$HISTFILE. The default for$HISTSIZEis 500 lines, and$HISTFILEis ~/.bash_history unless you are in POSIX mode, in which case it's ~/.sh_history. Increasing$HISTSIZEmay be useful, and unsetting it causes the$HISTFILElength to be unlimited. Changing$HISTFILEprobably isn't necessary, except that if it is not set or the file is not writable, no history will be written to disk. The$HISTSIZEvariable sets the number of lines permitted in the history stack in memory.$HISTIGNOREand$HISTCONTROLcontrol what goes into your history in the first place.$HISTIGNOREis more flexible since it allows you to specify patterns to decide what command lines to save to the history.$HISTCONTROLis more limited in that it sup-ports only the few keywords listed here (any other value is ignored):ignorespace- Command lines that begin with a space character are not saved in the history list.
ignoredups- Command lines that match the previous history entry are not saved in the history list.
ignoreboth- Shorthand for both ignorespace and ignoredups.
erasedups- All previous command lines that match the current line are removed from the history list before that line is saved.
If$HISTCONTROLis not set, or does not contain any of these keywords, all commands are saved to the history list, subject to processing$HISTIGNORE. The second and subsequent lines of a multiline compound command are not tested, and are added to the history regardless of the value of$HISTCONTROL.(Material in the preceding paragraphs has been adapted from Edition 2.5b of The GNU Bash Reference Manual for bash Version 2.05b, last updated July 15, 2002; http://www.gnu.org/software/bash/manual/bashref.html.)As of bash version 3, there is a fascinating new variable called$HISTTIMEFORMAT. If set and non-null, it specifies an strftime format string to use when displaying or writing the history. If you don't have bash version 3, but you do use a terminal with a scroll-back buffer, adding a date and time stamp to your prompt can also be very helpful. See , "Customizing Your Prompt." Watch out because stockEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating a Better cd Command
- InhaltsvorschauYou cd into a lot of deep directories and would like to type
cd….. instead ofcd../ ../../.. to move up four levels.Use this function:# cookbook filename: func_cd # Allow use of 'cd ...' to cd up 2 levels, 'cd ....' up 3, etc. (like 4NT/4DOS) # Usage: cd ..., etc. function cd { local option= length= count= cdpath= i= # Local scope and start clean # If we have a -L or -P sym link option, save then remove it if [ "$1" = "-P" -o "$1" = "-L" ]; then option="$1" shift fi # Are we using the special syntax? Make sure $1 isn't empty, then # match the first 3 characters of $1 to see if they are '...' then # make sure there isn't a slash by trying a substitution; if it fails, # there's no slash. Both of these string routines require Bash 2.0+ if [ -n "$1" -a "${1:0:3}" = '...' -a "$1" = "${1%/*}" ]; then # We are using special syntax length=${#1} # Assume that $1 has nothing but dots and count them count=2 # 'cd ..' still means up one level, so ignore first two # While we haven't run out of dots, keep cd'ing up 1 level for ((i=$count;i<=$length;i++)); do cdpath="${cdpath}../" # Build the cd path done # Actually do the cd builtin cd $option "$cdpath" elif [ -n "$1" ]; then # We are NOT using special syntax; just plain old cd by itself builtin cd $option "$*" else # We are NOT using special syntax; plain old cd by itself to home dir builtin cd $option fi } # end of cdThe cd command takes an optional-Lor-Pargument that respectively follow symbolic links or follow the physical directory structure. Either way, we have to take them into account if we want to redefine how cd works.Then, we make sure$1isn't empty and match the first three characters of$1to see if they are '…'. We then make sure there isn't a slash by trying a substitution; if it fails, there's no slash. Both of these string routines require bash version 2.0+. After that, we build the actual cd command using a portableEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating and Changing into a New Directory in One Step
- InhaltsvorschauYou often create new directories and immediately change into them for some operation, and all that typing is tedious.Add the following function to an appropriate configuration file such as your ~/.bashrc file and source it:
# cookbook filename: func_mcd # mkdir newdir then cd into it # usage: mcd (<mode>) <dir> function mcd { local newdir='_mcd_command_failed_' if [ -d "$1" ]; then # Dir exists, mention that... echo "$1 exists..." newdir="$1" else if [ -n "$2" ]; then # We've specified a mode command mkdir -p -m $1 "$2" && newdir="$2" else # Plain old mkdir command mkdir -p "$1" && newdir="$1" fi fi builtin cd "$newdir" # No matter what, cd into it } # end of mcdFor example:$ source mcd $ pwd /home/jp $ mcd 0700 junk $ pwd /home/jp/junk $ ls -ld . drwx------ 2 jp users 512 Dec 6 01:03 .
This function allows you to optionally specify a mode for the mkdir command to use when creating the directory. If the directory already exists, it will mention that fact but still cd into it. We use the command command to make sure that we ignore any shell functions for mkdir, and the builtin command to make sure we only use the shell cd.We also assign_mcd_command_failed_to a local variable in case the mkdir fails. If it works, the correct new directory is assigned. If it fails, when the cd tries to execute it will display a reasonably useful message, assuming you don't have a lot of _mcd_ command_failed_ directories lying around:$ mcd /etc/junk mkdir: /etc/junk: Permission denied -bash: cd: _mcd_command_failed_: No such file or directory
You might think that we could easily improve this usingbreakorexitif themkdirfails.breakonly works in afor, while, oruntilloop andexitwill actually exit our shell, since a sourced function runs in the same process as the shell. We could, however, usereturn, which we will leave as an exercise for the reader.command mkdir -p "$1" && newdir="$1" || exit 1 # This will exit our shell command mkdir -p "$1" && newdir="$1" || break # This will fail
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting to the Bottom of Things
- InhaltsvorschauYou work in a lot of narrow but deep directory structures, where all the content is at the bottom and you're tired of having to manually cd so many levels.
alias bot='cd $(dirname $(find . | tail -1))'
This use of find in a large directory structure such as /usr could take a while and isn't recommended.Depending on how your directory structure is set up, this may not work for you; you'll have to try it and see. Thefind. will simply list all the files and directories in the current directory and below, thetail -1will grab the last line,dirnamewill extract just the path, andcdwill take you there. It may be possible for you to tweak the command to get it to put you in the right place. For example:alias bot='cd $(dirname $(find . | sort -r | tail -5 | head -1))' alias bot='cd $(dirname $(find . | sort -r | grep -v 'X11' | tail -3 | head -1))'
Keep trying the part in the inner-most parentheses, especially tweaking the find command, until you get the results you need. Perhaps there is a key file or directory at the bottom of the structure, in which case the following function might work:function bot { cd $(dirname $(find . | grep -e "$1" | head -1)); }Note that aliases can't use arguments, so this must be a function. We use grep rather than a-nameargument to find because grep is much more flexible. Depending on your structure, you might want to use tail instead of head. Again, test the find command first.- man find
- man dirname
- man head
- man tail
- man grep
- man sort
- , "Creating a Better cd Command"
- , "Creating and Changing into a New Directory in One Step"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adding New Features to bash Using Loadable Built-ins
- InhaltsvorschauThe material in this recipe also appears in Learning the bash Shell by Cameron Newham (O'Reilly).You have something that you'd like bash to do, but there's no built-in command for it. For efficiency reasons, you want it to be built-in to the shell rather than an external program. Or, you already have the code in C and don't want to or can't rewrite it.Use the dynamically loadable built-ins introduced in bash version 2.0. The bash archive contains a number of pre-written built-ins in the directory ./examples/ loadables/, especially the canonical hello.c. You can build them by uncommenting the lines in the file Makefile that are relevant to your system, and typing make. We'll take one of these built-ins, tty, and use it as a case study for built-ins in general.The following is a list of the built-ins provided in bash version 3.2's ./examples/basename.cid.cpush.ctruefalse.ccat.cln.crealpath.ctty.ccut.clogname.crmdir.cuname.cdirname.cmkdir.csleep.cunlink.cfinfo.cnecho.cstrftime.cwhoami.cgetconf.cpathchk.csync.cperl/bperl.chead.cprint.ctee.cperl/iperl.chello.cprintenv.ctemplate.cOn systems that support dynamic loading, you can write your own built-ins in C, compile them into shared objects, and load them at any time from within the shell with the enable built-in.We will discuss briefly how to go about writing a built-in and loading it in bash. The discussion assumes that you have experience with writing, compiling, and linking C programs.
ttywill mimic the standard Unix command tty. It will print the name of the terminal that is connected to standard input. The built-in will, like the command, return true if the device is a TTY and false if it isn't. In addition, it will take an option,-s, which specifies that it should work silently (i.e., print nothing and just return a result).The C code for a built-in can be divided into three distinct sections: the code that implements the functionality of the built-in, a help text message definition, and a structure describing the built-in so that bash can access it.The description structure is quite straightforward and takes the form:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Improving Programmable Completion
- InhaltsvorschauThis recipe was adapted directly from Learning the bash Shell by Cameron Newham (O'Reilly).You love bash's programmable completion but wish it could be more aware of context, especially for commands that you use often.Find and install additional programmable completion libraries, or write your own. Some examples are provided in the bash tarball in ./examples/complete. Some distributions (e.g., SUSE) have their own version in /etc/profile.d/complete.bash. However, the largest and most well known of the third-party libraries is certainly Ian Macdonald's, which you may download as a tarball or RPM from http://www.caliban.org/bash/index.shtml#completion or http://freshmeat.net/projects/bashcompletion/. This library is already included in Debian (and derivatives like Ubuntu and MEPIS), and it is present in Fedora Extras as well as other third-party repositories.According to Ian's README: "Many of the completion functions assume GNU versions of the various text utilities that they call (e.g., grep, sed, and awk). Your mileage may vary."At the time of this writing there are 103 modules provided by the bash-completion-20060301.tar.gz library. The following is an excerpted list:# bash alias completion
# bash export completion
# bash shell function completion
# chown(1) completion
# chgrp(1) completion
# RedHat & Debian GNU/Linux if{up,down} completion
# cvs(1) completion
# rpm(8) completion
# chsh(1) completion
# chkconfig(8) completion
# ssh(1) completion
# GNU make(1) completion
# GNU tar(1) completion
# jar(1) completion
# Linux iptables(8) completion
# tcpdump(8) completion
# ncftp(1) bookmark completion
# Debian dpkg(8) completion
# Java completion
# PINE address-book completion
# mutt completion
# Python completion
# Perl completion
# FreeBSD package management tool completion
# mplayer(1) completion
# gpg(1) completion
# dict(1) completion
# cdrecord(1) completion
# yum(8) completion
# smartctl(8) completion
# vncviewer(1) completion
# svn completionProgrammable completion is a feature that was introduced in bash version 2.04. It extends the built-in textual completion by providing hooks into the completion mechanism. This means that it is possible to write virtually any form of completion desired. For instance, if you were typing theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using Initialization Files Correctly
- InhaltsvorschauYou'd like to know just what the heck is with all the initialization, or rc, files.Here's the cheat sheet for files and what do with them. Some or all of these files may be missing from your system, depending on how it is set up. Systems that use bash by default (e.g., Linux) tend to have a complete set; systems that use some other shell by default are usually missing at least some of them.
- /etc/profile
- Global login environment file for Bourne and similar login shells. We recommend you leave this alone unless you are the system administrator and know what you are doing.
- /etc/bashrc (Red Hat) /etc/bash.bashrc (Debian)
- Global environment file for interactive bash subshells. We recommend you leave this alone unless you are the system administrator and know what you are doing.
- /etc/bash_completion
- If this exists, it's almost certainly the configuration file for Ian Macdonald's programmable completion library (see , "Improving Programmable Completion"). We recommend looking into it—it's pretty cool.
- /etc/inputrc
- Global GNU Readline configuration. We recommend tweaking this as desired for the entire system (if you are the administrator), or tweaking ~/.inputrc for just you (, "Getting Started with a Custom Configuration"). This is not executed or sourced but read in via Readline and
$INPUTRC, and$include(orbind -f). Note that it may containincludestatements to other Readline files. - ~/.bashrc
- Personal environment file for interactive bash subshells. We recommend that you place your aliases, functions, and fancy prompts here.
- ~/.bash_profile
- Personal profile for bash login shells. We recommend that you make sure this sources ~/.bashrc, then ignore it.
- ~/.bash_login
- Personal profile file for Bourne login shells; only used by bash if ~/.bash_profile is not present. We recommend you ignore this.
- ~/.profile
- Personal profile file for Bourne login shells; only used by bash if ~/.bash_profile and ~/.bash_login are not present. We recommend you ignore this unless you also use other shells that use it.
- ~/.bash_history
- Default storage file for your shell command history. We recommend you use the history tools (, "Setting Shell History Options") to manipulate it instead of trying to directly edit it. This is not executed or sourced, it's just a data file.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating Self-Contained, Portable RC Files
- InhaltsvorschauYou work on a number of machines, some of which you have limited or full root control over, and some of which you do not, and you want to replicate a consistent bash environment while still allowing custom settings by operating system, machine, or other (e.g., work, home) criteria.Put all of your customizations in files in a settings subdirectory, copy or rsync that directory to a location such as ~/ or /etc, and use includes and symbolic links (e.g.,
ln -s ~/settings/screenrc ~/.screenrc) as necessary. Use logic in your customization files to account for criteria such as operating system, location, etc.You may also choose not to use leading dots in the filenames to make it a little easier to manage the files. As you saw in , "Showing All Hidden (dot) Files in the Current Directory," the leading dot causes ls not to show the file by default, thus eliminating some clutter in your home directory listing. But since we'll be using a directory that exists only to hold configuration files, using the dot is not necessary. Note that dot files are usually not used in /etc either, for the same reason.See , "Getting Started with a Custom Configuration" for a sample to get you started.Here are the assumptions and criteria we used in developing this solution:Section 16.19.3.1: Assumptions
- You have a complex environment in which you control some, but not all, of the machines you use.
- For machines you control, one machine exports /opt/bin and all other machines NFS-mount it, so all configuration files reside there. We used /opt/bin because it's short and less likely to collide with existing directories than /usr/local/bin, but feel free to use whatever makes sense.
- For some machines with partial control, a system-wide configuration in /etc is used.
- For machines on which you have no administrative control, dot files are used in ~/.
- You have settings that will vary from machine to machine, and in different environments (e.g., home or work).
Section 16.19.3.2: Criteria
- Require as few changes as possible when moving configuration files between operating systems and environments.
- Supplement, but do not replace, operating system default or system administrator supplied configurations.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Getting Started with a Custom Configuration
- InhaltsvorschauYou'd like to tweak your environment but aren't quite sure where to start.Here are some samples to give you an idea of what you can do. We follow the suggestion in , "Creating Self-Contained, Portable RC Files" to keep customizations separate for easy back-outs and portability between systems.For system-wide profile settings, add the following to /etc/profile. Since that file is also used by the true Bourne shell, be careful not to use any bash-only features (e.g., source instead of .) if you do this on a non-Linux system. Linux uses bash as the default shell for both /bin/sh and /bin/bash (except when it doesn't, as in Ubuntu 6– 10+, which uses dash). For user-only settings, add it to only one of ~/.bash_profile, ~/.bash_login, or ~/.profile, in that order, whichever exists first:
# cookbook filename: add_to_bash_profile # If we're running in bash, search for then source our settings # You can also just hard code $SETTINGS, but this is more flexible if [ -n "$BASH_VERSION" ]; then for path in /opt/bin /etc ~ ; do # Use the first one found if [ -d "$path/settings" -a -r "$path/settings" -a -x "$path/settings" ] then export SETTINGS="$path/settings" fi done source "$SETTINGS/bash_profile" #source "$SETTINGS/bash_rc" # If necessary fi
For system-wide environment settings, add the following to /etc/bashrc (or /etc/bash. bashrc):# cookbook filename: add_to_bashrc # If we're running in bash, and it isn't already set, # search for then source our settings # You can also just hard code $SETTINGS, but this is more flexible if [ -n "$BASH_VERSION" ]; then if [ -z "$SETTINGS" ]; then for path in /opt/bin /etc ~ ; do # Use the first one found if [ -d "$path/settings" -a -r "$path/settings" -a -x "$path/settings" ] then export SETTINGS="$path/settings" fi done fi source "$SETTINGS/bashrc" fi
Sample bash_profile:# cookbook filename: bash_profile # settings/bash_profile: Login shell environment settings # To re-read (and implement changes to this file) use: # source $SETTINGS/bash_profile # Fail-safe. This should be set when we're called, but if not, the # "not found" error messages should be pretty clear. # Use leading ':' to prevent this from being run as a program after # it is expanded. : ${SETTINGS:='SETTINGS_variable_not_set'} # DEBUGGING only--will break scp, rsync # echo "Sourcing $SETTINGS/bash_profile..." # export PS4='+xtrace $LINENO: ' # set -x # Debugging/logging--will not break scp, rsync #case "$-" in # *i*) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Interactive" \ # "$SETTINGS/bash_profile ssh=$SSH_CONNECTION" >> ~/rc.log ;; # * ) echo "$(date '+%Y-%m-%d_%H:%M:%S_%Z') Non-interactive" \ # "$SETTINGS/bash_profile ssh=$SSH_CONNECTION" >> ~/rc.log ;; #esac # Use the keychain (http://www.gentoo.org/proj/en/keychain/) shell script # to manage ssh-agent, if it's available. If it's not, you should look # into adding it. for path in $SETTINGS ${PATH//:/ }; do if [ -x "$path/keychain" ]; then # Load default id_rsa and/or id_dsa keys, add others here as needed # See also --clear --ignore-missing --noask --quiet --time-out $path/keychain ~/.ssh/id_?sa break fi done # Apply interactive subshell customizations to login shells too. # The system profile file in /etc probably already does this. # If not, it's probably better to do in manually in wherever you: # source "$SETTINGS/bash_profile" # But just in case... #for file in /etc/bash.bashrc /etc/bashrc ~/.bashrc; do # [ -r "$file" ] && source $file && break # Use the first one found #done # Do site or host specific things here case $HOSTNAME in *.company.com ) # source $SETTINGS/company.com ;; host1.* ) # host1 stuff ;; host2.company.com ) # source .bashrc.host2 ;; drake.* ) # echo DRAKE in bash_profile.jp! ;; esac # Do this last because we basically fork off from here. If we exit screen # we return to a fully configured session. The screen session gets configured # as well, and if we never leave it, well, this session isn't that bloated. # Only run if we are not already running screen AND '~/.use_screen' exists. if [ $TERM != "screen" -a "$USING_SCREEN" != "YES" -a -f ~/.use_screen ]; then # We'd rather use 'type -P' here, but that was added in bash-2.05b and we # use systems we don't control with versions older than that. We can't # easily use 'which' since on some systems that produces output whether # the file is found or not. for path in ${PATH//:/ }; do if [ -x "$path/screen" ]; then # If screen(1) exists and is executable, run our wrapper [ -x "$SETTINGS/run_screen" ] && $SETTINGS/run_screen fi done fiEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 17: Housekeeping and Administrative Tasks
- InhaltsvorschauThese recipes cover tasks that come up in the course of using or administering computers. They are presented here because they don't fit well anywhere else in the book.You want to rename many files, but
mv *.foo *.bardoesn't work. Or, you want to rename a group of files in arbitrary ways.We presented a simple loop to change file extensions in , "Changing Pieces of a String"; see that recipe for more details. Here is aforloop example:for FN in *.bad do mv "${FN}" "${FN%bad}bash" doneWhat about more arbitrary changes? For example, say you are writing a book and want the chapter file names to follow a certain format, but the publisher has a conflicting format. You could name the files like chNN=Title=Author.odt, then use a simpleforloop and cut in a command substitution to rename them.$ for i in *.odt; do mv "$i" "$(echo $i | cut -d'=' -f1,3)"; doneYou should always use quotes around file arguments in case there's a space. While testing the code in the solution we also used echo and angle brackets to make it very clear what the arguments are (usingset -xis also helpful).Once we were very sure our command worked, we removed the angle brackets and replaced echo with mv.# Testing$ for i in *.odt; do echo "<$i>" "<$(echo $i | cut -d'=' -f1,3)>"; done <ch01=Beginning Shell Scripting=JP.odt> <ch01=JP.odt> <ch02=Standard Output=CA.odt> <ch02=CA.odt> <ch03=Standard Input=CA.odt> <ch03=CA.odt> <ch04=Executing Commands=CA.odt> <ch04=CA. odt> [...] # Even more testing $ set -x $ for i in *.odt; do echo "<$i>" "<$(echo $i | cut -d'=' -f1,3)>"; done ++xtrace 1: echo ch01=Beginning Shell Scripting=JP.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch01=Beginning Shell Scripting=JP.odt>' '<ch01=JP.odt>' <ch01=Beginning Shell Scripting=JP.odt> <ch01=JP.odt> ++xtrace 1: echo ch02=Standard Output=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch02=Standard Output=CA.odt>' '<ch02=CA.odt>' <ch02=Standard Output=CA.odt> <ch02=CA.odt> ++xtrace 1: echo ch03=Standard Input=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch03=Standard Input=CA.odt>' '<ch03=CA.odt>' <ch03=Standard Input=CA.odt> <ch03=CA.odt> ++xtrace 1: echo ch04=Executing Commands=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch04=Executing Commands=CA.odt>' '<ch04=CA.odt>' <ch04=Executing Commands=CA.odt> <ch04=CA.odt> $ set +x +xtrace 536: set +x
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Renaming Many Files
- InhaltsvorschauYou want to rename many files, but
mv *.foo *.bardoesn't work. Or, you want to rename a group of files in arbitrary ways.We presented a simple loop to change file extensions in , "Changing Pieces of a String"; see that recipe for more details. Here is aforloop example:for FN in *.bad do mv "${FN}" "${FN%bad}bash" doneWhat about more arbitrary changes? For example, say you are writing a book and want the chapter file names to follow a certain format, but the publisher has a conflicting format. You could name the files like chNN=Title=Author.odt, then use a simpleforloop and cut in a command substitution to rename them.$ for i in *.odt; do mv "$i" "$(echo $i | cut -d'=' -f1,3)"; doneYou should always use quotes around file arguments in case there's a space. While testing the code in the solution we also used echo and angle brackets to make it very clear what the arguments are (usingset -xis also helpful).Once we were very sure our command worked, we removed the angle brackets and replaced echo with mv.# Testing$ for i in *.odt; do echo "<$i>" "<$(echo $i | cut -d'=' -f1,3)>"; done <ch01=Beginning Shell Scripting=JP.odt> <ch01=JP.odt> <ch02=Standard Output=CA.odt> <ch02=CA.odt> <ch03=Standard Input=CA.odt> <ch03=CA.odt> <ch04=Executing Commands=CA.odt> <ch04=CA. odt> [...] # Even more testing $ set -x $ for i in *.odt; do echo "<$i>" "<$(echo $i | cut -d'=' -f1,3)>"; done ++xtrace 1: echo ch01=Beginning Shell Scripting=JP.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch01=Beginning Shell Scripting=JP.odt>' '<ch01=JP.odt>' <ch01=Beginning Shell Scripting=JP.odt> <ch01=JP.odt> ++xtrace 1: echo ch02=Standard Output=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch02=Standard Output=CA.odt>' '<ch02=CA.odt>' <ch02=Standard Output=CA.odt> <ch02=CA.odt> ++xtrace 1: echo ch03=Standard Input=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch03=Standard Input=CA.odt>' '<ch03=CA.odt>' <ch03=Standard Input=CA.odt> <ch03=CA.odt> ++xtrace 1: echo ch04=Executing Commands=CA.odt ++xtrace 1: cut -d= -f1,3 +xtrace 535: echo '<ch04=Executing Commands=CA.odt>' '<ch04=CA.odt>' <ch04=Executing Commands=CA.odt> <ch04=CA.odt> $ set +x +xtrace 536: set +x
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using GNU Texinfo and Info on Linux
- InhaltsvorschauYou are having trouble accessing documentation because much of the documentation for GNU tools on Linux are in Texinfo documents, the traditional manpages are just a stub, and the default info program is user-hostile (and you don't feel like learning yet another single-use program).Pipe the info command into a useful pager, such as less.
$ info bash | less
info is basically a stand-alone version of the Emacs info reader, so if you are an Emacs fan, maybe it will make sense to you. However, piping it into less is a quick and simple way to view the documentation using a tool with which you're already familiar.The idea behind Texinfo is good: generate various output formats from a single source. It's not new, since many other mark-up languages exist to do the same thing; we even talk about one in , "Embedding Documentation in Shell Scripts." But if that's the case, why isn't there a TeX to man output filter? Perhaps because manpages follow a standard, structured, and time-tested format while Texinfo is more free form.There are other Texinfo viewers and converters if you don't like info, such as pinfo, info2www, tkman, and even info2man (which cheats and converts to POD and then to manpage format).- man info
- man man
- , "Embedding Documentation in Shell Scripts"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Unzipping Many ZIP Files
- InhaltsvorschauYou want to unzip many ZIP files in a directory, but
unzip *.zipdoesn't work.Put the pattern in single quotes:unzip '*.zip'
You could also use a loop to unzip each file:for x in /path/to/date*/name/*.zip; do unzip "$x"; done
or:for x in $(ls /path/to/date*/name/*.zip 2>/dev/null); do unzip $x; done
Unlike many Unix commands (e.g., gzip and bzip2), the last argument to unzip isn't an arbitrarily long list of files. To process the commandunzip *.zip, the shell expands the wildcard, so (assuming you have files named zipfile1.zip to zipfile4.zip)unzip*.zipexpands tounzip zipfile1.zip zipfile2.zip zipfile3.zip zipfile4.zip. This command attempts to extract zipfile2.zip, zipfile3.zip, and zipfile4.zip from zipfile1.zip. That command will fail unless zipfile1.zip actually contains files with those names.The first method prevents the shell from expanding the wildcard by using single quotes. However, that only works if there is only one wildcard. The second and third methods work around that by running an explicit unzip command for each ZIP file found when the shell expands the wildcards, or returns the result of the ls command.The ls version is used because the default behavior of bash (and sh) is to return unmatched patterns unchanged. That means you would be trying to unzip a file called /path/to/date*/name/*.zip if no files match the wildcard pattern. ls will simply return null on STDOUT, and an error that we throw away on STDERR. You can set theshopt -s nullgloboption to cause filename patterns that match no files to expand to a null string, rather than themselves.- man unzip
- , "Working Around "argument list too long" Errors"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Recovering Disconnected Sessions Using screen
- InhaltsvorschauYou run long processes over SSH, perhaps over the WAN, and when you get disconnected you lose a lot of work. Or perhaps you started a long job from work, but need to go home and be able to check on the job later; you could run your process using nohup, but then you won't be able to reattach to it when your connection comes back or you get home.Install and use GNU screen.Using screen is very simple. Type
screenorscreen -a. The-aoption includes all of screen's capabilities even at the expense of some redraw (thus bandwidth) efficiency. Honestly, we use-abut have never noticed a difference.When you do this, it will look like nothing happened, but you are now running inside a screen.echo $SHLVLshould return a number greater than one if this worked (see also :L$SHLVLin , "Customizing Your Prompt"). To test it, do anls -la, thenkillyour terminal (do not exit cleanly, as you will exit screen as well). Log back into the machine and typescreen -rto reconnect to screen. If that doesn't put you back where you left off, tryscreen -d -r. If that doesn't work, tryps auwx | grep [s]creento see if screen is still running, and then tryman screenfor troubleshooting information—but it should just work. If you run into problems with that ps command on a system other than Linux, see , "Finding Out Whether a Process Is Running."Starting screen with something like the following will make it easier to figure out what session to reattach to later if necessary:screen -aS "$(whoami).$(date' +%Y-%m-%d_%H:%M:%S%z')". See the run_screen script in , "Getting Started with a Custom Configuration."To exit out of screen and your session, keep typingexituntil all the sessions are gone. You can also type Ctrl-A Ctrl-\ or Ctrl-A :quit to exit screen itself (assuming you haven't changed the default meta-key of Ctrl-A yet).According to the screen web site:Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). Each virtual terminal provides the functions of a DEC VT100 terminal and, in addition, several control functions from the ISO 6429 (ECMA 48, ANSI X3.64) and ISO 2022 standards (e.g., insert/delete line and support for multiple character sets). There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows moving text regions between windows.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Sharing a Single bash Session
- InhaltsvorschauYou need to share a single bash session for training or troubleshooting purposes, and there are too many people for "over the shoulder" to work. Or you need to help someone who's located somewhere else, and you need to share a session across a network.Use GNU screen in multiuser mode. The following assumes that you have not changed the default metakey from Ctrl-A as described in , "Recovering Disconnected Sessions Using screen." If you have, then use your new metakey (e.g., Ctrl-N) instead.As the host do the following:
screen -Ssession_name (no spaces allowed); e.g.,screen -S training.- Ctrl-A :addacl usernames of accounts (comma delimited, no spaces!) which may access the display; e.g., Ctrl-A: addacl alice,bob,carol. Note this allows full read/write access.
- Use the Ctrl-A: chacl usernames permbits list command to refine permissions if needed.
- Turn on multiuser mode: Ctrl-A: multiuser on.
As the viewer, do this:- Use
screen -xuser/name to connect to a shared screen; e.g.,screen -x host/training. - Hit Ctrl-A K to kill the window and end the session.
See , "Recovering Disconnected Sessions Using screen," for necessary details.For multiuser mode, /tmp/screens must exist and be world-readable and executable.screen versions 3.9.15-8 to 4.0.1-1 from Red Hat (i.e., RHEL3) are broken and should not be used if you want multiuser mode to work. Version 4.0.2-5 or later should work; for example, http://mirror.centos.org/centos/4.2/os/i386/CentOS/RPMS/screen-4.0.2-5.i386.rpm (or later) works even on RHEL3. Once you start using the new version of screen, existing screen sockets in $HOME/.screen are not found and are thus orphaned and unusable. Log out of all sessions, and use the new version to create new sockets in /tmp/screens/S-$USER, then remove the $HOME/.screen directory.- man screen
- , "Finding a File Using a List of Possible Locations"
- , "Getting Started with a Custom Configuration"
- , "Recovering Disconnected Sessions Using screen"
- , "Logging an Entire Session or Batch Job"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Logging an Entire Session or Batch Job
- InhaltsvorschauYou need to capture all the output from an entire session or a long batch job.There are many ways to solve this problem, depending on your needs and environment.The simplest solution is to turn on logging to memory or disk in your terminal program. The problems with that are that your terminal program may not allow that, and when it gets disconnected you lose your log.The next simplest solution is to modify the job to log itself, or redirect the entire thing to tee or a file. For example, one of the following might work:
$ long_noisy_job >& log_file $ long_noisy_job 2>&1 | tee log_file $ ( long_noisy_job ) >& log_file $ ( long_noisy_job ) 2>&1 | tee log_file
The problems here are that you may not be able to modify the job, or the job itself may do something that precludes these solutions (e.g., if it requires user input, it could get stuck asking for the input before the prompt is actually displayed). That can happen because STDOUT is buffered, so the prompt could be in the buffer waiting to be displayed when more data comes in, but no more data will come in since the program is waiting for input.There is an interesting program called script that exists for this very purpose and it's probably already on your system. You run script, and it logs everything that happens to the logfile (called a typescript) you've given it, which is OK if you want to log the entire session—just start script, then run your job. But if you only want to capture part of the session, there is no way to have your code start script, run something to log it, then stop script again. You can't script script because once you run it, you're in a subshell at a prompt (i.e., you can't do something likescriptfile_to_log_to some_command_to_run).Our final solution uses the terminal multiplexer screen. With screen, you can turn whole session logging on or off from inside your script. Once you are already running screen, do the following in your script:# Set a logfile and turn on logging screen -X logfile /path/to/logfile && screen -X log on # your commands here # Turn logging back off screen -X log off
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Clearing the Screen When You Log Out
- InhaltsvorschauYou use or administer some systems that do not clear the screen when you log out, and you'd rather not leave the tail end of whatever you were working on visible, since that could be an information leak.Put the clear command in your ~/.bash_logout:.
# ~/.bash_logout # Clear the screen on exit from the shell to prevent information leaks, # if not already set as an exit trap in bash_profile [ "$PS1" ] && clear
Or set a trap to run clear on shell termination:# ~/.bash_profile # Trap to clear the screen on exit from the shell to prevent # information leaks, if not already set in ~/.bash_logout trap ' [ "$PS1" ] && clear ' 0
Note that if you are connecting remotely and your client has a scroll-back buffer, whatever you were working on may still be in there. clear also has no effect on your shell's command history.Setting a trap to clear the screen is probably overkill, but could conceivably cover an error situation in which ~/.bash_logout is not executed. If you are really paranoid you can set both, but in that case you may also wish to look into TEMPEST and Faraday cages.If you skip the test to determine whether the shell is interactive, you'll get errors like this under some circumstances:# e.g., from tput No value for $TERM and no -T specified # e.g., from clear TERM environment variable not set.
- , "Getting Started with a Custom Configuration"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Capturing File Metadata for Recovery
- InhaltsvorschauYou want to create a list of files and details about them for archive purposes, for example, to verify backups, re-create directories, etc. Or maybe you are about to do a large
chmod -Rand need a back-out plan. Or perhaps you keep /etc/* in a revision control system that does not preserve permissions or ownership.Use GNU find with some printf formats:#!/usr/bin/env bash # cookbook filename: archive_meta-data printf "%b" "Mode\tUser\tGroup\tBytes\tModified\tFileSpec\n" > archive_file find / \( -path /proc -o -path /mnt -o -path /tmp -o -path /var/tmp \ -o -path /var/cache -o -path /var/spool \) -prune \ -o -type d -printf 'd%m\t%u\t%g\t%s\t%t\t%p/\n' \ -o -type l -printf 'l%m\t%u\t%g\t%s\t%t\t%p -> %l\n' \ -o -printf '%m\t%u\t%g\t%s\t%t\t%p\n' \) >> archive_file
Note that the-printfexpression is in the GNU version of find.The (-path /foo-o -path…)-prunepart removes various directories you probably don't want to bother with, e.g.,-type dis for directories. The printf format is prefixed with ad, then uses an octal mode, user, group, and so forth.-type 1is for symbolic links and also shows you where the link points. With the contents of this file and some additional scripting, you can determine at a high level if anything has changed, or re-create mangled ownership or permissions. Note that this does not take the place of more security-oriented programs like Tripwire, AIDE, Osiris, or Samhain.- man find
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Creating an Index of Many Files
- InhaltsvorschauYou have a number of files for which you'd like to create an index.Use the find command in conjunction with head, grep, or other commands that can parse out comments or summary information from each file.For example, if the second line of all your shell scripts follows the format "name— description" then this example will create a nice index:
$ for i in $(grep -El '#![[:space:]]?/bin/sh' *); do head -2 $i | tail -1; done
As noted, this technique depends on each file having some kind of summary information, such as comments, that may be parsed out. We then look for a way to identify the type of file, in this case a shell script, and grab the second line of each file.If the files do not have easily parsed summary information, you can try something like this and manually work through the output to create an index:for dir in $(find . -type d); do head -15 $dir/*; done
Watch out for binary files!- man find
- man grep
- man head
- man tail
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using diff and patch
- InhaltsvorschauYou can never remember how to use diff to create patches that may later be applied using patch.If you are creating a simple patch for a single file, use:
$ diff -u original_file modified_file > your_patch
If you are creating a patch for multiple files in parallel directory structures, use:$ cp -pR original_dirs/ modified_dirs/ # Make changes here $ diff -Nru original_dirs/ modified_dirs/ > your_comprehensive_patch
To be especially careful, force diff to treat all files as ASCII using-a, and set your language and timezone to the universal defaults as shown:$ LC_ALL=C TZ=UTC diff -aNru original_dirs/ modified_dirs/ > your_comprehensive_patch $ LC_ALL=C TZ=UTC diff -aNru original_dirs/ modified_dirs/ diff -aNru original_dirs/changed_file modified_dirs/changed_file --- original_dirs/changed_file 2006-11-23 01:04:07.000000000 +0000 +++ modified_dirs/changed_file 2006-11-23 01:04:35.000000000 +0000 @@ -1,2 +1,2 @@ This file is common to both dirs. -But it changes from one to the other. +But it changes from 1 to the other. diff -aNru original_dirs/only_in_mods modified_dirs/only_in_mods --- original_dirs/only_in_mods 1970-01-01 00:00:00.000000000 +0000 +++ modified_dirs/only_in_mods 2006-11-23 01:05:58.000000000 +0000 @@ -0,0 +1,2 @@ +While this file is only in the modified dirs. +It also has two lines, this is the last. diff -aNru original_dirs/only_in_orig modified_dirs/only_in_orig --- original_dirs/only_in_orig 2006-11-23 01:05:18.000000000 +0000 +++ modified_dirs/only_in_orig 1970-01-01 00:00:00.000000000 +0000 @@ -1,2 +0,0 @@ -This file is only in the original dirs. -It has two lines, this is the last.
To apply a patch file, cd to the directory of the single file, or to the parent of the directory tree and use the patch command:cd /path/to/files patch -Np1 < your_patch
The-Nargument to patch prevents it from reversing patches or re-applying patches that have already been made.-pnumber removes number of leading directories to allow for differences in directory structure between whoever created the patch and whoever is applying it. UsingEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Counting Differences in Files
- InhaltsvorschauYou have two files and need to know about how many differences exist between them.Count the hunks (i.e., sections of changed data) in diff's output:
$ diff -C0 original_file modified_file | grep -c "^\*\*\*\*\*" 2 $ diff -C0 original_file modified_file *** original_file Fri Nov 24 12:48:35 2006 --- modified_file Fri Nov 24 12:48:43 2006 *************** *** 1 **** ! This is original_file, and this line is different. --- 1 --- ! This is modified_file, and this line is different. *************** *** 6 **** ! But this one is different. --- 6 --- ! But this 1 is different.
If you only need to know whether the files are different and not how many differences there are, use cmp. It will exit at the first difference, which can save time on large files. Like diff it is silent when the files are identical, but it reports the location of the first difference if not:$ cmp original_file modified_file original_file modified_file differ: char 9, line 1
Hunk is actually the technical term, though we've also seen hunks referred to as chunks in some places. Note that it is possible, in theory, to get slightly different results for the same files across different machines or versions of diff, since the number of hunks is a result of the algorithm diff uses. You will certainly get different answers when using different diff output formats, as demonstrated below.We find a zero-context contextual diff to be the easiest to use for this purpose, and using-C0instead of-ccreates fewer lines for grep to have to search. A unified diff tends to combine more changes than expected into one hunk, leading to fewer differences being reported:$ diff -u original_file modified_file | grep -c "^@@" 1 $ diff -u original_file modified_file --- original_file 2006-11-24 12:48:35.000000000 -0500 +++ modified_file 2006-11-24 12:48:43.000000000 -0500 @@ -1,8 +1,8 @@ -This is original_file, and this line is different. +This is modified_file, and this line is different. This line is the same. So is this one. And this one. Ditto. -But this one is different. +But this 1 is different. However, not this line. And this is the last same, same, same.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Removing or Renaming Files Named with Special Characters
- InhaltsvorschauYou need to remove or rename a file that was created with a special character that causes rm or mv to behave in unexpected ways. The canonical example of this is any file starting with a dash, such as -f or --help, which will cause any command you try to use to interpret the filename as an argument.If the file begins with a dash, use -- to signal the end of arguments to the command, or use a full (/tmp/-f) or relative (./-f) path. If the file contains other special characters that are interpreted by the shell, such as a space or asterisk, use shell quoting. If you use filename completion (the Tab key by default), it will automatically quote special characters for you. You can also use single-quotes around the troublesome name.
$ ls --help this is a *crazy* file name! $ mv --help help mv: unknown option -- - usage: mv [-fiv] source target mv [-fiv] source ... directory $ mv -- --help my_help $ mv this\ is\ a\ \*crazy\*\ file\ name\! this_is_a_better_name $ ls my_help this_is_a_better_name
To understand what is actually being executed after shell expansion, preface your command with echo:$ rm * rm: unknown option -- - usage: rm [-f|-i] [-dPRrvW] file ... $ echo rm * rm --help this is a *crazy* file name!- Sections 2.1 and 2.2 of http://www.faqs.org/faqs/unix-faq/faq/part2/
- , "Using Shell Quoting"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Prepending Data to a File
- InhaltsvorschauYou want to prepend data to an existing file, for example to add a header after sorting.Use cat in a subshell.
temp_file="temp.$RANDOM$RANDOM$$" (echo 'static header line1'; cat data_file) > $temp_file \ && cat $temp_file > data_file rm $temp_file unset temp_file
You could also use sed, the streaming editor. To prepend static text, note that back-slash escape sequences are expanded in GNU sed but not in some other versions. Also, under some shells the trailing backslashes may need to be doubled:# Any sed, e.g., Solaris 10 /usr/bin/sed $ sed -e '1i\ > static header line1 > ' data_file static header line1 1 foo 2 bar 3 baz $ sed -e '1i\ > static header line1\ > static header line2 > ' data_file static header line1 static header line2 1 foo 2 bar 3 baz # GNU sed $ sed -e '1istatic header line1\nstatic header line2' data_file static header line1 static header line2 1 foo 2 bar 3 baz
To prepend an existing file:$ sed -e '$r data_file' header_file Header Line1 Header Line2 1 foo 2 bar 3 baz
This one seems to be a love/hate kind of thing. People either love the cat solution or love the sed solution, but not both. The cat version is probably faster and simpler, the sed solution is arguably more flexible.You can also store a sed script in a file, instead of leaving it on the command line. And of course you would usually redirect the output into a new file, likesed -e '$rdata' header> new_file, but note that will change the file's inode and may change other attributes such as permissions or ownership. To preserve everything but the inode, use-ifor in-place editing if your version of sed supports that. Don't use-iwith the reversed header file prepend form shown previously or you will edit your header file. Also note that Perl has a similar-ioption that also writes a new file like sed, though Perl itself works rather differently than sed for this example:# Show inode $ ls -i data_file 509951 data_file $ sed -i -e '1istatic header line1\nstatic header line2' data_file $ cat data_file static header line1 static header line2 1 foo 2 bar 3 baz # Verify inode has changed $ ls -i data_file 509954 data_file
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Editing a File in Place
- InhaltsvorschauYou want to edit an existing file without affecting the inode or permissions.This is trickier than it sounds because many tools you might ordinarily use, such as sed, will write to a new file (thus changing the inode) even if they go out of their way to preserve other attributes.The obvious solution is to simply edit the file and make your updates. However, we admit that that may be of limited use in a scripting situation. Or is it?In , "Prepending Data to a File," you saw that sed writes a brand new file one way or another; however, there is an ancestor of sed that doesn't do that. It's called, anticlimactically, ed, and it is just as ubiquitous as its other famous descendant, vi. And interestingly, ed is scriptable. So here is our "prepend a header" example again, this time using ed:
# Show inode $ ls -i data_file 306189 data_file # Use printf "%b" to avoid issues with 'echo -e' or not. $ printf "%b" '1\ni\nHeader Line1\nHeader Line2\n.\nw\nq\n' | ed -s data_file 1 foo $ cat data_file Header Line1 Header Line2 1 foo 2 bar 3 baz # Verify inode has NOT changed $ ls -i data_file 306189 data_file
Of course you can store an ed script in a file, just as you can with sed. In this case, it might be useful to see what that file looks like, to explain the mechanics of the ed script:$ cat ed_script 1 i Header Line1 Header Line2 . w q $ ed -s data_file < ed_script 1 foo $ cat data_file Header Line1 Header Line2 1 foo 2 bar 3 baz
The1in the ed script means to go to the first line.iputs us into insert mode, and the next two lines are literal. A single . all by itself on a line exits insert mode,wwrites the file andqquits. The-ssuppresses some output, specifically for use in scripts, but you can see from the1 foothat not everything is suppressed; of course,ed -sdata_file< ed_script>/dev/nulltakes care of that.One disadvantage to ed is that there isn't that much documentation for it anymore. It's been around since the beginning of Unix, but it's not commonly used anymore even though it exists on every system we checked. Since bothEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Using sudo on a Group of Commands
- InhaltsvorschauYou are running as a regular user and need to sudo several commands at once, or you need to use redirection that applies to the commands and not to sudo.Use sudo to run a subshell in which you may group your commands and use pipe-lines and redirection:
sudo bash -c 'command1 && command2 || command3'This requires the ability to run a shell as root. If you can't, have your system administrator write a quick script and add it to your sudo privilege specification.If you try something likesudocommand1 && command2|| command3 you'll find that command2 and command3 are running as you, not as root. That's because sudo's influence only extends to the first command and your shell is doing the redirection.Note the use of the-cargument to bash, which causes it to just execute the given commands and exit. Without that you will just end up running a new interactive root shell, which is probably not what you wanted. But as noted above, with-cyou are still running a (non-interactive) root shell, so you need to have the sudo rights to do that. Mac OS X and some Linux distributions, such as Ubuntu, actually disable the root user to encourage you to only log in as a normal user and sudo as needed (the Mac hides this better) for administration. If you are using an OS like that, or have rolled your own sudo setup, you should be fine. However, if you are running a locked-down environment, this recipe may not work for you.To learn whether you may use sudo and what you are and are not allowed to do, usesudo -l. Almost any other use of sudo will probably trigger a security message to your administrator tattling on you. You can try usingsudo sudo-V | lessas a regular user or justsudo -V | lessif you are already root to get a lot of information about how sudo is compiled and configured on your system.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Lines in One File But Not in the Other
- InhaltsvorschauYou have two data files and you need to compare them and find lines that exist in one file but not in the other.Sort the files and isolate the data of interest using cut or awk if necessary, and then use comm, diff, grep, or uniq depending on your needs.comm is designed for just this type of problem:
$ cat left record_01 record_02.left only record_03 record_05.differ record_06 record_07 record_08 record_09 record_10 $ cat right record_01 record_02 record_04 record_05 record_06.differ record_07 record_08 record_09.right only record_10 # Only show lines in the left file $ comm -23 left right record_02.left only record_03 record_05.differ record_06 record_09 # Only show lines in the right file $ comm -13 left right record_02 record_04 record_05 record_06.differ record_09.right only # Only show lines common to both files $ comm -12 left right record_01 record_07 record_08 record_10
diff will quickly show you all the differences from both files, but its output is not terribly pretty and you may not need to know all the differences. GNU grep's-yand-woptions can be handy for readability, but you can get used to the regular output as well. Some systems (e.g., Solaris) may use sdiff instead ofdiff-yor have a separate binary such as bdiff to process very large files.$ diff -y -W 60 left right record_01 record_01 record_02.left only | record_02 record_03 | record_04 record_05.differ | record_05 record_06 | record_06.differ record_07 record_07 record_08 record_08 record_09 | record_09.right only record_10 record_10 $ diff -y -W 60 --suppress-common-lines left right record_02.left only | record_02 record_03 | record_04 record_05.differ | record_05 record_06 | record_06.differ record_09 | record_09.right only $ diff left right 2,5c2,5 < record_02.left only < record_03 < record_05.differ < record_06 --- > record_02 > record_04 > record_05 > record_06.differ 8c8 < record_09 --- > record_09.right only
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Keeping the Most Recent N Objects
- InhaltsvorschauYou need to keep the most recent N logfiles or backup directories, and purge the remainder, no matter how many there are.Create an ordered list of the objects, pass them as arguments to a function, shift the arguments by N, and return the remainder:
# cookbook filename: func_shift_by # Pop a given number of items from the top of a stack, # such that you can then perform an action on whatever is left. # Called like: shift_by <# to keep> <ls command, or whatever> # Returns: the remainder of the stack or list # # For example, list some objects, then keep only the top 10. # # It is CRITICAL that you pass the items in order with the objects to # be removed at the top (or front) of the list, since all this function # does is remove (pop) the number of entries you specify from the top # of the list. # # You should experiment with echo before using rm! # # For example: # rm -rf $(shift_by $MAX_BUILD_DIRS_TO_KEEP $(ls -rd backup.2006*)) #function shift_by { # If $1 is zero or greater than $#, the positional parameters are # not changed. In this case that is a BAD THING! if (( $1 == 0 || $1 > ( $# - 1 ) )); then echo '' else # Remove the given number of objects (plus 1) from the list. shift $(( $1 + 1 )) # Return whatever is left echo "$*" fi }If you try to shift the positional parameters by zero or by more than the total number of positional parameters ($#), shift will do nothing. If you are using shift to process a list then delete what it returns, that will result in you deleting everything. Make sure to test the argument to shift to make sure that it's not zero and it is greater than the number of positional parameters. Our shift_by function does this.For example:$ source shift_by $ touch {1..9} $ ls ? 1 2 3 4 5 6 7 8 9 $ shift_by 3 $(ls ?) 4 5 6 7 8 9 $ shift_by 5 $(ls ?) 6 7 8 9 $ shift_by 5 $(ls -r ?) 4 3 2 1 $ shift_by 7 $(ls ?) 8 9 $ shift_by 9 $(ls ?) # Keep only the last 5 objects $ echo "rm -rf $(shift_by 5 $(ls ?))" rm -rf 6 7 8 9 # In production we'd test this first! See discussion. $ rm -rf $(shift_by 5 $(ls ?)) $ ls ? 1 2 3 4 5Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Grepping ps Output Without Also Getting the grep Process Itself
- InhaltsvorschauYou want to grep output from the ps command without also getting the grep process itself.Change the pattern you are looking for so that it is a valid regular expression that will not match the literal text that ps will display:
$ ps aux | grep 'ssh' root 366 0.0 1.2 340 1588 ?? Is 20Oct06 0:00.68 /usr/sbin/sshd root 25358 0.0 1.9 472 2404 ?? Ss Wed07PM 0:02.16 sshd: root@ttyp0 jp 27579 0.0 0.4 152 540 p0 S+ 3:24PM 0:00.04 grep ssh $ ps aux | grep '[s]sh' root 366 0.0 1.2 340 1588 ?? Is 20Oct06 0:00.68 /usr/sbin/sshd root 25358 0.0 1.9 472 2404 ?? Ss Wed07PM 0:02.17 sshd: root@ttyp0This works because[s]is a regular expression character class containing a single lowercase letters, meaning that[s]shwill matchsshbut not the literal stringgrep [s]shthat ps will display.The other less efficient and more clunky solution you might see is something like this:$ ps aux | grep 'ssh' | grep -v grep
- man ps
- man grep
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finding Out Whether a Process Is Running
- InhaltsvorschauYou need to determine whether a process is running, and you might or might not already have a process ID (PID).If you don't already have a PID, grep the output of the ps command to see if the program you are looking for is running. See , "Grepping ps Output With-out Also Getting the grep Process Itself," for details on why our pattern is
[s]sh.$ [ "$(ps -ef | grep 'bin/[s]shd')" ] && echo 'ssh is running' || echo 'ssh not running'
That's nice, but you know it's not going to be that easy, right? Right. It's difficult because ps can be wildly different from system to system.# cookbook filename: is_process_running # Can you believe this?!? case `uname` in Linux|AIX) PS_ARGS='-ewwo pid,args' ;; SunOS) PS_ARGS='-eo pid,args' ;; *BSD) PS_ARGS='axwwo pid,args' ;; Darwin) PS_ARGS='Awwo pid,command' ;; esac if ps $PS_ARGS | grep -q 'bin/[s]shd'; then echo 'sshd is running' else echo 'sshd not running' fi
If you do have a PID, say from a lock file or an environment variable, just search for it. Be careful to match the PID up with some other recognizable string so that you don't have a collision where some other random process just happens to have the stale PID you are using. Just obtain the PID and use it in the grep or in a-pargument to ps:# Linux $ ps -wwo pid,args -p 1394 | grep 'bin/sshd' 1394 /usr/sbin/sshd # BSD $ ps ww -p 366 | grep 'bin/sshd' 366 ?? Is 0:00.76 /usr/sbin/sshd
The test and grep portion of the solution requires a little explanation. You need " " around the $( ) so that if grep outputs anything, the test is true. If the grep is silent because nothing matches, then the test is false. You just have to make sure your ps and greps do exactly what you want.Unfortunately, the ps command is one of the most fragmented in all of Unix. It seems like every flavor of Unix and Linux has different arguments and processes them in different ways. All we can tell you is that you'll need to thoroughly test against all systems on which your script will be running.You can easily search for anything you can express as a regular expression, but make sure your expressions are specific enough not to match anything else. That's why we usedEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adding a Prefix or Suffix to Output
- InhaltsvorschauYou'd like to add a prefix or a suffix to each line of output from a given command for some reason. For example, you're collecting last statistics from many machines and it's much easier to grep or otherwise parse the data you collect if each line contains the hostname.Pipe the appropriate data into a
while readloop and printf as needed. For example, this prints the$HOSTNAME, followed by a tab, followed by any nonblank lines of out-put from the last command:$ last | while read i; do [[ -n "$i" ]] && printf "%b" "$HOSTNAME\t$i\n"; done # Write a new logfile $ last | while read i; do [[ -n "$i" ]] && printf "%b" "$HOSTNAME\t$i\n"; done > last_$HOSTNAME.log
Or you can use awk to add text to each line:$ last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \"$HOSTNAME\", \$0}" $ last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \"$HOSTNAME\", \$0}" \ > last_$HOSTNAME.logWe use [[ -n "$i" ]] to remove any blank lines from the last output, and then we use printf to display the data. Quoting for this method is simpler, but it uses more steps (last,while, andread, as opposed to just last and awk). You may find one method easier to remember, more readable, or faster than the other, depending on your needs.There is a trick to the awk command we used here. Often you will see single quotes surrounding awk commands to prevent the shell from interpreting awk variables as shell variables. However in this case we want the shell to interpolate$HOSTNAME, so we surround the command with double quotes. That requires us to use backslash escapes on the elements of the command that we do not want the shell to handle, namely the internal double quotes and the awk$0variable, which contains the current line.For a suffix, simply move the$0variable:$ last | while read i; do [[ -n "$i" ]] && printf "%b" "$i\t$HOSTNAME\n"; done $ last | awk "BEGIN { OFS=\"\t\" } ! /^\$/ { print \"$HOSTNAME\", \$0}"You could also use Perl or sed (note the → denotes a literal tab character, typed by pressing Ctrl-V then Ctrl-I):$ last | perl -ne "print qq($HOSTNAME\t\$_) if ! /^\s*$/;" $ last | sed "s/./$HOSTNAME → &/; /^$/d"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Numbering Lines
- InhaltsvorschauYou need to number the lines of a text file for reference or for use as an example.Thanks to Michael Wang for contributing the following shell-only implementation and reminding us about
cat -n. Note that our sample file named lines has a trailing blank line:$ i=0; while IFS= read -r line; do (( i++ )); echo "$i $line"; done < lines 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6
Or a useful use of cat:$ cat -n lines 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6 $ cat -b lines 1 Line 1 2 Line 2 3 Line 4 4 Line 5
If you only need to display the line numbers on the screen, you can useless -N:$ /usr/bin/less -N filename 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6 lines (END)
Line numbers are broken in old versions of less on some obsolete Red Hat systems. Check your version withless -V. Version 358+iso254 (e.g., Red Hat 7.3 & 8.0) is known to be bad. Version 378+iso254 (e.g., RHEL3) and version 382 (RHEL4, Debian Sarge) are known to be good; we did not test other versions. The problem is subtle and may be related to an older iso256 patch. You can easily compare last line numbers as the vi and Perl examples are correct.You can also use vi (or view, which is read-only vi) with the :set nu!command:$ vi filename 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6 ~ :set nu!
vi has many options, so you can start vi by doing things likevi +3 -c 'set nu!'filename to turn on line numbering and place your cursor on line 3. If you'd like more control over how the numbers are displayed, you can also use nl, awk, or perl:$ nl lines 1 Line 1 2 Line 2 3 Line 4 4 Line 5 $ nl -ba lines 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6 $ awk '{ print NR, $0 }' filename 1 Line 1 2 Line 2 3 4 Line 4 5 Line 5 6 $ perl -ne 'print qq($.\t$_);' filename 1 → Line 1 2 → Line 2 3 → 4 → Line 4 5 → Line 5 6 →NRand $. are the line number in the current input file in awk and Perl respectively, so it's easy to use them to print the line number. Note that we are using a → to denote a Tab character in the Perl output, whileEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Writing Sequences
- InhaltsvorschauYou need to generate a sequence of numbers, possibly with other text, for testing or some other purpose.Use awk because it should work everywhere no matter what:
$ awk 'END { for (i=1; i <= 5; i++) print i, "text"}' /dev/null 1 text 2 text 3 text 4 text 5 text $ awk 'BEGIN { for (i=1; i <= 5; i+=.5) print i}' /dev/null 1 1.5 2 2.5 3 3.5 4 4.5 5On some systems, notably Solaris, awk will hang waiting for a file unless you give it one, such as /dev/null. This has no effect on other systems, so it's fine to use everywhere.Note that the variable in theprintstatement isi, not$i. If you accidentally use$iit will be interpolated as a field from the current line being processed. Since we're processing nothing, that's what you'll get if you use$iby accident (i.e., nothing).TheBEGINorENDpatterns allow for startup or cleanup operations when actually processing files. Since we're not processing a file, we need to use one of them so that awk knows to actually do something even though it has no normal input. In this case, it doesn't matter which we use.There is a GNU utility called seq that does exactly what this recipe calls for, but it does not exist by default on many systems, for example BSD, Solaris, and Mac OS X. It offers some useful formatting options and is numeric only.Thankfully, as of bash 2.04 and later, you can do arithmetic integerforloops:# Bash 2.04+ only, integer only $ for ((i=1; i<=5; i++)); do echo "$i text"; done 1 text 2 text 3 text 4 text 5 text
As of bash 3.0 and later, there is also the {x..y} brace expansion, which allows integers or single characters:# Bash 3.0+ only, integer or single character only $ printf "%s text\n" {1..5} 1 text 2 text 3 text 4 text 5 text $ printf "%s text\n" {a..e} a text b text c text d text e text- man seq
- man awk
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Emulating the DOS Pause Command
- InhaltsvorschauYou are migrating from DOS/Windows batch files and want to emulate the DOS
pausecommand.To do that, use theread -pcommand in a function:pause () { read -p 'Press any key when ready...' }The-poption followed by a string argument prints the string before reading input. In this case the string is the same as the DOSpausecommand's output.- help read
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Commifying Numbers
- InhaltsvorschauYou'd like to add a thousands-place separator to long numbers.Depending on your system and configuration, you may be able to use printf's ' format flag with a suitable local. Thanks to Chet Ramey for this solution, which is by far the easiest if it works:
$ LC_NUMERIC=en_US.UTF-8 printf "%'d\n" 123456789 123,456,789 $ LC_NUMERIC=en_US.UTF-8 printf "%'f\n" 123456789.987 123,456,789.987000
Thanks to Michael Wang for contributing the following shell-only implementation and relevant discussion:# cookbook filename: func_commify function commify { typeset text=${1} typeset bdot=${text%%.*} typeset adot=${text#${bdot}} typeset i commified (( i = ${#bdot} - 1 )) while (( i>=3 )) && [[ ${bdot:i-3:1} == [0-9] ]]; do commified=",${bdot:i-2:3}${commified}" (( i -= 3 )) done echo "${bdot:0:i+1}${commified}${adot}" }The shell function is written to follow the same logical process as a person using a pencil and paper. First you examine the string and find the decimal point, if any. You ignore everything after the dot, and work on the string before the dot.The shell function saves the string before the dot in$bdot, and after the dot (including the dot) in$adot. If there is no dot, then everything is in$bdot, and$adotis empty. Next a person would move from right to left in the part before the dot and insert a comma when these two conditions are met:- There are four or more characters left.
- The character before the comma is a number.
The function implements this logic in thewhileloop.Tom Christiansen and Nathan Torkington's Perl Cookbook, Second Edition (O'Reilly), also provides a string processing solution:# cookbook filename: perl_sub_commify #+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ # Add comma thousands separator to numbers # Returns: input string, with any numbers commified # From Perl Cookbook2 2.16, pg 84 sub commify { @_ == 1 or carp ('Sub usage: $withcomma = commify($somenumber);'); # From _Perl_Cookbook_1 page 64, 2.17 or _Perl_Cookbook_2 page 84, 2.16 my $text = reverse $_[0]; $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; return scalar reverse $text; }Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 18: Working Faster by Typing Less
- InhaltsvorschauDespite all the improvements in processor speed, transmission rates, network speed, and I/O capabilities, there is still a limiting factor in many uses of bash—the typing speed of the user. Scripting has been our focus, of course, but interactive use of bash is still a significant part of its use and usefulness. Many of the scripting techniques we have described can be used interactively as well, but then you find yourself faced with a lot of typing, unless you know some shortcuts.Now "back in the day," when Unix was first invented, there were teletype machines that could only crank out about 10 characters per second, and a good touch typist could type faster than the keyboard could handle it. It was in this milieu that Unix was developed and some of its terseness is likely due to the fact that no one wanted to type more than absolutely necessary to get across their command.At the other end of the historical perspective (i.e., now) processors are so fast that they can be quite idle while waiting for user input, and can look back through histories of previous commands as well as in directories along your
$PATHto find possible commands and valid arguments even before you finish typing them.Combining techniques developed for each of these situations, we can greatly reduce the amount of typing required to issue shell commands—and not just out of sheer laziness. Rather, you may quickly find that these keystroke-saving measures are so useful because of the increased accuracy they provide, the mistakes they help you avoid, and the backups that you don't need to reload.Do you find yourself moving frequently between two or more directories? Are you changing directories to here, then there, and then back again? Do you tire of always typing long path names since the directories never seem to be close by?Use the pushd and popd built-in commands to manage a stack of directory locations, and to switch between them easily. Here is a simple example:$ cd /tmp/tank $ pwd /tmp/tank $ pushd /var/log/cups /var/log/cups /tmp/tank $ pwd /var/log/cups $ ls access_log error_log page_log $ popd /tmp/tank $ ls empty full $ pushd /var/log/cups /var/log/cups /tmp/tank $ pushd /tmp/tank /var/log/cups $ pushd /var/log/cups /tmp/tank $ pushd /tmp/tank /var/log/cups $ dirs /tmp/tank /var/log/cups
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Moving Quickly Among Arbitrary Directories
- InhaltsvorschauDo you find yourself moving frequently between two or more directories? Are you changing directories to here, then there, and then back again? Do you tire of always typing long path names since the directories never seem to be close by?Use the pushd and popd built-in commands to manage a stack of directory locations, and to switch between them easily. Here is a simple example:
$ cd /tmp/tank $ pwd /tmp/tank $ pushd /var/log/cups /var/log/cups /tmp/tank $ pwd /var/log/cups $ ls access_log error_log page_log $ popd /tmp/tank $ ls empty full $ pushd /var/log/cups /var/log/cups /tmp/tank $ pushd /tmp/tank /var/log/cups $ pushd /var/log/cups /tmp/tank $ pushd /tmp/tank /var/log/cups $ dirs /tmp/tank /var/log/cups
Stacks are last in, first out mechanisms, which is how these commands behave. When you pushd to a new directory, it keeps the previous directory on a stack. Then when you popd, it pops the current location off of the stack and puts you back in that first location. When you change locations using these commands, they will print the values on the stack, left to right, corresponding to the top-to-bottom ordering of a stack.If you pushd without any directory, it swaps the top item on the stack with the next one down, so that you can alternate between two directories using repeated pushd commands with no arguments. You can do the same thing using thecd- command.You can still cd to locations—that will change the current directory, which is also the top of the directory stack. If you can't remember what is on your stack of directories, use the dirs command to echo the stack, left-to-right. For a more stack-like display, use the-voption:$ dirs -v 0 /var/tmp 1 ~/part/me/scratch 2 /tmp $
The tilde (~) is a shorthand for your home directory. The numbers can be used to reorder the stack. If youpushd +2then bash will put the #2 entry on the top of the stack (and cd you there) and push the others down:$ pushd +2 /tmp /var/tmp ~/part/me/scratch $ dirs -v 0 /tmp 1 /var/tmp 2 ~/part/me/scratch $
Once you get a little practice with these commands, you will find it much faster and easier to move repeatedly between directories.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Repeating the Last Command
- InhaltsvorschauYou just typed a long and difficult command line, one with long pathnames and complicated sets of arguments. Now you need to run it again. Do you have to type it all again?There are two very different solutions to this problem. First, just type two exclamation marks at the prompt, and bash will echo and repeat the previous command. For example:
$ /usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon ... $ !! /usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon ...
The other (more modern) solution involves using the arrow keys. Typing the uparrow key will scroll back through the previous commands that you have issued. When you find the one you want, just press the Enter key and that command will be run (again).The command is echoed when you type !! (sometimes called bang bang) so that you can see what is running.- , "Adjusting readline Behavior Using .inputrc"
- , "Setting Shell History Options"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Running Almost the Same Command
- InhaltsvorschauAfter running a long and difficult-to-type command, you get an error message indicating that you made one tiny little typo in the middle of that command line. Do you have to retype the whole line?The !! command that we discussed in , "Repeating the Last Command" allows you to add an editing qualifier. How good are your sed-like skills? Add a colon after the bang-bang and then a sed-like substitution expression, as in the following example:
$ /usr/bin/somewhere/someprog -g -H -yknot -w /tmp/soforthandsoon Error: -H not recognized. Did you mean -A? $ !!:s/H/A/ /usr/bin/somewhere/someprog -g -A -yknot -w /tmp/soforthandsoon ...
You can always just use the arrow keys to navigate your history and commands, but for long commands on slow links this syntax is great once you get used to it.If you're going to use this feature, just be careful with your substitutions. If you had tried to change the -goption by typing!!:s/g/h/you would have ended up changing the first letter g, which is at the end of the command name, and you would be trying to run/usr/bin/somewhere/someproh.The comparison with sed is apt here because the substitution is applied successively to each word in the command line. That means that the expressions that you use for substitutions cannot cross word boundaries. You could not, for example, use:s/-g -A/-gA/
as a command, since the -g and -A are separate words to bash.But that doesn't mean that your changes can't effect the whole line. If you want to change all occurrences of an expression in a command line, you need to precede theswith ag(for global substitution), as follows:$ /usr/bin/somewhere/someprog -g -s -yknots -w /tmp/soforthandsoon ... $ !!:gs/s/S/ /usr/bin/Somewhere/Someprog -g -S -yknotS -w /tmp/SoforthandSoon ...
Why does thisghave to appear before thesand not after it, like in sed syntax? Well, anything that appears after the closing slash is considered new text to append to the command—which is quite handy if you want to add another argument to the command when you run it again.- , "Adjusting readline Behavior Using .inputrc"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Substituting Across Word Boundaries
- InhaltsvorschauThe
!!:s/a/b/syntax is restricted to substitutions within a word; what if you need to make a substitution that crosses word boundaries?Use the caret (^) substitution mechanism:$ /usr/bin/somewhere/someprog -g -A -yknot -w /tmp/soforthandsoon ... $ ^-g -A^-gB^ /usr/bin/somewhere/someprog -gB -yknot -w /tmp/soforthandsoon
You can always just use the arrow keys to navigate your history and commands, but for long commands on slow links this syntax is great once you get used to it.Write the substitution on the command line by starting with a caret (^) and then the text you want replaced, then another caret and the new text. A trailing (third) caret is needed only if you want to add more text at the end of the line, as in:$ /usr/bin/somewhere/someprog -g -A -yknot ... $ ^-g -A^-gB^ /tmp^ /usr/bin/somewhere/someprog -gB -yknot /tmp
If you want to remove something, substitute an empty value; i.e., don't put anything for the new text. Here are two examples:$ /usr/bin/somewhere/someprog -g -A -yknot /tmp ... $ ^-g -A^^ /usr/bin/somewhere/someprog -yknot /tmp ... $ ^knot^ /usr/bin/somewhere/someprog -gA -y /tmp ... $
The first example uses all three carets. The second example leaves off the third caret; since we want to replace the "knot" with nothing, we just end the line with a newline (the Enter key).The use of the caret substitution not only spans word boundaries, it's just plain handy. Many bash users find it easier to use than!!:s/…/…/syntax. Wouldn't you agree?- , "Adjusting readline Behavior Using .inputrc"
- , "Setting Shell History Options"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Reusing Arguments
- InhaltsvorschauReusing the last command was easy with !! but you might not want the whole command. How can you reuse just the last argument?Use !$ to indicate the last command. Use
!:1for the first argument on the command line,!:2for the second, and so on.It is quite common to hand the same filename to a series of commands. One of the most common occurrences might be the way a programmer would edit and then compile, edit and then compile.… Here, the !$ comes in quite handy:$ vi /some/long/path/name/you/only/type/once ... $ gcc !$ gcc /some/long/path/name/you/only/type/once ... $ vi !$ vi /some/long/path/name/you/only/type/once ... $ gcc !$ gcc /some/long/path/name/you/only/type/once
Get the idea? It saves a lot of typing but it also avoids errors. If you mistype the filename when you compile, then you are not compiling the file that you just edited. With !$you always get the name of the file on which you just worked. If the argument you want is buried in the middle of the command line, you can get at it with the numbered "bang-colon" commands. Here's an example:$ munge /opt/my/long/path/toa/file | more ... $ vi !:1 vi /opt/my/long/path/toa/file
You might be tempted to try to use !$, but in this instance it would yieldmore, which is not the name of the file that you want to edit.- The bash manpage to read about "Word Designators"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Finishing Names for You
- InhaltsvorschauSome of these path names are pretty long. This is a computer that bash is running on… can't it help?When in doubt, press the Tab key. bash will try to finish the pathname for you. If it does nothing, it may be because there are no matches, or because there is more than one. Press the Tab key a second time and it will list the choices and then repeat the command up to where you stopped typing, so that you can continue. Type a bit more (to disambiguate) then press the Tab key again to have bash finish off the argument for you.bash is even smart enough to limit the selection to certain types of files. If you type "unzip" and then the beginning of a pathname, and then you press the Tab key, it will only finish off with files that end in .zip even if you have other files whose names match as much as you have typed. For example:
$ ls myfile.c myfile.o myfile.zip $ ls -lh myfile<tab><tab> myfile.c myfile.o myfile.zip $ ls -lh myfile.z<tab>ip -rw-r--r-- 1 me mygroup 1.9M 2006-06-06 23:26 myfile.zip $ unzip -l myfile<tab>.zip ...
- , "Adjusting readline Behavior Using .inputrc"
- , "Improving Programmable Completion"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Playing It Safe
- InhaltsvorschauIt is so easy to type the wrong character by mistrake (see!). Even for simple bash commands this can be quite serious—you could move or remove the wrong files. When pattern matching is added to the mix, the results can be even more exciting, as a typo in the pattern can lead to wildly different-than-intended consequences. What's a conscientious person to do?You can use these history features and keyboard shortcuts to repeat arguments without retyping them, thereby reducing the typos. If you need a tricky pattern match for files, try it out with echo to see that it works, and then when you've got it right use !$ to use it for real. For example:
$ ls ab1.txt ac1.txt jb1.txt wc3.txt $ echo *1.txt ab1.txt ac1.txt jb1.txt $ echo [aj]?1.txt ab1.txt ac1.txt jb1.txt $ echo ?b1.txt ab1.txt jb1.txt $ rm !$ rm ?b1.txt $
The echo is a way to see the results of your pattern match. Once you're convinced it gives you what you want, then you can use it for your intended command. Here we remove the named files—not something that one wants to get wrong.Also, when you're using the history commands, you can add a:pmodifier and it will cause bash to print but not execute the command—another handy way to see if you got your history substitutions right. From the Solution's example, we add:$ echo ?b1.txt ab1.txt jb1.txt $ rm !$:p rm ?b1.txt $
The:pmodifier caused bash to print but not execute the command—but notice that the argument is?b1.txtand not expanded to the two filenames. That shows you what will be run, and only when it is run will the shell expand that pattern to the two filenames. If you want to see how it will be expanded, use the echo command.- The bash manpage on "Modifiers" for more colon (:) modifiers that can be used on history commands
- "Command-Line Processing Steps" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Chapter 19: Tips and Traps: Common Goofs for Novices
- InhaltsvorschauNobody's perfect. We all make mistakes, especially when we are first learning something new. We have all been there, done that. You know, the silly mistake that seems so obvious once you've had it explained, or the time you thought for sure that the system must be broken because you were doing it exactly right, only to find that you were off by one little character, one which made all the difference. Certain mistakes seem common, almost predictable, among beginners. We've all had to learn the hard way that scripts don't run unless you set execute permissions on them—a real new-bie kind of error. Now that we're experienced, we never make those mistakes anymore. What, never? Well, hardly ever. After all, nobody's perfect.You got your script all written and want to try it out, but when you go to run the script you get an error message:
$ ./my.script bash: ./my.script: Permission denied $
You have two choices. First, you could invoke bash and give it the name of the script as a parameter:$ bash my.script
Or second (and better still), you could set the execute permission on the script so that you can run it directly:$ chmod a+x my.script $ ./my.script
Either method will get the script running. You'll probably want to set the execute permissions on the script if you intend to use it over and over. You only have to set the permissions once, thereafter allowing you to invoke it directly. With the permissions set it feels more like a command, since you don't have to explicitly invoke bash (of course behind the scenes bash is still being invoked, but you don't have to type it).In setting the execute permissions, we useda+xto give execute permissions to all. There's little reason to restrict execute permissions on the file unless it is in some directory where others might accidentally encounter your executable (e.g., if as a system admin you were putting something of your own in/usr/bin). Besides, if the file has read permissions for all then others can still execute the script if they use our first form of invocation, with the explicit reference to bashEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Forgetting to Set Execute Permissions
- InhaltsvorschauYou got your script all written and want to try it out, but when you go to run the script you get an error message:
$ ./my.script bash: ./my.script: Permission denied $
You have two choices. First, you could invoke bash and give it the name of the script as a parameter:$ bash my.script
Or second (and better still), you could set the execute permission on the script so that you can run it directly:$ chmod a+x my.script $ ./my.script
Either method will get the script running. You'll probably want to set the execute permissions on the script if you intend to use it over and over. You only have to set the permissions once, thereafter allowing you to invoke it directly. With the permissions set it feels more like a command, since you don't have to explicitly invoke bash (of course behind the scenes bash is still being invoked, but you don't have to type it).In setting the execute permissions, we useda+xto give execute permissions to all. There's little reason to restrict execute permissions on the file unless it is in some directory where others might accidentally encounter your executable (e.g., if as a system admin you were putting something of your own in/usr/bin). Besides, if the file has read permissions for all then others can still execute the script if they use our first form of invocation, with the explicit reference to bash. Common permissions on shell scripts are0700for the suspicious/careful folk (giving read/write/execute permission to only the owner) and0755for the more open/carefree folk (giving read and execute permissions to all others).- man chmod
- , "Setting Permissions"
- , "Finding bash Portably for #!"
- , "Forgetting That the Current Directory Is Not in the $PATH"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Fixing "No such file or directory" Errors
- InhaltsvorschauYou've set the execute permission as described in , "Forgetting to Set Execute Permissions," but when you run the script you get a "No such file or directory" error.Try running the script using bash explicitly:
$ bash ./busted
If it works, you have some kind of permissions error, or a typo in your shebang line. If you get a bunch more errors, you probably have the wrong line endings. This can happen if you edit the file on Windows (perhaps via Samba), or if you've simply copied the file around.To fix it, try the dos2unix program if you have it, or see , "Converting DOS Files to Linux Format." Note that if you use dos2unix it will probably create a new file and delete the old one, which will change the permissions and might also change the owner or group and affect hard links. If you're not sure what any of that means, the key point is that you'll probably have to chmod it again (, "Forgetting to Set Execute Permissions").If you really do have bad line endings (i.e., anything that isn't ASCII10or hex0a), the error you get depends on your shebang line. Here are some examples for a script named busted:$ cat busted #!/bin/bash - echo "Hello World!" # This works $ ./busted Hello World! # But if the file gets DOS line endings, we get: $ ./busted : invalid option Usage: /bin/bash [GNU long option] [option] ... [...] # Different shebang line $ cat ./busted #!/usr/bin/env bash echo "Hello World!" $ ./busted : No such file or directory
- , "Converting DOS Files to Linux Format"
- , "Avoiding Interpreter Spoofing"
- , "Finding bash Portably for #!"
- , "Forgetting to Set Execute Permissions"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Forgetting That the Current Directory Is Not in the $PATH
- InhaltsvorschauYou've got your script all written and want to try it out—you even remembered to add the execute permissions to the script, but when you go to run the script you get an error message:
$ my.script bash: my.script: command not found $
Either add the current directory to the$PATHvariable, which we do not recommend, or reference the script via the current directory with a leading./before the script name, as in:$ ./my.script
It is a common mistake for beginners to forget to add the leading./to the script that they want to execute. We have had a lot of discussion about the$PATHvariable, so we won't repeat ourselves here except to remind you of a solution for frequently used scripts.A common practice is to keep your useful and often-used scripts in a directory called bin inside of your home directory, and to add that bin directory to your$PATHvariable so that you can execute those scripts without needing the leading./.The important part about adding your own bin directory to your$PATHvariable is to place the change that modifies your$PATHvariable in the right startup script. You don't want it in the .bashrc script because that gets invoked by every subshell, which would mean that your path would get added to every time you "shell out" of an editor, or run some other commands. You don't need repeated copies your bin directory in the$PATHvariable.Instead, put it in the appropriate login profile for bash. According to the bash manpage, when you log in bash "looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable." So edit whichever one of those you already have in your home directory or if none exists, create ~/.bash_profile and put this line in at the bottom of the file (or elsewhere if you understand enough of what else the profile is doing):PATH="${PATH}:$HOME/bin"- , "Running Any Executable"
- , "Setting a Secure $PATH"
- , "Finding World-Writable Directories in Your $PATH"
- , "Adding the Current Directory to the $PATH"
- , "Setting a POSIX $PATH"
- , "Change Your $PATH Permanently"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Naming Your Script Test
- InhaltsvorschauYou typed up a bash script to test out some of this interesting material that you've been reading about. You typed it exactly right, you even remembered to set the execute permissions on the file and put the file in one of the directories in
$PATH, but when you try to run it, nothing happens.Name it something other than test. That name is a shell built-in command.It is natural enough to want to name a file test when you just want a quick scratch file for trying out some small bit of code. The problem is that test is a shell built-in command, making it a kind of shell reserved word. You can see this with thetypecommand:$ type test test is a shell builtin $
Since it is a built-in, no adjusting of the path will override this. You would have to create an alias, but we strongly advise against it in this case. Just name your script something else, or invoke it with a pathname, as in:./testor/home/path/test.- "Built-in Commands and Reserved Words" in
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Expecting to Change Exported Variables
- InhaltsvorschauA common beginner mistake is to treat exported shell variables like globals in a programming environment. But exported variables are only one way: they are included in the environment of the invoked shell script, but if you change their values, those changes are not seen by the calling script.Here is the first of two scripts. This one will set a value, invoke a second script, and then display the value after the second script completes, so as to see what (if anything) has changed:
$ cat first.sh # # a simple example of a common mistake # # set the value: export VAL=5 printf "VAL=%d\n" $VAL # invoke our other script: ./second.sh # # now see what changed (hint: nothing!) printf "%b" "back in first\n" printf "VAL=%d\n" $VAL $
The second script messes with a variable named$VAL, too:$ cat second.sh printf "%b" "in second\n" printf "initially VAL=%d\n" $VAL VAL=12 printf "changed so VAL=%d\n" $VAL $
When we run the first script (which invokes the second one, too) here's what we get:$ ./first.sh VAL=5 in second initially VAL=5 changed so VAL=10 back in first VAL=5 $
The old joke goes something like this:Patient: "Doctor, it hurts when I do this."
Doctor: "Then don't do that."The solution here is going to sound like the doctor's advice: don't do that. You will have to structure your shell scripts so that such a hand-off is not necessary. One way to do that is by explicitly echoing the results of the second script so that the first script can invoke it with the$()operator (or``for the old shell hands). In the first script, the line./second.shbecomesVAL=$(./second.sh), and the second script has to echo the final value (and only the final value) to STDOUT (it could redirect its other messages to STDERR):$ cat second.sh printf "%b" "in second\n" >&2 printf "initially VAL=%d\n" $VAL >&2 VAL=12 printf "changed so VAL=%d\n" $VAL >&2 echo $VAL $
Exported environment variables are not globals that are shared between scripts. They are a one-way communication. All the exported environment variables are marshaled and passed together as part of the invocation of a Linux or Unix (sub) process (see the fork(2) manpage). There is no mechanism whereby these environment variables are passed back to the parent process. (Remember that a parent process can fork lots and lots of subprocesses…so if you could return values from a child process,Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Forgetting Quotes Leads to "command not found" on Assignments
- InhaltsvorschauYour script is assigning some values to a variable, but when you run it, the shell reports "command not found" on part of the value of the assignment.
$ cat goof1.sh #!/bin/bash - # common goof: # X=$Y $Z # isn't the same as # X="$Y $Z" # OPT1=-l OPT2=-h ALLOPT=$OPT1 $OPT2 ls $ALLOPT . $ $ ./goof1.sh goof1.sh: line 10: -h: command not found aaa.awk cdscript.prev ifexpr.sh oldsrc xspin2.sh $
You need quotes around the righthand side of the assignment to$ALLOPT. What is written above as:ALLOPT=$OPT1 $OPT2
really should be:ALLOPT="$OPT1 $OPT2"
It isn't just that you'll lose the embedded spaces between the arguments; it is precisely because there are spaces that this problem arises. If the arguments were combined with an intervening slash, for example, or by no space at all, this problem wouldn't crop up—it would all be a single word, and thus a single assignment.But that intervening space tells bash to parse this into two words. The first word is a variable assignment. Such assignments at the beginning of a command tell bash to set a variable to a given value just for the duration of the command—the command being the word that follows next on the command line. At the next line, the variable is back to its prior value (if any) or just not set.The second word of our example statement is therefore seen as a command. That word is the command that is reported as "not found." Of course it is possible that the value for$OPT2might have been something that actually was the name of an executable (though not likely in this case with ls). Such a situation could lead to very undesirable results.Did you notice, in our example, that when ls ran, it didn't use the long format output even though we had (tried to) set the-loption? That shows that$ALLOPTwas no longer set. It had only been set for the duration of the previous command, which was the attempt to run the (nonexistent)-hcommand.An assignment on a line by itself sets a variable for the remainder of the script. An assignment at the beginning of a line, one that has an additional command invoked on that line, sets the variable only for the execution of that command.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Forgetting That Pattern Matching Alphabetizes
- InhaltsvorschauWarning—bash will alphabetize the data in a pattern match:
$ echo x.[ba] x.a x.b $
Even though you specifiedbthenain the square brackets, when the pattern matching is done and the results found, they will be alphabetized before being given to the command to execute. That means that you don't want to do this:$ mv x.[ba] $
thinking that it will expand to:$ mv x.b x.a
Rather, it will expand to:$ mv x.a x.b
since it alpha-sorts them before putting them in the command line, which is exactly the opposite of what you intended!Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Forgetting That Pipelines Make Subshells
- InhaltsvorschauYou have a script that works just fine, reading input in a
while loop:COUNT=0 while read PREFIX GUTS do # ... if [[ $PREFIX == "abc" ]] then let COUNT++ fi # ... done echo $COUNT
and then you change it to read from a file:cat $1 | while read PREFIX GUTS do # ...
only now it no longer works…$COUNTkeeps coming out as zero.Pipelines create subshells. Changes in thewhileloop do not effect the variables in the outer part of the script, as thewhileloop is run in a subshell.One solution: don't do that (if you can help it). In this example, instead of using cat to pipe the file's content into thewhilestatement, you could use I/O redirection to have the input come from a redirected input rather than setting up a pipeline:COUNT=0 while read PREFIX GUTS do # ... done < $1 echo $COUNT
Such a rearrangement might not be appropriate for your problem, in which case you'll have to find other techniques.If you add an echo statement inside thewhileloop, you can see$COUNTincreasing, but once you exit the loop,$COUNTwill be back to zero. The way that bash sets up the pipeline of commands means that each command in the pipeline will execute in its own subshell. So thewhileloop is in a subshell, not in the main shell. If you have exported$COUNT, then thewhileloop will begin with the same value that the main shell script was using for$COUNT, but since thewhileloop is executing in a subshell there is no way to get the value back up to the parent shell.Depending on how much information you need to get back to the parent shell and how much more work the outer level needs to do after the pipeline, there are different techniques you could use. One technique is to take the additional work and make it part of a subshell that includes thewhileloop. For example:COUNT=0 cat $1 | ( while read PREFIX GUTS do # ... done echo $COUNT )
The placement of the parentheses is crucial here. What we've done is explicitly delineated a section of the script to be run in a subshell. It includes both thewhileloop and the other work that we want to do after theEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Making Your Terminal Sane Again
- InhaltsvorschauYou have aborted an SSH session and now you can't see what you are typing. Or perhaps you accidentally displayed a binary file and your terminal window is now gibberish.Type
stty saneand then the Enter key, even if you can't see what you are typing, to restore sane terminal settings. You may want to hit Enter a few times first, to make sure you don't have anything else on your input line before you start typing the stty command.If you do this a lot, you might consider creating an alias that's easier to type blind.Aborting some older versions of ssh at a password prompt may leave terminal echo (the displaying of characters as you type them, not the shell echo command) turned off so you can't see what you are typing. Depending on what kind of terminal emulation you are using, displaying a binary file can also accidentally change terminal settings. In either case, stty'ssanesetting attempts to return all terminal settings to their default values. This includes restoring echo capability, so that what you type on the keyboard appears in your terminal window. It will also likely undo whatever strangeness has occurred with other terminal settings.Your terminal application may also have some kind of reset function, so explore the menu options and documentation. You may also want to try the reset and tset commands, though in our testingstty saneworked as desired while reset and tset were more drastic in what they fixed.- man reset
- man stty
- man tset
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Deleting Files Using an Empty Variable
- InhaltsvorschauYou have a variable that you think contains a list of files to delete, perhaps to clean up after your script. But in fact, the variable is empty and Bad Things happen.Never do:
rm -rf $files_to_delete
Never, ever, ever do:rm -rf /$files_to_delete
Use this instead:[ "$files_to_delete" ] && rm -rf $files_to_delete
The first example isn't too bad, it'll just throw an error. The second one is pretty bad because it will try to delete your root directory. If you are running as a regular user (and you should be, see , "Running As a Non-root User"), it may not be too bad, but if you are running as root then you've just killed your system but good. (Yes, we've done this.)The solution is easy. First, make sure that there is some value in the variable you're using, and second, never precede that variable with a /.- , "Running As a Non-root User"
- , "Playing It Safe"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Seeing Odd Behavior from printf
- InhaltsvorschauYour script is giving you values that don't match what you expected. Consider this simple script and its output:
$ bash oddscript good nodes: 0 bad nodes: 6 miss nodes: 0 GOOD=6 BAD=0 MISS=0 $ $ cat oddscript #!/bin/bash - badnode=6 printf "good nodes: %d\n" $goodnode printf "bad nodes: %d\n" $badnode printf "miss nodes: %d\n" $missnode printf "GOOD=%d BAD=%d MISS=%d\n" $goodnode $badnode $missnode
Why is6showing up as the value for the good count, when it is supposed to be the value for the bad count?Either give the variables an initial value (e.g.,0) or put quotes around the references to them onprintflines.What's happening here? bash does its substitutions on that last line and when it evaluates$goodnodeand$missnodethey both come out null, empty, not there. So the line that is handed off to printf to execute looks like this:printf "GOOD=%d BAD=%d MISS=%d\n" 6
When printf tries to print the three decimal values (the three%dformats) it has a value (i.e.,6) for the first one, but doesn't have anything for the next two, so they come out zero and we get:GOOD=6 BAD=0 MISS=0
You can't really blame printf, since it never saw the other arguments; bash had done its parameter substitution before printf ever got to run.Even declaring them as integer values, like this:declare -i goodnode badnode missnode
isn't enough. You need to actually assign them a value.The other way to avoid this problem is to quote the arguments when they are used in theprintfstatement, like this:printf "GOOD=%d BAD=%d MISS=%d\n" "$goodnode" "$badnode" "$missnode"
Then the first argument won't disappear, but an empty string will be put in its place, so that what printf gets are the three needed arguments:printf "GOOD=%d BAD=%d MISS=%d\n" "" "6" ""
While we're on the subject of printf, it has one other odd behavior. We have just seen how it behaves when there are too few arguments; when there are too many arguments, printf will keep repeating and reusing the format line and it will look like you are getting multiple lines of output when you expected only one.Of course this can be put to good use, as in the following case:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Testing bash Script Syntax
- InhaltsvorschauYou are editing a bash script and want to make sure that your syntax is correct.Use the
-nargument to bash to test syntax often, ideally after every save, and certainly before committing any changes to a revision control system:$ bash -n my_script $ $ echo 'echo "Broken line' >> my_script $ bash -n my_script my_script: line 4: unexpected EOF while looking for matching `"' my_script: line 5: syntax error: unexpected end of file
The-noption is tricky to find in the bash manpage or other reference material since it's located under the set built-in. It is noted in passing inbash --helpfor-D, but it is never explained there. This flag tells bash to "read commands but do not execute them," which of course will find bash syntax errors.As with all syntax checkers, this will not catch logic errors or syntax errors in other commands called by the script.- man bash
- bash --help
- bash -c "help set"
- , "bash Startup Options"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Debugging Scripts
- InhaltsvorschauYou can't figure out what's happening in your script and why it doesn't work as expected.Add
set-xto the top of the script when you run it. Or useset-xto turn on xtrace before a troublesome spot andset+xto turn it off after. You may also wish to experiment with the$PS4prompt (, "Customizing Your Prompt"). xtrace also works on the interactive command line (, "Customizing Your Prompt"). Here's a script that we suspect is buggy:#!/usr/bin/env bash # cookbook filename: buggy # set -x result=$1 [ $result = 1 ] \ && { echo "Result is 1; excellent." ; exit 0; } \ || { echo "Uh-oh, ummm, RUN AWAY! " ; exit 120; }Now we invoke this script, but first we set and export the value of thePS4prompt. bash will print out the value ofPS4before each command that it displays during an execution trace (i.e., after aset -x):$ export PS4='+xtrace $LINENO:' $ echo $PS4 +xtrace $LINENO: $ ./buggy +xtrace 4: result= +xtrace 6: '[' = 1 ']' ./buggy: line 6: [: =: unary operator expected +xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! ' Uh-oh, ummm, RUN AWAY! $ ./buggy 1 +xtrace 4: result=1 +xtrace 6: '[' 1 = 1 ']' +xtrace 7: echo 'Result is 1; excellent.' Result is 1; excellent. $ ./buggy 2 +xtrace 4: result=2 +xtrace 6: '[' 2 = 1 ']' +xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! ' Uh-oh, ummm, RUN AWAY! $ /tmp/jp-test.sh 3 +xtrace 4: result=3 +xtrace 6: '[' 3 = 1 ']' +xtrace 8: echo 'Uh-oh, ummm, RUN AWAY! ' Uh-oh, ummm, RUN AWAY!
It may seem odd to turn something on using - and turn it off using +, but that's just the way it worked out. Many Unix tools use -n for options or flags, and since you need a way to turn-xoff,+xseems natural.As of bash 3.0 there are a number of new variables to better support debugging:$BASH_ARGC, $BASH_ARGV, $BASH_SOURCE, $BASH_LINENO, $BASH_SUBSHELL, $BASH_ EXECUTION_STRING, and$BASH_COMMAND. This is in addition to existing bash variables like$LINENOand the array variable$FUNCNAME.Using xtrace is a very handy debugging technique, but it is not the same as having a real debugger. See The Bash Debugger Project (Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Avoiding "command not found" When Using Functions
- InhaltsvorschauYou are used to other languages, such as Perl, which allow you to call a function in a section of your code that comes before the actual function definition.Shell scripts are read and executed in a top-to-bottom linear way, so you must define any functions before you use them.Some other languages, such as Perl, go through intermediate steps during which the entire script is parsed as a unit. That allows you to write your code so that
main()is at the top, and function (or subroutines) are defined later. By contrast, a shell script is read into memory and then executed one line at a time, so you can't use a function before you define it.- , "Defining Functions"
- , "Using Functions: Parameters and Return Values"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Confusing Shell Wildcards and Regular Expressions
- InhaltsvorschauSometimes you see .* sometimes just *, and sometimes you see [
a-z]* but it means something other than what you thought. You use regular expressions for grep and sed but not in some places in bash. You can't keep it all straight.Relax; take a deep breath. You're probably confused because you're learning so much (or just using it too infrequently to remember it). Practice makes perfect, so keep trying.The rules aren't that hard to remember for bash itself. After all, regular expression syntax is only used with the =~ comparison operator in bash. All of the other expressions in bash use shell pattern matching.The pattern matching used by bash uses some of the same symbols as regular expressions, but with different meanings. But it is also the case that you often have calls in your shell scripts to commands that use regular expressions—commands like grep and sed.We asked Chet Ramey, the current keeper of the bash source and all-around bash guru, if it was really the case that the =~ was the only use of regular expressions in bash. He concurred. He also was kind enough to supply a list of the various parts of bash syntax that use shell pattern matching. We've covered most, but not all of these topics in various recipes in this book. We offer the list here for completeness.Shell pattern matching is performed by:- Filename globbing (pathname expansion)
- == and != operators for [[
casestatements$GLOBIGNOREhandling$HISTIGNOREhandling- ${parameter#[#]word}
- ${parameter%[%]word}
- ${parameter/pattern/string}
- Several bindable readline commands (glob-expand-word, glob-complete-word, etc.)
complete -Gandcompgen -Gcomplete -Xandcompgen -X- The help built-in's `pattern` argument
Thanks, Chet!- Learn to read the manpage for bash and refer to it often—it is long but precise. If you want an online version of the bash manpage or other bash-related documents, visit http://www.bashcookbook.com for the latest bash information.
- Keep this book handy for reference, too.
- , "Changing Pieces of a String"
- , "Testing for Equal"
- , "Testing with Pattern Matches"
- , "Testing with Regular Expressions"
- , "Trimming Whitespace"
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Appendix A: Reference Lists
- InhaltsvorschauThis appendix collects many tables of values, settings, operators, commands, variables, and more in one place for easy reference.
Section A.1: bash Invocation
Section A.2: Prompt String Customizations
Section A.3: ANSI Color Escape Sequences
Section A.4: Built-in Commands and Reserved Words
Section A.5: Built-in Shell Variables
Section A.6: set Options
Section A.7: shopt options
Section A.8: Adjusting Shell Behavior Using set, shopt, and Environment Variables
Section A.9: Test Operators
Section A.10: I/O Redirection
Section A.11: echo Options and Escape Sequences
Section A.12: printf
Section A.13: Date and Time String Formatting with strftime
Section A.14: Pattern-Matching Characters
Section A.15: extglob Extended Pattern-Matching Operators
Section A.16: tr Escape Sequences
Section A.17: Readline Init File Syntax
Section A.18: emacs Mode Commands
Section A.19: vi Control Mode Commands
Section A.20: Table of ASCII Values
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - bash Invocation
- InhaltsvorschauHere are the options you can use when invoking current versions of bash. The multi-character options must appear on the command line before the single-character options. Login shells are usually invoked with the options
-i(interactive),-s(read from standard input), and-m(enable job control).In addition to these listed in , any set option can be used on the command line; see the "set Options" section later in this chapter. In particular, the-noption is invaluable for syntax checking, see , "Testing bash Script Syntax."Table A-1: Command-line options to bash OptionMeaning-cstringCommands are read from string, if present. Any arguments after string are interpreted as positional parameters, starting with$0.-DA list of all double-quoted strings preceded by $ is printed on the standard output. These are the strings that are subject to language translation when the current locale is not C or POSIX. This also turns on the-noption.-iInteractive shell. Ignores signalsTERM, INT, andQUIT. With job control in effect,TTIN, TTOU, andTSTPare also ignored.-lMakes bash act as if invoked as a login shell.-ooptionTakes the same arguments asset -o.O, +Oshopt-optionshopt-option is one of the shell options accepted by the shopt built-in. If shopt-option is present,-Osets the value of that option;+Ounsets it. If shopt-option is not supplied, the names and values of the shell options accepted by shopt are printed on the standard output. If the invocation option is+O, the output is displayed in a format that may be reused as input.-sReads commands from the standard input. If an argument is given to bash, this flag takes precedence (i.e., the argument won't be treated as a script name and standard input will be read).-rRestricted shell.-vPrints shell input lines as they're read.-Signals the end of options and disables further option processing. Any options after this are treated as filenames and arguments. -- is synonymous with -.--debuggerArranges for the debugger profile to be executed before the shell starts. Turns on extended debugging mode and shell function tracing inEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Prompt String Customizations
- Inhaltsvorschaushows a summary of the prompt customizations that are available. The customizations \[ and \] are not available in bash versions prior to 1.14.
\a, \e, \H, \T, \@, \v, and\Vare not available in versions prior to 2.0.\A, \D, \j, \l, and\rare only available in later versions of bash 2.0 and in bash 3.0.Table A-2: Prompt string format codes CommandMeaningAdded\aThe ASCII bell character (007).bash-1.14.7\AThe current time in 24-hour HH:MM format.bash-2.05\dThe date in "Weekday Month Day" format.\D{format}The format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation; the braces are required.bash-2.05b\eThe ASCII escape character (033).bash-1.14.7\HThe hostname.bash-1.14.7\hThe hostname up to the first ".".\jThe number of jobs currently managed by the shell.bash-2.03\lThe basename of the shell's terminal device name.bash-2.03\nA carriage return and line feed.\rA carriage return.bash-2.01.1\sThe name of the shell.\TThe current time in 12-hour HH:MM:SS format.bash-1.14.7\tThe current time in HH:MM:SS format.\@The current time in 12-hour a.m./p.m. format.bash-1.14.7\uThe username of the current user.\vThe version of bash (e.g., 2.00).bash-1.14.7\VThe release of bash; the version and patchlevel (e.g., 3.00.0).bash-1.14.7\wThe current working directory.\WThe basename of the current working directory.\#The command number of the current command.\!The history number of the current command.\$If the effective UID is 0, print a #, otherwise print a $.\nnnCharacter code in octal.\\Print a backslash.\[Begin a sequence of nonprinting characters, such as terminal control sequences.\]End a sequence of nonprinting characters.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - ANSI Color Escape Sequences
- Inhaltsvorschaushows the ANSI color escape sequences.
Table A-3: ANSI color escape sequences CodeCharacter attributeFG codeForeground colorBG codeBackground color0Reset all attributes30Black40Black1Bright31Red41Red2Dim32Green42Green4Underscore33Yellow43Yellow5Blink34Blue44Blue7Reverse35Magenta45Magenta8Hidden36Cyan46Cyan37White47WhiteEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Built-in Commands and Reserved Words
- Inhaltsvorschaushows a summary of all built-in commands and reserved words. The let-ters in the Type column of the table have the following meanings: R = reserved word, blank = built-in.
Table A-4: Built-in commands and reserved words CommandTypeSummary!RLogical NOT of a command exit status.:Do nothing (just do expansions of any arguments)..Read file and execute its contents in current shell.aliasSet up shorthand for command or command line.bgPut job in background.bindBind a key sequence to areadlinefunction or macro.breakExit from surroundingfor, select, while,oruntilloop.builtinExecute the specified shell built-in.caseRReserved word. Multi-way conditional construct.cdChange working directory.commandRun a command bypassing shell function lookup.compgenGenerate possible completion matches.completeSpecify how completion should be performed.continueSkip to next iteration offor, select, while, oruntilloop.declareDeclare variables and give them attributes. Same astypeset.dirsDisplay the list of currently remembered directories.disownRemove a job from the job table.doRPart of afor, select, while, oruntillooping construct.doneRPart of afor, select, while, oruntillooping construct.echoOutput arguments.elifRPart of anifconstruct.elseRPart of anifconstruct.enableEnable and disable built-in shell commands.esacREnd of acaseconstruct.evalRun the given arguments through command-line processing.execReplace the shell with the given program.exitExit from the shell.exportCreate environment variables.fcFix command (edit history file).fgEnd background job in foreground.fiRPart of anifconstruct.forRLooping construct.functionRDefine a function.getoptsProcess command-line options.hashFull pathnames are determined and remembered.helpDisplay helpful information on built-in commands.historyDisplay command history.ifRConditional construct.inRPart of acaseconstruct.jobsList any background jobs.killSend a signal to a process.letArithmetic variable assignment.localCreate a local variable.logoutExit a login shell.popdRemove a directory from the directory stack.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Built-in Shell Variables
- Inhaltsvorschaushows a complete list of environment variables available in bash 3.0. The letters in the Type column of the table have the following meanings: A = Array, L = colon-separated list, R = read-only, U = unsetting it causes it to lose its special meaning.Note that the variables beginning
BASH_and beginningCOMP, as well as the variablesDIRSTACK, FUNCNAME, GLOBIGNORE, GROUPS, HISTIGNORE, HOSTNAME, HISTTIMEFORMAT, LANG, LC_ALL, LC_COLLATE, LC_MESSAGE, MACHTYPE, PIPESTATUS, SHELLOPTS, andTIMEFORMATare not available in versions prior to 2.0.BASH_ENVreplacesENVfound in earlier versions.Table A-5: Built-in shell environment variables VariableTypeDescription*RA single string containing the positional parameters given to the current script or function, separated by the first character of$IFS(e.g.,arg1 arg2 arg3).@REach of the positional parameters given to the current script or function, given as a list of double-quoted strings (e.g.,"arg1" "arg2" "arg3").#RThe number of arguments given to the current script or function.-ROptions given to the shell on invocation.?RExit status of the previous command._RLast argument to the previous command.$RProcess ID of the shell process.!RProcess ID of the last background command.0RName of the shell or shell script.BASHThe full pathname used to invoke this instance of bash.BASH_ARGCAAn array of values, which are the number of parameters in each frame of the current bash execution call stack. The number of parameters to the current subroutine (shell function or script executed with . orsource) is at the top of the stack.BASH_ARGVAAll of the parameters in the current bash execution call stack. The final parameter of the last subroutine call is at the top of the stack; the first parameter of the initial call is at the bottom.BASH_COMMANDThe command currently being executed or about to be executed, unless the shell is executing a command as the result of a trap, in which case it is the command executing at the time of the trap.BASH_EXECUTION_STRINGThe command argument to the-cinvocation option.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - set Options
- InhaltsvorschauThe options in can be turned on with the
set -argcommand. They are all initially off except where noted. Full names, where listed, are arguments to set that can be used withset -o. The full namesbraceexpand, histexpand, history, keyword, andonecmdare not available in versions of bash prior to 2.0. Also, in those versions, hashing is switched with-d.Table A-6: set options OptionFull name (-o)Meaning-aallexportExport all subsequently defined or modified variables.-BbraceexpandThe shell performs brace expansion. This is on by default.-bnotifyReport the status of terminating background jobs immediately.-CnoclobberDon't allow redirection to overwrite existing files.-EerrtraceAny trap onERRis inherited by shell functions, command substitutions, and commands executed in a subshell environment.-eerrexitExit the shell when a simple command exits with nonzero status. A simple command is a command not part of awhile, until, orif;nor part of a && or || list; nor a command whose return value is inverted by !.emacsUse Emacs-style command-line editing.-fnoglobDisable pathname expansion.-HhistexpandEnable ! style history substitution. On by default in an interactive shell.historyEnable command history. On by default in interactive shells.-hhashallEnable the hashing of commands.ignoreeofDisallow Ctrl-D to exit the shell.-kkeywordAll arguments in the form of assignment statements are placed in the environment for a command, not just those that precede the command name.-mmonitorEnable job control (on by default in interactive shells).-nnoexecRead commands and check syntax but do not execute them. Ignored for interactive shells.-PphysicalDo not follow symbolic links on commands that change the current directory. Use the physical directory.-pprivilegedScript is running in suid mode.pipefailThe return value of a pipeline is the value of the last (rightmost) command to exit with a nonzero status, or zero if all commands in the pipeline exit successfully. This option is disabled by default.posixChange the default behavior to that of POSIX 1003.2 where it differs from the standard.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - shopt options
- InhaltsvorschauThe shopt options are set with
shopt -sarg and unset withshopt -uarg (see ). Versions of bash prior to 2.0 had environment variables to perform some of these settings. Setting them equated toshopt -s. The variables (and corresponding shopt options) were:allow_null_glob_expansion (nullglob), cdable_vars (cdable_vars), command_oriented_history (cmdhist), glob_dot_filenames (dotglob), no_exit_ on_failed_exec (execfail). These variables no longer exist.The optionsextdebug, failglob, force_fignore, andgnu_errfmtare not available in versions of bash prior to 3.0.Table A-7: shopt options OptionMeaning if setcdable_varsAn argument to cd that is not a directory is assumed to be the name of a variable whose value is the directory to change to.cdspellMinor errors in the spelling of a directory supplied to the cd command will be corrected if there is a suitable match. This correction includes missing letters, incorrect letters, and letter transposition. It works for interactive shells only.checkhashCommands found in the hash table are checked for existence before being executed and nonexistence forces a$PATHsearch.checkwinsizeChecks the window size after each command and, if it has changed, updates the variables$LINESand$COLUMNSaccordingly.cmdhistAttempt to save all lines of a multiline command in a single history entry.dotglobFilenames beginning with a . are included in pathname expansion.execfailA noninteractive shell will not exit if it cannot execute the argument to an exec. Interactive shells do not exit if exec fails.expand_aliasesAliases are expanded.extdebugBehavior intended for use by debuggers is enabled. This includes: the-Foption of declare displays the source filename and line number corresponding to each function name supplied as an argument; if the command run by theDEBUGtrap returns a nonzero value, the next command is skipped and not executed; and if the command run by theDEBUGtrap returns a value of 2, and the shell is executing in a subroutine, a call toreturnis simulated.extglobExtended pattern matching features are enabled.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Adjusting Shell Behavior Using set, shopt, and Environment Variables
- Inhaltsvorschaucombines , , and and provides a quick way to look for what you can configure and which of the three mechanisms you use to configure it. The options are loosely grouped according to function or purpose, but it's worthwhile to scan the entire table to get an overall sense of what you can configure.The "Set option" column contains the options that can be turned on with the
set-arg command. All are initially off except where noted. Items in the "Set full name" column, where listed, are arguments to set that can be used withset -o. The full namesbraceexpand, histexpand, history, keyword, andonecmdare not available in versions of bash prior to 2.0. Also, in those versions, hashing is switched with-d.The "Shopt option" column shows the options set withshopt -sarg and unset withshopt -uarg. Versions of bash prior to 2.0 had environment variables to perform some of these settings. Setting them equated toshopt -s. The variables (and corresponding shopt options) were:allow_null_glob_expansion (nullglob), cdable_vars (cdable_vars), command_oriented_history (cmdhist), glob_dot_filenames (dotglob), no_exit_on_failed_exec (execfail). These variables no longer exist.The optionsextdebug, failglob, force_fignore, andgnu_errfmtare not available in versions of bash prior to 3.0.The "Environment variable" column lists environment variables that affect bash configuration and operation. The letters in the Type column of the table have the following meanings: A = Array, L = colon-separated list, R = read-only, U = unsetting it causes it to lose its special meaning.Note that the variables beginningBASH_and beginningCOMP, as well as the variablesDIRSTACK, FUNCNAME, GLOBIGNORE, GROUPS, HISTIGNORE, HOSTNAME, HISTTIMEFORMAT, LANG, LC_ALL, LC_COLLATE, LC_MESSAGE, MACHTYPE, PIPESTATUS, SHELLOPTS, andTIMEFORMATare not available in versions prior to 2.0.BASH_ENVreplacesENVfound in earlier versions.Table A-8: Adjusting shell behavior using set, shopt, and environment variables Set optionSet full nameShopt optionEnvironment variableEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Test Operators
- InhaltsvorschauThe operators in are used with test and the […] and [[…]] constructs. They can be logically combined with
-a("and") and-o("or") and grouped with escaped parenthesis (\(…\)). The string comparisons < and > and the [[…]] construct are not available in versions of bash prior to 2.0, and =~ is only available in bash version 3.0 and later as noted.Table A-9: Test operators OperatorTrue if-afilefile exists, deprecated, same as-e-bfilefile exists and is a block device file-cfilefile exists and is a character device file-dfilefile exists and is a directory-efilefile exists; same as-a-ffilefile exists and is a regular file-gfilefile exists and has its setgid bit set-Gfilefile exists and is owned by the effective group ID-hfilefile exists and is a symbolic link, same as-L-kfilefile exists and has its sticky bit set-Lfilefile exists and is a symbolic link, same as-h-nstringstring is non-null-Nfilefile was modified since it was last read-Ofilefile exists and is owned by the effective user ID-pfilefile exists and is a pipe or named pipe (FIFO file)-rfilefile exists and is readable-sfilefile exists and is not empty-Sfilefile exists and is a socket-t NFile descriptor N points to a terminal-ufilefile exists and has its setuid bit set-wfilefile exists and is writeable-xfilefile exists and is executable, or file is a directory that can be searched-zstringstring has a length of zerofileA-ntfileBfileA modification time is newer than fileAfileA-otfileBfileA modification time is older than fileAfileA-effileBfileA and fileB point to the same filestringA = stringBstringA equals stringB (POSIX version)stringA == stringBstringA equals stringBstringA != stringBstringA does not match stringBstringA =~ regexpstringA matches the extended regular expression regexpstringA < stringBstringA sorts before stringB lexicographicallystringA > stringBstringA sorts after stringB lexicographicallyexprA-eqexprBArithmetic expressions exprA and exprB are equalexprA-neexprBArithmetic expressions exprA and exprB are not equalEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - I/O Redirection
- Inhaltsvorschauis a complete list of I/O redirectors. Note that there are two formats for specifying STDOUT and STDERR redirection: &>file and >&file. The second of these (which is the one used throughout this book) is the preferred way.
Table A-10: Input/output redirection RedirectorFunctioncmd1 | cmd2Pipe; take standard output of cmd1 as standard input to cmd2.> fileDirect standard output to file.< fileTake standard input from file.>> fileDirect standard output to file; append to file if it already exists.>| fileForce standard output to file even ifnoclobberis set.n>| fileForce output to file from file descriptorneven ifnoclobberis set.<> fileUse file as both standard input and standard output.n<> fileUse file as both input and output for file descriptor n.<< labelHere-document.n> fileDirect file descriptor n to file.n< fileTake file descriptor n from file.>> fileDirect file descriptor n to file; append to file if it already exists.n>&Duplicate standard output to file descriptor n.n<&Duplicate standard input from file descriptor n.n>&mFile descriptor n is made to be a copy of the output file descriptor m.n<&mFile descriptor n is made to be a copy of the input file descriptor m.&>fileDirects standard output and standard error to file.<&-Close the standard input.>&-Close the standard output.n>&-Close the output from file descriptor n.n<&-Close the input from file descriptor n.n>&wordIf n is not specified, the standard output (file descriptor 1) is used; if the digits in word do not specify a file descriptor open for output, a redirection error occurs; as a special case, if n is omitted, and word does not expand to one or more digits, the standard output and standard error are redirected as described previously.n<&wordIf word expands to one or more digits, the file descriptor denoted by n is made to be a copy of that file descriptor; if the digits in word do not specify a file descriptor open for input, a redirection error occurs; if word evaluates to -, file descriptor n is closed; if n is not specified, the standard input (file descriptor 0) is used.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - echo Options and Escape Sequences
- Inhaltsvorschauecho accepts a number of arguments (see ).
Table A-11: echo options OptionsFunction-eTurns on the interpretation of backslash-escaped characters-ETurns off the interpretation of backslash-escaped characters on systems where this mode is the default-nOmits the final newline (same as the\cescape sequence)echo accepts a number of escape sequences that start with a backslash.These sequences in exhibit fairly predictable behavior, except for\f, which on some displays causes a screen clear while on others it causes a line feed, and it ejects the page on most printers.\vis somewhat obsolete; it usually causes a line feed.Table A-12: echo escape sequences SequenceCharacter printed\aAlert or Ctrl-G (bell)\bBackspace or Ctrl-H\cOmit final newline\eEscape character (same as\E)\EEscape character\fFormfeed or Ctrl-L\nNewline (not at end of command) or Ctrl-J\rReturn (Enter) or Ctrl-M\tTab or Ctrl-I\vVertical Tab or Ctrl-K\nnnnThe eight-bit character whose value is the octal (base-8) value nnn where nnn is 1 to 3 digits\0nnnThe eight-bit character whose value is the octal (base-8) value nnn where nnn is 0 to 3 digits\xHHThe eight-bit character whose value is the hexadecimal (base-16) value HH (one or two digits)\\Single backslashThe\n,\0, and\xsequences are even more device-dependent and can be used for complex I/O, such as cursor control and special graphics characters.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - printf
- InhaltsvorschauThe printf command, available in bash since version 2.02, has two parts (beyond the command name): a format string and a variable number of arguments:
printf format-string [arguments]format-string describes the format specifications; this is best supplied as a string constant in quotes. arguments is a list, such as a list of strings or variable values that correspond to the format specifications.The format is reused as necessary to use up all of the arguments. If the format requires more arguments than are supplied, the extra format specifications behave as if a zero value or null string, as appropriate, had been supplied.A format specification is preceded by a percent sign (%), and the specifier is one of the characters described below. Two of the main format specifiers are %s for strings and %d for decimal integers (see ).Table A-13: printf format specifiers Format characterMeaning%cASCII character (prints first character of corresponding argument)%d, %iDecimal (base 10) integer%eFloating-point format ([-]d.precisione[+-]dd)—see the text after the table for the meaning of precision%EFloating-point format ([-]d.precisionE[+-]dd)%fFloating-point format ([-]ddd.precision)%g%eor%fconversion, whichever is shorter, with trailing zeros removed%G%Eor%fconversion, whichever is shortest, with trailing zeros removed%oUnsigned octal value%sString%uUnsigned decimal value%xUnsigned hexadecimal number; usesa-ffor 10 to 15%XUnsigned hexadecimal number; usesA-Ffor 10 to 15%%Literal %The printf command can be used to specify the width and alignment of output fields. A format expression can take three optional modifiers following % and preceding the format specifier:%flags width.precision format-specifier
The width of the output field is a numeric value. When you specify a field width, the contents of the field are right-justified by default. You must specify a flag of -to get left-justification (the rest of the flags are shown in the table). Thus,%-20soutputs a left-justified string in a field 20-characters wide. If the string is less than 20 characters, the field is padded with whitespace to fill. In the following examples, we put our format specifier between a pair of | in our format string so you can see the width of the field in the output. The first example right-justifies the text:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Date and Time String Formatting with strftime
- Inhaltsvorschaushows common date and time string formatting options. Consult your system's manpages for date and strftime(3), as both the options and what they mean vary from system to system.
Table A-18: strftime format codes FormatDescription%%A literal %.%aThe locale's abbreviated weekday name (Sun..Sat).%AThe locale's full weekday name (Sunday..Saturday).%BThe locale's full month name (January..December).%b or %hThe locale's abbreviated month name (Jan..Dec).%cThe locale's default/preferred date and time representation.%CThe century (a year divided by 100 and truncated to an integer) as a decimal number (00..99).%dThe day of the month as a decimal number (01..31).%DThe date in the format%m/%d/%y(MM/DD/YY). Note that the United States uses MM/DD/YY while everyone else uses DD/MM/YY, so this format is ambiguous and should be avoided. Use%Finstead, since it's a recognized standard and it sorts well.%eThe day of month as a blank padded decimal number ( 1..31).%FThe date in the format%Y-%m-%d(the ISO 8601 date format: CCYY-MM-DD); except when it's the full month name, as on HP-UX.%gThe two-digit year corresponding to the%Vweek number (YY).%GThe four-digit year corresponding to the%Vweek number (CCYY).%HThe hour (24-hour clock) as a decimal number (00..23).%h or %bThe locale's abbreviated month name (Jan..Dec).%IThe hour (12-hour clock) as a decimal number (01..12).%jThe day of the year as a decimal number (001..366).%kThe hour (24-hour clock) as a blank padded decimal number ( 0..23).%lThe hour (12-hour clock) as a blank padded decimal number ( 1,12).%mThe month as a decimal number (01..12).%MThe minute as a decimal number (00..59).%nA literal newline.%NNanoseconds (000000000..999999999). [GNU]%pThe locale's equivalent of either "AM" or "PM".%PThe locale's equivalent of either "am" or "pm". [GNU]%rThe locale's representation of 12-hour clock time using AM/PM notation (HH:MM:SS AM/PM).%RThe time in the format%H:%M(HH:MM).%sThe number of seconds since the Epoch, UTC (January 1, 1970 at 00:00:00).%SThe second as a decimal number (00..61). The range of seconds is (00-61) instead of (00-59) to allow for the periodic occurrence of leap seconds and double leap seconds.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Pattern-Matching Characters
- InhaltsvorschauThe material in this section is adapted from the Bash Reference Manual (http://www.gnu.org/software/bash/manual/bashref.html; see ).
Table A-19: Pattern-matching characters CharacterMeaning*Matches any string, including the null string.?Matches any single character.[ … ]Matches any one of the enclosed characters.[! … ] or [^ … ]Matches any character not enclosed.The following POSIX character classes may be used within [], e.g.,[[:alnum:]]; consult the grep or egrep manpage on your system for more details.[[:alnum:]] [[:alpha:]] [[:ascii:]] [[:blank:]] [[:cntrl:]] [[:digit:]] [[:graph:]] [[:lower:]] [[:print:]] [[:punct:]] [[:space:]] [[:upper:]] [[:word:]] [[:xdigit:]]
Thewordcharacter class matches letters, digits, and the character _.[=c=]matches all characters with the same collation weight (as defined by the current locale) as the character c, while [.symbol.] matches the collating symbol symbol.These character classes are affected by the locale setting. To get the traditional Unix values, useLC_COLLATE=CorLC_ALL=C.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - extglob Extended Pattern-Matching Operators
- InhaltsvorschauThe operators in apply when using
shopt -s extglob. Matches are case-sensitive, but you may useshopt -s nocasematch(bash 3.1+) to change that. This option affectscaseand [[ commands.Table A-20: extglob extended pattern-matching operators GroupingMeaning@( … )Only one occurrence*( … )Zero or more occurrences+( … )One or more occurrences?( … )Zero or one occurrences!( … )Not these occurrences, but anything elseEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - tr Escape Sequences
- Inhaltsvorschau
Table A-21: tr escape sequences SequenceMeaning\oooCharacter with octal value ooo (1-3 octal digits)\\A backslash character (i.e., escapes the backslash itself)\a"Audible" bell, the ASCII BEL character (since"b"was taken for backspace)\bBackspace\fForm feed\nNewline\rReturn\tTab (sometimes called a horizontal tab)\vVertical tabEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Readline Init File Syntax
- InhaltsvorschauThe GNU Readline library provides the command line on which you type to communicate with bash and some other GNU utilities. It is amazingly configurable, but most people are not aware of this., , and are a subset of what is available to work with. See the Readline documentation for the full details.The following is adapted directly from Chet Ramey's documentation (http://tiswww.tis.case.edu/~chet/readline/readline.html).You can modify the run-time behavior of Readline by altering the values of variables in Readline using the set command within the init file. The syntax is simple:
set variable value
Here, for example, is how to change from the default Emacs-like key binding to use vi line-editing commands:set editing-mode vi
Variable names and values, where appropriate, are recognized without regard to case. Unrecognized variable names are ignored.Boolean variables (those that can be set to on or off) are set to on if the value is null or empty, on (case-insensitive), or 1. Any other value results in the variable being set to off.Table A-22: Readline configuration settings VariableDescriptionbell-styleControls what happens when Readline wants to ring the terminal bell. If set tonone, Readline never rings the bell. If set tovisible, Readline uses a visible bell if one is available. If set toaudible(the default), Readline attempts to ring the terminal's bell.bind-tty-special-charsIf set toon, Readline attempts to bind the control characters treated specially by the kernel's terminal driver to their Readline equivalents.comment-beginThe string to insert at the beginning of the line when the insert-comment command is executed. The default value is #.completion-ignore-caseIf set toon, Readline performs filename matching and completion in a case-insensitive fashion. The default value isoff.completion-query-itemsThe number of possible completions that determines when the user is asked whether the list of possibilities should be displayed. If the number of possible completions is greater than this value, Readline will ask the user whether he wishes to view them; otherwise, they are simply listed. This variable must be set to an integer value greater than or equal to 0. A negative value means Readline should never ask. The default limit isEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - emacs Mode Commands
- InhaltsvorschauThe material in this section also appears in Learning the bash Shell by Cameron Newham (O'Reilly).is a complete list of readline Emacs editing mode commands.
Table A-23: emacs mode commands CommandMeaningCtrl-AMove to beginning of line.Ctrl-BMove backward one character.Ctrl-DDelete one character forward.Ctrl-EMove to end of line.Ctrl-FMove forward one character.Ctrl-GAbort the current editing command and ring the terminal bell.Ctrl-JSame as Return.Ctrl-KDelete (kill) forward to end of line.Ctrl-LClear screen and redisplay the line.Ctrl-MSame as Return.Ctrl-NNext line in command history.Ctrl-OSame as Return, then display next line in history file.Ctrl-PPrevious line in command history.Ctrl-RSearch backward.Ctrl-SSearch forward.Ctrl-TTranspose two characters.Ctrl-UKill backward from point to the beginning of line.Ctrl-VMake the next character typed verbatim.Ctrl-V TabInsert a Tab.Ctrl-WKill the word behind the cursor, using whitespace as the boundary.Ctrl-X/List the possible filename completions of the current word.Ctrl-X~List the possible username completions of the current word.Ctrl-X $List the possible shell variable completions of the current word.Ctrl-X@List the possible hostname completions of the current word.Ctrl-X!List the possible command name completions of the current word.Ctrl-X(Begin saving characters into the current keyboard macro.Ctrl-X)Stop saving characters into the current keyboard macro.Ctrl-XeRe-execute the last keyboard macro defined.Ctrl-X Ctrl-RRead in the contents of the readline initialization file.Ctrl-X Ctrl-VDisplay version information on this instance of bash.Ctrl-YRetrieve (yank) last item killed.DeleteDelete one character backward.Ctrl-[Same as Esc (most keyboards).Esc-BMove one word backward.Esc-CChange word after point to all capital letters.Esc-DDelete one word forward.Esc-FMove one word forward.Esc-LChange word after point to all lowercase letters.Esc-NNonincremental forward search.Esc-PNonincremental reverse search.Esc-RUndo all the changes made to this line.Esc-TTranspose two words.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - vi Control Mode Commands
- InhaltsvorschauThe material in this section also appears in Learning the bash Shell by Cameron Newham (O'Reilly).shows a complete list of readline vi control mode commands.
Table A-24: vi mode commands CommandMeaninghMove left one character.lMove right one character.wMove right one word.bMove left one word.WMove to beginning of next nonblank word.BMove to beginning of preceding nonblank word.eMove to end of current word.EMove to end of current nonblank word.0Move to beginning of line..Repeat the lastainsertion.^Move to first nonblank character in line.$Move to end of line.iInsert text before current character.aInsert text after current character.IInsert text at beginning of line.AInsert text at end of line.ROverwrite existing text.dhDelete one character backward.dlDelete one character forward.dbDelete one word backward.dwDelete one word forward.dBDelete one nonblank word backward.dWDelete one nonblank word forward.d$Delete to end of line.d0Delete to beginning of line.DEquivalent tod$(delete to end of line).ddEquivalent to0d$(delete entire line).CEquivalent toc$(delete to end of line, enter input mode).ccEquivalent to0c$(delete entire line, enter input mode).xEquivalent todl(delete character forwards).XEquivalent todh(delete character backwards).kor -Move backward one line.jor +Move forward one line.GMove to line given by repeat count./stringSearch forward for string.?stringSearch backward for string.nRepeat search forward.NRepeat search backward.fxMove right to next occurrence of x.FxMove left to previous occurrence of x.txMove right to next occurrence of x, then back one space.TxMove left to previous occurrence of x, then forward one space.;Redo last character finding command.,Redo last character finding command in opposite direction.\Do filename completion.*Do wildcard expansion (onto command line).\=Do wildcard expansion (as printed list).~Invert (twiddle) case of current character(s).\Append last word of previous command, enter input mode.Ctrl+LStart a new line and redraw the current line on it.#Prepend # (comment character) to the line and send it to history.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Table of ASCII Values
- InhaltsvorschauMany of our favorite computer books have an ASCII chart. Even in the era of GUIs and web servers you may be surprised to find that you still need to look up a character every now and then. It's certainly useful when working with tr or finding some special sequence of escape characters.IntOctalHexASCII000000^@100101^A200202^B300303^C400404^D500505^E600606^F700707^G801008^H901109^I100120a^J110130b^K120140c^L130150d^M140160e^N150170f^O1602010^P1702111^Q1802212^R1902313^S2002414^T2102515^U2202616^V2302717^W2403018^X2503119^Y260321a^Z270331b^[280341c^\290351d^]300361e^^310371f^_32040203304121!3404222"3504323#3604424$3704525%3804626&3904727'4005028(4105129)420522a*430532b+440542c,450552d-460562e.470572f/48060300490613115006232251063333520643445306535554066366550673775607038857071399580723a:590733b;600743c<610753d=620763e>630773f?6410040@6510141A6610242B6710343C6810444D6910545E7010646F7110747G7211048H7311149I741124aJ751134bK761144cL771154dM781164eN791174fO8012050P8112151Q8212252R8312353S8412454T8512555U8612656V8712757W8813058X8913159Y901325aZ911335b[921345c\931355d]941365e^951375f_9614060`9714161a9814262b9914363c10014464d10114565e10214666f10314767g10415068h10515169i1061526aj1071536bk1081546cl1091556dm1101566en1111576fo11216070p11316171q11416272r11516373s11616474t11716575u11816676v11916777w12017078x12117179y1221727az1231737b{1241747c|1251757d}1261767e~1271777f^?Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Appendix B: Examples Included with bash
- InhaltsvorschauThe bash tarball archive includes an examples directory that is well worth exploring (after you've finished reading this book, of course). It includes sample code, scripts, functions, and startup files.
Section B.1: Startup-Files Directory Examples
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Startup-Files Directory Examples
- InhaltsvorschauThe startup-files directory provides many examples of what you can put in your own startup files. In particular, bash_aliases has many useful aliases. Bear in mind that if you copy these files wholesale, you'll have to edit them for your system because many of the paths will be different. Refer to for further information on changing these files to suit your needs.The functions directory contains many function definitions that you might find useful. Among them are:
basename- The basename utility, missing from some systems
dirfuncs- Directory manipulation facilities
dirname- The dirname utility, missing from some systems
whatis- An implementation of the Tenth Edition Bourne shell whatis built-in
whence- An almost exact clone of the Korn shell whence built-in
If you come from a Korn shell background, you may find kshenv especially helpful. This contains function definitions for some common Korn facilities such as whence, print, and the two-parameter cd built-ins.The scripts directory contains many examples of bash scripts. The two largest scripts are examples of the complex things you can do with shell scripts. The first is a (rather amusing) adventure game interpreter and the second is a C shell interpreter. The other scripts include examples of precedence rules, a scrolling text display, a "spinning wheel" progress display, and how to prompt the user for a particular type of answer.Not only are the script and function examples useful for including in your environment, they also provide many alternative examples that you can learn from when reading this book. We encourage you to experiment with them.is an index of what you will find as of bash 3.1 or newer.Table B-1: Paths for bash 3.1. and newer PathDescriptionX-ref./bashdbDeprecated sample implementation of a bash debugger../completeShell completion code../functionsExample functions../functions/array-stuffVarious array functions (ashift, array_sort, reverse)../functions/array-to-stringConvert an array to a string../functions/autoloadAn almost ksh-compatible 'autoload' (no lazy load).Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Appendix C: Command-Line Processing
- InhaltsvorschauThroughout the book we've seen a variety of ways in which the shell processes input lines, especially using
read. We can think of this process as a subset of the things the shell does when processing command lines. This appendix provides a more detailed description of the steps involved in processing the command line and how you can get bash to make a second pass with eval. The material in this lso appears in Learning the bash Shell by Cameron Newham (O'Reilly).Section C.1: Command-Line Processing Steps
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Command-Line Processing Steps
- InhaltsvorschauWe've touched upon command-line processing throughout this book; we've mentioned how bash deals with single quotes (''), double quotes (";"), and backslashes (\); how it separates characters on a line into words, even allowing you to specify the delimiter it uses via the environment variable
$IFS; how it assigns the words to shell variables (e.g.,$1, $2, etc); and how it can redirect input and output to/from files or to other processes (pipeline). In order to be a real expert at shell scripting (or to debug some gnarly problems), you might need to understand the various steps involved in command-line processing—especially the order in which they occur.Each line that the shell reads from STDIN or from a script is called a pipeline because it contains one or more commands separated by zero or more pipe characters (|). shows the steps in command-line processing. For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command.- Splits the command into tokens that are separated by the fixed set of metacharacters: space, tab, newline, ;, (, ), <, >, |, and &. Types of tokens include words, keywords, I/O redirectors, and semicolons.
- Checks the first token of each command to see if it is a keyword with no quotes or backslashes. If it's an opening keyword such as
ifand other control-structure openers,function, {,or (, then the command is actually a compound command. The shell sets things up internally for the compound command, reads the next command, and starts the process again. If the keyword isn't a compound command opener (e.g., it is a control-structure "middle" likethen, else, ordo;an "end" likefiordone; or a logical operator), the shell signals a syntax error.
Figure C-1: Steps in command-line processing - Checks the first word of each command against the list of aliases. If a match is found, it substitutes the alias' definition and goes back to Step 1; otherwise, it goes on to Step 4. This scheme allows recursive aliases. It also allows aliases for keywords to be defined, e.g.,
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Appendix D: Revision Control
- InhaltsvorschauRevision control systems are a way to not only travel back in time, but to see what has changed at various points in your timeline. They are also called versioning or version control systems, which is actually a more technically accurate name. Such a system allows you to maintain a central repository of files in a project, and to keep track of changes to those files, as well as the reason for those changes. Some revision control systems allow more than one developer to work concurrently on the same project, or even the same file.Revision control systems are essential to modern software development efforts, but they are also useful in many other areas, such as writing documentation, tracking system configurations (e.g., /etc), and even writing books. We kept this book under revision control using Subversion while writing it.Some of the useful features of revision control systems include:
- Making it very difficult to lose code, especially when the repository is properly backed up.
- Facilitating change control practices, and encourage documenting why a change is being made.
- Allowing people in multiple locations to work together on a project, and to keep up with others' changes, without losing data by saving on top of each other.
- Allowing one person to work from multiple locations over time without losing work or stepping on changes made at other locations.
- Allowing you to back out changes easily or to see exactly what has changed between one revision and another (except binary files). If you follow effective logging practices, they will even tell you why a change was made.
- Allowing, usually, a form of keyword expansion that lets you embed revision metadata in nonbinary files.
There are many different free and commercial revision control systems, and we would like to strongly encourage you to use one. If you already have one, use it. If you don't, we'll briefly cover three of the most common systems (CVS, Subversion, and RCS), all of which either come with or are available for every major modern operating system.Before using a revision control system, you must first decide:Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - CVS
- InhaltsvorschauThe Concurrent Versions System (CVS) is a widely used and mature revision control system, with command-line tools for all major modern operating systems (including Windows), and GUI tools for some of them (notably Windows).
- It is everywhere and is very mature.
- Many Unix system administrators and virtually every open source or free software developer is familiar with it.
- It's easy to use for simple projects.
- It's easy to access remote repositories.
- It's based on RCS, which allows for some hacking of the central repository.
- Commits are not atomic, so the repository could be left in an inconsistent state if a commit fails half-way through.
- Commits are by file only; you must also tag if you need to reference a group of files.
- Directory structure support is poor.
- Does not allow easy renaming of files and directories while retaining history.
- Poor support for binary files, and little support for other objects such as symbolic links.
- Based on RCS, which allows for some hacking of the central repository.
CVS tracks revisions by file, which means that each file has its own internal CVS revision number. As each file is changed, that number changes, so a single project can't be tracked by a single revision number, since each file is different. Use tags for that kind of tracking.This example is not suitable for enterprise or multiuser access (see the "More Resources" section in the Preface). This is just to show how easy the basics are. This example has theEDITORenvironment variable set to nano (export EDITOR='nano --smooth --const --nowrap --suspend'), which some people find more user-friendly than the default vi.Thecvscommand (with no options), thecvs helpcommand (wherehelpis not a valid argument, but is easy to remember and still triggers a useful response), and thecvs --helpcvs_command command are very useful.Create a new repository for personal use in a home directory:/home/jp$ mkdir -m 0775 cvsroot /home/jp$ chmod g+srwx cvsroot /home/jp$ cvs -d /home/jp/cvsroot init
Create a new project and import it:/home/jp$ cd /tmp /tmp$ mkdir 0700 scripts /tmp$ cd scripts/ /tmp/scripts$ cat << EOF > hello > #!/bin/sh > echo 'Hello World!' > EOF /tmp/scripts$ cvs -d /home/jp/cvsroot import scripts shell_scripts NA GNU nano 1.2.4 File: /tmp/cvsnJgYmG Initial import of shell scripts CVS: ---------------------------------------------------------------------- CVS: Enter Log. Lines beginning with `CVS:' are removed automatically* CVS: CVS: --------------------------------------------------------------------- [ Wrote 5 lines ] N scripts/hello No conflicts created by this import
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Subversion
- InhaltsvorschauAccording to the Subversion web site, "The goal of the Subversion project is to build a version control system that is a compelling replacement for CVS in the open source community." Enough said.
- Newer than CVS and RCS.
- Simpler and arguably easier to understand and use than CVS (less historical baggage).
- Atomic commits means the commit either fails or succeeds as a whole, and makes it easy to track the state of an entire project as a single revision.
- Easy to access remote repositories.
- Allows easy renaming of files and directories while retaining history.
- Easily handles binary files (no native diff support) and other objects such as symbolic links.
- Central repository hacking is more officially supported, but less trivial.
- Not 100 percent CVS compatible for more complicated projects (e.g., branching and tagging).
- Can be more complicated to build or install from scratch due to many dependencies. Use the version that came with your operating system if possible.
SVN tracks revisions by repository, which means that each commit has its own internal SVN revision number. Thus consecutive commits by a single person may not have consecutive revision numbers since the global repository revision is incremented as other changes (possibly to other projects) are committed by other people.This example is not suitable for enterprise or multiuser access (see the "More Resources" section in the Preface). This is just to show how easy the basics are. This example also has theEDITORenvironment variable set to nano(export EDITOR='nano --smooth --const --nowrap --suspend'), which some people find more user-friendly than the default vi.Thesvn helpandsvn help helpcommands are very useful.Create a new repository for personal use in a home directory:/home/jp$ svnadmin --fs-type=fsfs create /home/jp/svnroot
Create a new project and import it:/home/jp$ cd /tmp /tmp$ mkdir -p -m 0700 scripts/trunk scripts/tags scripts/branches /tmp$ cd scripts/trunk /tmp/scripts/trunk$ cat << EOF > hello > #!/bin/sh > echo 'Hello World!' > EOF /tmp/scripts/trunk$ cd .. /tmp/scripts$ svn import /tmp/scripts file:///home/jp/svnroot/scripts GNU nano 1.2.4 File: svn-commit.tmp Initial import of shell scripts --This line, and those below, will be ignored-- A . [ Wrote 4 lines ] Adding /tmp/scripts/trunk Adding /tmp/scripts/trunk/hello Adding /tmp/scripts/branches Adding /tmp/scripts/tags Committed revision 1.
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - RCS
- InhaltsvorschauRCS was a revolution in its time, and is the underlying basis for CVS.
- It's better than nothing.
- Does not allow concurrent access to the same file.
- Does not have the inherent concept of a central repository, though you can go out of your way to create one using symbolic links.
- No concept of remote repositories.
- Only tracks changes to files, and does not store or consider directories at all.
- Poor support for binary files, and no support for other objects such as symbolic links. Unlike CVS or SVN, which have a single main end-user binary, RCS is a collection of binaries.
Create a new script directory for personal use in a home directory:/home/jp$ mkdir -m 0754 bin
Create some scripts:/home/jp$ cd bin /tmp/scripts/bin$ cat << EOF > hello > #!/bin/sh > echo 'Hello World!' > EOF /home/jp/bin$ ci hello hello,v <-- hello enter description, terminated with single '.' or end of file: NOTE: This is NOT the log message! >> Obligatory Hello World >> . initial revision: 1.1 done /home/jp/bin$ ls -l total 4.0K -r--r--r-- 1 jp jp 228 Jul 20 02:25 hello,v
Huh? What happened? It turns out that if a directory called RCS does not exist, the current directory is used for the RCS file. And if the-u or -lswitches are not used, the file is checked in and then removed.-lcauses the file to be checked back out and locked so you can edit it, while-uis unlocked (that is, read-only). OK, let's try that again. First, let's get our file back, then create an RCS directory and check it in again./home/jp/bin$ co -u hello hello,v --> hello revision 1.1 (unlocked) done /home/jp/bin$ ls -l total 8.0K -r--r--r-- 1 jp jp 30 Jul 20 02:29 hello -r--r--r-- 1 jp jp 228 Jul 20 02:25 hello,v /home/jp/bin$ rm hello,v rm: remove write-protected regular file `hello,v'? y /home/jp/bin$ mkdir -m 0755 RCS /home/jp/bin$ ci -u hello RCS/hello,v <-- hello enter description, terminated with single '.' or end of file: NOTE: This is NOT the log message! >> Obligatory Hello World >> . initial revision: 1.1 done /home/jp/bin$ ls -l total 8.0K drwxr-xr-x 2 jp jp 4.0K Jul 20 02:31 RCS /-r--r--r-- 1 jp jp 30 Jul 20 02:29 hello /home/jp/bin$ ls -l RCS total 4.0K -r--r--r-- 1 jp jp 242 Jul 20 02:31 hello,v
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Other
- InhaltsvorschauFinally, it is worth noting that some word processors, such as OpenOffice.org Writer and Microsoft Word, have three relevant features: document comparison, change tracking, and versions.Document Comparison allows you to compare documents when their native file format makes use of other diff tools difficult. You would use this when you have two copies of a document that didn't have change tracking turned on, or when you need to merge feedback from various sources.While it is trivial to unzip the content.xml file from a given OpenDoc file, the result has no line breaks and is not terribly pretty or readable. See , "Comparing Two Documents" for a bash script that will do this low-level kind of difference.Refer to the table below for information on how to access the built-in GUI comparison function, which is much easier than trying to do it manually.The change-tracking feature saves information about changes made to a document. Review mode uses various copyediting markup on the screen to display who did what, when. This is obviously useful for all kinds of creation and editing purposes, but please read our warnings.The versions feature allows you to save more than one version of a document in a single file. This can be handy in all sorts of odd ways. For example, we've seen router configurations copied and pasted from a terminal into different versions inside the same document for archival and change control purposes.The change tracking and versions features will cause your document to continually grow in size, since items that are changed are still kept and deleted items are not really deleted, but only marked as deleted.If accidentally turned on, change tracking and versions can be very dangerous information leaks! For example, if you send similar proposals to competing companies after doing a search and replace and other editing, someone at one of those companies can see exactly what you changed and when you changed it. The most recent versions of these tools have various methods that attempt to warn you or clear private information before a given document is converted to PDF or emailed.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
- Appendix E: Building bash from Source
- InhaltsvorschauIn this appendix we'll show you how to get the latest version of bash and install it on your system from source, and we'll discuss potential problems you might encounter along the way. We'll also look briefly at the examples that come with bash and how you can report bugs to the bash maintainer. The material in this lso appears in Learning the bash Shell by Cameron Newham (O'Reilly).
Section E.1: Obtaining bash
Section E.2: Unpacking the Archive
Section E.3: What's in the Archive
Section E.4: Who Do I Turn To?
Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Obtaining bash
- InhaltsvorschauIf you have a direct connection to the Internet, you should have no trouble obtaining bash; otherwise, you'll have to do a little more work. The bash home page is located at http://www.gnu.org/software/bash/bash.html and you can find the very latest details of the current distribution and where to obtain it from there.You can also get bash on CD-ROM by ordering it directly from the Free Software Foundation, either via the web-ordering page at http://order.fsf.org or from:The Free Software Foundation (FSF)
59 Temple Place – Suite 330
Boston, MA 02111-1307 USA
Phone: +1-617-542-5942
Fax: +1-617-542-2652
Email: order@fsf.org(Valid as of Thursday April 20, 2006 11:45:40 PDT.)Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Unpacking the Archive
- InhaltsvorschauHaving obtained the archive file by one of the above methods, you need to unpack it and install it on your system. Unpacking can be done anywhere—we'll assume you're unpacking it in your home directory. Installing it on the system requires you to have root privileges. If you aren't a system administrator with root access, you can still compile and use bash; you just can't install it as a system-wide utility. The first thing to do is uncompress the archive file:
gunzipbash-3.1.tar.gz. Then you need tountarthe archive:tar -xf bash- 3.1.tar. The-xfmeans "extract the archived material from the specified file." This will create a directory called bash-3.1 in your home directory. If you do not have the gunzip utility, you can obtain it in the same way you obtained bash or simply usegzip -dinstead.The archive contains all of the source code needed to compile bash and a large amount of documentation and examples. We'll look at these things and how you go about making a bash executable in the rest of this appendix.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - What's in the Archive
- InhaltsvorschauThe bash archive contains a main directory (bash-3.1 for the current version) and a set of files and subdirectories. Among the first files you should examine are:
- CHANGES
- A comprehensive list of bug fixes and new features since the last version
- COPYING
- The GNU Copyleft for bash
- MANIFEST
- A list of all the files and directories in the archive
- NEWS
- A list of new features since the last version
- README
- A short introduction and instructions for compiling bash
You should also be aware of two directories:- doc
- Information related to bash in various formats
- examples
- Examples of startup files, scripts, and functions
The other files and directories in the archive are mostly things that are needed during the build. Unless you are going to go hacking into the internal workings of the shell, they shouldn't concern you.The doc directory contains a few articles that are worth reading. Indeed, it would be well worth printing out the manual entry for bash so you can use it in conjunction with this book. The README file gives a short summary of the files.The document you'll most often use is the manpage entry bash.1. The file is in troff format—the same format used by the manpages. You can read it by processing it with the text-formatter nroff and piping the output to a pager utility; e.g.,nroff-man bash.1 | moreshould do the trick. You can also print it off by piping it to the line-printer (lp). This summarizes all of the facilities your version of bash has and is the most up-to-date reference you can get. This document is also available through the man facility once you've installed the package, but sometimes it's nice to have a hardcopy so you can write notes all over it.Of the other documents, FAQ is a Frequently Asked Questions document with answers, readline.3 is the manual entry for the readline facility, and article.ms is an article about the shell that appeared in Linux Journal, and was written by the current bash maintainer Chet Ramey.To compile bash "straight out of the box" is easy—you just type ./configure and then make! The configure script attempts to work out whether you have various utilities and C library functions, and their location on your system. It then stores the relevant information in the fileEnde der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar. - Who Do I Turn To?
- InhaltsvorschauNo matter how good something is or how much documentation comes with it, you'll eventually come across something that you don't understand or that doesn't work. In such cases it can't be stressed enough to carefully read the documentation (in more casual computer parlance: RTFM). In many cases, this will answer your question or point out what you're doing wrong.Sometimes you'll find this only adds to your confusion or confirms that there is something wrong with the software. The next thing to do is to talk to a local bash guru to sort out the problem. If that fails, or there is no guru, you'll have to turn to other means (currently only via the Internet).If you have any questions about bash, there are currently two ways to go about getting them answered. You can email questions to bash-maintainers@gnu.org or you can post your question to the USENET newsgroup gnu.bash.bug.In both cases either the bash maintainer or some knowledgeable person on USENET will give you advice. When asking a question, try to give a meaningful summary of your question in the subject line (see http://www.catb.org/~esr/faqs/smart-questions.html).Bug reports should be sent to bug-bash@gnu.org, and include the version of bash and the operating system it is running on, the compiler used to compile bash, a description of the problem, a description of how the problem was produced, and, if possible, a fix for the problem. The best way to do this is with the bashbug script, installed with bash.Before you run bashbug, make sure that you've set your
EDITORenvironment variable to your favorite editor and have exported it (bashbug defaults to Emacs, which might not be installed on your system). When you execute bashbug it will enter the editor with a partially blank report form. Some of the information (bash version, operating system version, etc.) will have been filled in automatically. We'll take a brief look at the form, but most of it is self-explanatory.TheFrom: field should be filled out with your email address. For example:Next comes theSubject: field; make an effort to fill it out, as this makes it easier for the maintainers when they need to look up your submission. Just replace the line surrounded by square brackets with a meaningful summary of the problem.Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.
Zurück zu bash Cookbook
