Inhaltsverzeichnis von Classic Shell Scripting : Hidden Commands that Unlock the Power of Unix

Inhaltsvorschau

This chapter provides a brief history of the development of the Unix system. Understanding where and how Unix developed and the intent behind its design will help you use the tools better. The chapter also introduces the guiding principles of the Software Tools philosophy, which are then demonstrated throughout the rest of the book.

It is likely that you know something about the development of Unix, and many resources are available that provide the full story. Our intent here is to show how the environment that gave birth to Unix influenced the design of the various tools.

Unix was originally developed in the Computing Sciences Research Center at Bell Telephone Laboratories. The first version was developed in 1970, shortly after Bell Labs withdrew from the Multics project. Many of the ideas that Unix popularized were initially pioneered within the Multics operating system; most notably the concepts of devices as files, and of having a command interpreter (or shell ) that was intentionally not integrated into the operating system. A well-written history may be found at http://www.bell-labs.com/history/unix.

Because Unix was developed within a research-oriented environment, there was no commercial pressure to produce or ship a finished product. This had several advantages:

The system was developed by its users. They used it to solve real day-to-day computing problems.
The researchers were free to experiment and to change programs as needed. Because the user base was small, if a program needed to be rewritten from scratch, that generally wasn't a problem. And because the users were the developers, they were free to fix problems as they were discovered and add enhancements as the need for them arose.
Unix itself went through multiple research versions, informally referred to with the letter "V" and a number: V6, V7, and so on. (The formal name followed the edition number of the published manual: First Edition, Second Edition, and so on. The correspondence between the names is direct: V6 = Sixth Edition, and V7 = Seventh Edition. Like most experienced Unix programmers, we use both nomenclatures.) The most influential Unix system was the Seventh Edition, released in 1979, although earlier ones had been available to educational institutions for several years. In particular, the Seventh Edition system introduced both

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

It is likely that you know something about the development of Unix, and many resources are available that provide the full story. Our intent here is to show how the environment that gave birth to Unix influenced the design of the various tools.

Unix was originally developed in the Computing Sciences Research Center at Bell Telephone Laboratories. The first version was developed in 1970, shortly after Bell Labs withdrew from the Multics project. Many of the ideas that Unix popularized were initially pioneered within the Multics operating system; most notably the concepts of devices as files, and of having a command interpreter (or shell ) that was intentionally not integrated into the operating system. A well-written history may be found at http://www.bell-labs.com/history/unix.

Because Unix was developed within a research-oriented environment, there was no commercial pressure to produce or ship a finished product. This had several advantages:

The system was developed by its users. They used it to solve real day-to-day computing problems.
The researchers were free to experiment and to change programs as needed. Because the user base was small, if a program needed to be rewritten from scratch, that generally wasn't a problem. And because the users were the developers, they were free to fix problems as they were discovered and add enhancements as the need for them arose.
Unix itself went through multiple research versions, informally referred to with the letter "V" and a number: V6, V7, and so on. (The formal name followed the edition number of the published manual: First Edition, Second Edition, and so on. The correspondence between the names is direct: V6 = Sixth Edition, and V7 = Seventh Edition. Like most experienced Unix programmers, we use both nomenclatures.) The most influential Unix system was the Seventh Edition, released in 1979, although earlier ones had been available to educational institutions for several years. In particular, the Seventh Edition system introduced both

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Over the course of time, a set of core principles developed for designing and writing software tools. You will see these exemplified in the programs used for problem solving throughout this book. Good software tools should do the following things:

Do one thing well: In many ways, this is the single most important principle to apply. Programs that do only one thing are easier to design, easier to write, easier to debug, and easier to maintain and document. For example, a program like grep that searches files for lines matching a pattern should not also be expected to perform arithmetic.

A natural consequence of this principle is a proliferation of smaller, specialized programs, much as a professional carpenter has a large number of specialized tools in his toolbox.
Process lines of text, not binary: Lines of text are the universal format in Unix. Datafiles containing text lines are easy to process when writing your own tools, they are easy to edit with any available text editor, and they are portable across networks and multiple machine architectures. Using text files facilitates combining any custom tools with existing Unix programs.
Use regular expressions: Regular expressions are a powerful mechanism for working with text. Understanding how they work and using them properly simplifies your script-writing tasks.

Furthermore, although regular expressions varied across tools and Unix versions over the years, the POSIX standard provides only two kinds of regular expressions, with standardized library routines for regular-expression matching. This makes it possible for you to write your own tools that work with regular expressions identical to those of

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Unix was originally developed at Bell Labs by and for computer scientists. The lack of commercial pressure, combined with the small capacity of the PDP-11 minicomputer, led to a quest for small, elegant programs. The same lack of commercial pressure, though, led to a system that wasn't always consistent, nor easy to learn.

As Unix spread and variant versions developed (notably the System V and BSD variants), portability at the shell script level became difficult. Fortunately, the POSIX standardization effort has borne fruit, and just about all commercial Unix systems and free Unix workalikes are POSIX-compliant.

The Software Tools principles as we've outlined them provide the guidelines for the development and use of the Unix toolset. Thinking with the Software Tools mindset will help you write clear shell programs that make correct use of the Unix tools.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When you need to get some work done with a computer, it's best to use a tool that's appropriate to the job at hand. You don't use a text editor to balance your checkbook or a calculator to write a proposal. So too, different programming languages meet different needs when it comes time to get some computer-related task done.

Shell scripts are used most often for system administration tasks, or for combining existing programs to accomplish some small, specific job. Once you've figured out how to get the job done, you can bundle up the commands into a separate program, or script, which you can then run directly. What's more, if it's useful, other people can make use of the program, treating it as a black box, a program that gets a job done, without their having to know how it does so.

In this chapter we'll make a brief comparison between different kinds of programming languages, and then get started writing some simple shell scripts.

Most medium and large-scale programs are written in a compiled language, such as Fortran, Ada, Pascal, C, C++, or Java. The programs are translated from their original source code into object code which is then executed directly by the computer's hardware.

The benefit of compiled languages is that they're efficient. Their disadvantage is that they usually work at a low level, dealing with bytes, integers, floating-point numbers, and other machine-level kinds of objects. For example, it's difficult in C++ to say something simple like "copy all the files in this directory to that directory over there."

So-called scripting languages are usually interpreted. A regular compiled program, the interpreter , reads the program, translates it into an internal form, and then executes the program.

The advantage to scripting languages is that they often work at a higher level than compiled languages, being able to deal more easily with objects such as files and directories. The disadvantage is that they are often less efficient than compiled languages. Usually the tradeoff is worthwhile; it can take an hour to write a simple script that would take two days to code in C or C++, and usually the script will run fast enough that performance won't be a problem. Examples of scripting languages include

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Most medium and large-scale programs are written in a compiled language, such as Fortran, Ada, Pascal, C, C++, or Java. The programs are translated from their original source code into object code which is then executed directly by the computer's hardware.

The benefit of compiled languages is that they're efficient. Their disadvantage is that they usually work at a low level, dealing with bytes, integers, floating-point numbers, and other machine-level kinds of objects. For example, it's difficult in C++ to say something simple like "copy all the files in this directory to that directory over there."

So-called scripting languages are usually interpreted. A regular compiled program, the interpreter , reads the program, translates it into an internal form, and then executes the program.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The advantage to scripting languages is that they often work at a higher level than compiled languages, being able to deal more easily with objects such as files and directories. The disadvantage is that they are often less efficient than compiled languages. Usually the tradeoff is worthwhile; it can take an hour to write a simple script that would take two days to code in C or C++, and usually the script will run fast enough that performance won't be a problem. Examples of scripting languages include awk, Perl, Python, Ruby, and the shell.

Because the shell is universal among Unix systems, and because the language is standardized by POSIX, shell scripts can be written once and, if written carefully, used across a range of systems. Thus, the reasons to use a shell script are:

Simplicity: The shell is a high-level language; you can express complex operations clearly and simply using it.
Portability: By using just POSIX-specified features, you have a good chance of being able to move your script, unchanged, to different kinds of systems.
Ease of development: You can often write a powerful, useful script in little time.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Let's start with a simple script. Suppose that you'd like to know how many users are currently logged in. The who command tells you who is logged in:

$ who

george     pts/2        Dec 31 16:39    (valley-forge.example.com)

betsy      pts/3        Dec 27 11:07    (flags-r-us.example.com)

benjamin   dtlocal      Dec 27 17:55    (kites.example.com)

jhancock   pts/5        Dec 27 17:55    (:32)

camus      pts/6        Dec 31 16:22

tolstoy    pts/14       Jan  2 06:42

On a large multiuser system, the listing can scroll off the screen before you can count all the users, and doing that every time is painful anyway. This is a perfect opportunity for automation. What's missing is a way to count the number of users. For that, we use the wc (word count) program, which counts lines, words, and characters. In this instance, we want

wc

-l

, to count just lines:

$ who | wc -l                       

            Count users

      6

The | (pipe) symbol creates a pipeline between the two programs: who's output becomes wc's input. The result, printed by wc, is the number of users logged in.

The next step is to make this pipeline into a separate command. You do this by entering the commands into a regular file, and then making the file executable, with chmod, like so:

$ cat > nusers                      

            Create the file, copy terminal input with cat

            who | wc -l                         

            Program text

            ^D                                  

            Ctrl-D is end-of-file

$ chmod +x nusers                   

            Make it executable

$ ./nusers                          

            Do a test run

      6                             Output is what we expect

This shows the typical development cycle for small one- or two-line shell scripts: first, you experiment directly at the command line. Then, once you've figured out the proper incantations to do what you want, you put them into a separate script and make the script executable. You can then use that script directly from now on.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

When the shell runs a program, it asks the Unix kernel to start a new process and run the given program in that process. The kernel knows how to do this for compiled programs. Our nusers shell script isn't a compiled program; when the shell asks the kernel to run it, the kernel will fail to do so, returning a "not executable format file" error. The shell, upon receiving this error, says "Aha, it's not a compiled program, it must be a shell script," and then proceeds to start a new copy of /bin/sh (the standard shell) to run the program.

The "fall back to /bin/sh" mechanism is great when there's only one shell. However, because current Unix systems have multiple shells, there needs to be a way to tell the Unix kernel which shell to use when running a particular shell script. In fact, it helps to have a general mechanism that makes it possible to directly invoke any programming language interpreter, not just a command shell. This is done via a special first line in the script file—one that begins with the two characters #!.

When the first two characters of a file are #!, the kernel scans the rest of the line for the full pathname of an interpreter to use to run the program. (Any intervening whitespace is skipped.) The kernel also scans for a single option to be passed to that interpreter. The kernel invokes the interpreter with the given option, along with the rest of the command line. For example, assume a csh script named /usr/ucb/whizprog, with this first line:

#! /bin/csh -f

Furthermore, assume that /usr/ucb is included in the shell's search path (described later). A user might type the command whizprog -q /dev/tty01. The kernel interprets the #! line and invokes csh as follows:

/bin/csh -f /usr/ucb/whizprog -q /dev/tty01

This mechanism makes it easy to invoke any interpreted language. For example, it is a good way to invoke a standalone awk program:

#! /bin/awk -f

awk program here

Shell scripts typically start with

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

In this section we introduce the basic building blocks used in just about all shell scripts. You will undoubtedly be familiar with some or all of them from your interactive use of the shell.

The shell's most basic job is simply to execute commands. This is most obvious when the shell is being used interactively: you type commands one at a time, and the shell executes them, like so:

$ cd work ; ls -l whizprog.c

-rw-r--r--    1 tolstoy   devel       30252 Jul  9 22:52 whizprog.c

$ make

...

These examples show the basics of the Unix command line. First, the format is simple, with whitespace (space and/or tab characters) separating the different components involved in the command.

Second, the command name, rather logically, is the first item on the line. Most typically, options follow, and then any additional arguments to the command follow the options. No gratuitous syntax is involved, such as:

COMMAND=CD,ARG=WORK

COMMAND=LISTFILES,MODE=LONG,ARG=WHIZPROG.C

Such command languages were typical of the larger systems available when Unix was designed. The free-form syntax of the Unix shell was a real innovation in its time, contributing notably to the readability of shell scripts.

Third, options start with a dash (or minus sign) and consist of a single letter. Options are optional, and may require an argument (such as

cc -o

whizprog

whizprog.c). Options that don't require an argument can be grouped together: e.g., ls -lt whizprog.c rather than

ls -l

-t whizprog.c

(which works, but requires more typing).

Long options are increasingly common, particularly in the GNU variants of the standard utilities, as well as in programs written for the X Window System (X11). For example:

$ cd whizprog-1.1

$ patch --verbose --backup -p1 < /tmp/whizprog-1.1-1.2-patch

Depending upon the program, long options start with either one dash, or with two (as just shown). (The

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The so-called positional parameters represent a shell script's command-line arguments. They also represent a function's arguments within shell functions. Individual arguments are named by integer numbers. For historical reasons, you have to enclose the number in braces if it's greater than nine:

echo first arg is $1

echo tenth arg is ${10}

Special "variables" provide access to the total number of arguments that were passed, and to all the arguments at once. We provide the details later, in Section 6.1.2.2.

Suppose you want to know what terminal a particular user is using. Well, once again, you could use a plain who command and manually scan the output. However, that's difficult and error prone, especially on systems with lots of users. This time what you want to do is search through who's output for a particular user. Well, anytime you want to do searching, that's a job for the grep command, which prints lines matching the pattern given in its first argument. Suppose you're looking for user betsy because you really need that flag you ordered from her:

$ who | grep betsy                      

            Where is betsy?

betsy      pts/3        Dec 27 11:07    (flags-r-us.example.com)

Now that we know how to find a particular user, we can put the commands into a script, with the script's first argument being the username we want to find:

$ cat > finduser                        

            Create new file

            #! /bin/sh



            # finduser --- see if user named by first argument is logged in



            who | grep $1

            ^D                                      

            End-of-file



$ chmod +x finduser                     

            Make it executable



$ ./finduser betsy                      

            Test it: find betsy

betsy      pts/3        Dec 27 11:07    (flags-r-us.example.com)



$ ./finduser benjamin                   

            Now look for good old Ben

benjamin   dtlocal      Dec 27 17:55    (kites.example.com)



$

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Because program development is a human activity, there will be times when your script just doesn't do what you want it to do. One way to get some idea of what your program is doing is to turn on execution tracing. This causes the shell to print out each command as it's executed, preceded by "+ "—that is, a plus sign followed by a space. (You can change what gets printed by assigning a new value to the PS4 shell variable.) For example:

$ sh -x nusers                          

            Run with tracing on

+ who                                   Traced commands

+ wc -l

      7                                 Actual output

You can turn execution tracing on within a script by using the command set -x, and turn it off again with set +x. This is more useful in fancier scripts, but here's a simple program to demonstrate:

$ cat > trace1.sh                       

            Create script

            #! /bin/sh



            set -x                                  

            Turn on tracing

            echo 1st echo                           

            Do something



            set +x                                  

            Turn off tracing

            echo 2nd echo                           

            Do something else

            ^D                                      

            Terminate with end-of-file



$ chmod +x trace1.sh                    

            Make program executable



$ ./trace1.sh                           

            Run it

+ echo 1st echo                         First traced line

1st echo                                Output from command

+ set +x                                Next traced line

2nd echo                                Output from next command

When run, the set -x is not traced, since tracing isn't turned on until after that command completes. Similarly, the set +x is traced, since tracing isn't turned off until after it completes. The final echo isn't traced, since tracing is turned off at that point.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

Writing software for an international audience is a challenging problem. The task is usually divided into two parts: internationalization (i18n for short, since that long word has 18 letters between the first and last), and localization (similarly abbreviated l10n).

Internationalization is the process of designing software so that it can be adapted for specific user communities without having to change or recompile the code. At a minimum, this means that all character strings must be wrapped in library calls that handle runtime lookup of suitable translations in message catalogs. Typically, the translations are specified in ordinary text files that accompany the software, and then are compiled by gencat or msgfmt into compact binary files organized for fast lookup. The compiled message catalogs are then installed in a system-specific directory tree, such as the GNU conventional /usr/share/locale and /usr/local/share/locale, or on commercial Unix systems, /usr/lib/nls or /usr/lib/locale. Details can be found in the manual pages for setlocale(3), catgets(3C), and gettext(3C).

Localization is the process of adapting internationalized software for use by specific user communities. This may require translating software documentation, and all text strings output by the software, and possibly changing the formats of currency, dates, numbers, times, units of measurement, and so on, in program output. The character set used for text may also have to be changed, unless the universal Unicode character set can be used, and different fonts may be required. For some languages, the writing direction has to be changed as well.

In the Unix world, ISO programming language standards and POSIX have introduced limited support for addressing these problems, but much remains to be done, and progress varies substantially across the various flavors of Unix. For the user, the feature that controls which language or cultural environment is in effect is called the

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The choice of compiled language versus scripting language is usually made based on the need of the application. Scripting languages generally work at a higher level than compiled languages, and the loss in performance is often more than made up for by the speed with which development can be done and the ability to work at a higher level.

The shell is one of the most important and widely used scripting languages in the Unix environment. Because it is ubiquitous, and because of the POSIX standard, it is possible to write shell programs that will work on many different vendor platforms. Because the shell functions at a high level, shell programs have a lot of bang for the buck; you can do a lot with relatively little work.

The #! first line should be used for all shell scripts; this mechanism provides you with flexibility, and the ability to write scripts in your choice of shell or other language.

The shell is a full programming language. So far we covered the basics of commands, options, arguments, and variables, and basic output with echo and printf. We also looked at the basic I/O redirection operators, <, >, >>, and |, with which we expect you're really already familiar.

The shell looks for commands in each directory in $PATH. It's common to have a personal bin directory in which to store your own private programs and scripts, and to list it in PATH by doing an assignment in your .profile file.

We looked at the basics of accessing command-line arguments and simple execution tracing.

Finally, we discussed internationalization and localization, topics that are growing in importance as computer systems are adapted to the computing needs of more of the world's people. While support in this area for shell scripts is still limited, shell programmers need to be aware of the influence of locales on their code.

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

As we discussed in Section 1.2, Unix programmers prefer to work on lines of text. Textual data is more flexible than binary data, and Unix systems provide a number of tools that make slicing and dicing text easy.

In this chapter, we look at two fundamental operations that show up repeatedly in shell scripting: text searching—looking for specific lines of text—and text substitution—changing the text that is found.

While you can accomplish many things by using simple constant text strings, regular expressions provide a much more powerful notation for matching many different actual text fragments with a single expression. This chapter introduces the two regular expression "flavors" provided by various Unix programs, and then proceeds to cover the most important tools for text extraction and rearranging.

The workhorse program for finding text (or "matching text," in Unix jargon) is grep. On POSIX systems, grep can use either of the two regular expression flavors, or match simple strings.

Traditionally, there were three separate programs for searching through text files:

grep: The original text-matching program. It uses Basic Regular Expressions (BREs) as defined by POSIX, and as we describe later in the chapter.
egrep: "Extended grep." This program uses Extended Regular Expressions (EREs), which are a more powerful regular expression notation. The cost of EREs is that they can be more computationally expensive to use. On the original PDP-11s this was important; on modern systems, there is little difference.
fgrep

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

The workhorse program for finding text (or "matching text," in Unix jargon) is grep. On POSIX systems, grep can use either of the two regular expression flavors, or match simple strings.

Traditionally, there were three separate programs for searching through text files:

grep: The original text-matching program. It uses Basic Regular Expressions (BREs) as defined by POSIX, and as we describe later in the chapter.
egrep: "Extended grep." This program uses Extended Regular Expressions (EREs), which are a more powerful regular expression notation. The cost of EREs is that they can be more computationally expensive to use. On the original PDP-11s this was important; on modern systems, there is little difference.
fgrep: "Fast grep." This variant matches fixed strings instead of regular expressions using an algorithm optimized for fixed-string matching. The original version was also the only variant that could match multiple strings in parallel. In other words, grep and egrep could match only a single regular expression, whereas fgrep used a different algorithm that could match multiple strings, effectively testing each input line for a match against all the requested search strings.

The 1992 POSIX standard merged all three variants into one grep program whose behavior is controlled by different options. The POSIX version can match multiple patterns, even for BREs and EREs. Both fgrep and egrep were also available, but they were marked as "deprecated," meaning that they would be removed from a subsequent standard. And indeed, the 2001 POSIX standard only includes the merged

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsvorschau

This section provides a brief review of regular expression construction and matching. In particular, it describes the POSIX BRE and ERE constructs, which are intended to formalize the two basic "flavors" of regular expressions found among most Unix utilities.

We expect that you've had some exposure to regular expressions and text matching prior to this book. In that case, these subsections summarize how you can expect to use regular expressions for portable shell scripting.

If you've had no exposure at all to regular expressions, the material here may be a little too condensed for you, and you should detour to a more introductory source, such as Learning the Unix Operating System (O'Reilly) or sed & awk (O'Reilly). Since regular expressions are a fundamental part of the Unix tool-using and tool-building paradigms, any investment you make in learning how to use them, and use them well, will be amply rewarded, multifold, time after time.

If, on the other hand, you've been chopping, slicing, and dicing text with regular expressions for years, you may find our coverage cursory. If such is the case, we recommend that you review the first part, which summarizes POSIX BREs and EREs in tabular form, skip the rest of the section, and move on to a more in-depth source, such as Mastering Regular Expressions (O'Reilly).

Regular expressions are a notation that lets you search for text that fits a particular criterion, such as "starts with the letter a." The notation lets you write a single expression that can select, or match, multiple data strings.

Above and beyond traditional Unix regular expression notation, POSIX regular expressions let you:

Write regular expressions that express locale-specific character sequence orderings and equivalences
Write your regular expressions in a way that does not depend upon the underlying character set of the system

Ende der Inhaltsvorschau. Der weiterere Inhalt dieses Abschnitts ist hier nicht einsehbar.

Inhaltsverzeichnis

Themen

Buchreihen

Special Interest

International Sites