The shell's job, then, is to translate the user's command lines into operating system instructions. For example, consider this command line:

sort -n phonelist > phonelist.sorted

This means, "Sort lines in the file phonelist in numerical order, and put the result in the file phonelist.sorted." Here's what the shell does with this command:

Of course, each step really involves several substeps, and each substep includes a particular instruction to the underlying operating system.

Remember that the shell itself is not Unix — just the user interface to it. This is illustrated in Figure 1-1. Unix is one of the first operating systems to make the user interface independent of the operating system.

In this book, you will learn about the Korn shell, which is the most recent and powerful of the shells distributed with commercial Unix systems. There are two ways to use the Korn shell: as a user interface and as a programming environment.

This chapter and the next cover interactive use. These two chapters should give you enough background to use the shell confidently and productively for most of your everyday tasks.

After you have been using the shell for a while, you will undoubtedly find certain characteristics of your environment (the shell's "look and feel") that you would like to change and tasks that you would like to automate. Chapter 3 shows several ways of doing this.

Chapter 3 also prepares you for shell programming, the bulk of which is covered in Chapter 4 through Chapter 6. You need not have any programming experience to understand these chapters and learn shell programming. Chapter 7 and Chapter 8 give more complete descriptions of the shell's I/O and process handling capabilities, and Chapter 9 discusses various techniques for finding and removing problems in your shell programs.

You'll learn a lot about the Korn shell in this book; you'll also learn about Unix utilities and the way the Unix operating system works in general. It's possible to become a virtuoso shell programmer without any previous programming experience. At the same time, we've carefully avoided going down past a certain level of detail about Unix internals. We maintain that you shouldn't have to be an internals expert to use and program the shell effectively, and we won't dwell on the few shell features that are intended specifically for low-level systems programmers.

The independence of the shell from the Unix operating system per se has led to the development of dozens of shells throughout Unix history, though only a few have achieved widespread use.

The first major shell was the Bourne shell (named after its inventor, Stephen Bourne); it was included in the first widely popular version of Unix, Version 7, starting in 1979. The Bourne shell is known on the system as sh. Although Unix has gone through many, many changes, the Bourne shell is still popular and essentially unchanged. Several Unix utilities and administration features depend on it.

The first widely used alternative shell was the C shell, or csh. It was written by Bill Joy at the University of California at Berkeley as part of the Berkeley Software Distribution (BSD) version of Unix that came out a couple of years after Version 7. It's included in essentially all recent Unix versions. (A popular variant is the so-called Twenex csh, tcsh.)

The C shell gets its name from the resemblance of its commands to statements in the C programming language, which makes the shell easier for programmers on Unix systems to learn. It supports a number of operating system features (e.g., job control; see Chapter 8) that were once unique to BSD Unix but by now have migrated to just about all other modern versions. It also has a few important features (e.g., aliases; see Chapter 3) that make it easier to use in general.

This book covers the 1993 version of the Korn shell. A large amount of what's covered is unique to that shell; a subset of what is unique applies only to the recent versions available directly from AT&T. In order to make best use of the book, you should be using the 1993 Korn shell. Use the following sequence of instructions to determine what shell you currently have and whether the 1993 Korn shell exists on your system, and to make the 1993 Korn shell be your login shell.

When you use the shell interactively, you engage in a login session that begins when you log in and ends when you exit or press CTRL-D. During a login session, you type command lines into the shell; these are lines of text ending in ENTER that you type into your terminal or workstation. By default, the shell prompts you for each command with a dollar sign, though, as you will see in Chapter 3 the prompt can be changed.

Although arguments to commands aren't always files, files are the most important types of "things" on any Unix system. A file can contain any kind of information, and there are different types of files. Four types are by far the most important:

The software field — really, any scientific field — tends to advance most quickly and impressively on those few occasions when someone (i.e., not a committee) comes up with an idea that is small in concept yet enormous in its implications. The standard input and output scheme of Unix has to be on the short list of such ideas, along with such classic innovations as the LISP language, the relational data model, and object-oriented programming.

The Unix I/O scheme is based on two dazzlingly simple ideas. First, Unix file I/O takes the form of arbitrarily long sequences of characters (bytes). In contrast, file systems of older vintage have more complicated I/O schemes (e.g., "block," "record," "card image," etc.). Second, everything on the system that produces or accepts data is treated as a file; this includes hardware devices like disk drives and terminals. Older systems treated every device differently. Both of these ideas have made systems programmers' lives much more pleasant.

Pipes are actually a special case of a more general feature: doing more than one thing at a time. any other commercial operating systems don't have this capability, because of the rigid limits that they tend to impose upon users. Unix, on the other hand, was developed in a research lab and meant for internal use, so it does relatively little to impose limits on the resources available to users on a computer — as usual, leaning towards uncluttered simplicity rather than overcomplexity.

"Doing more than one thing at a time" means running more than one program at the same time. You do this when you invoke a pipeline; you can also do it by logging on to a Unix system as many times simultaneously as you wish. (If you try that on an IBM VM/CMS system, for example, you get an obnoxious "already logged in" message.)

The shell also lets you run more than one command at a time during a single login session. Normally, when you type a command and hit ENTER, the shell lets the command have control of your terminal until it is done; you can't run further commands until the first one finishes. But if you want to run a command that does not require user input and you want to do other things while the command is running, put an ampersand (&) after the command.

This is called running the command in the background, and a command that runs in this way is called a background job; for contrast, a job run the normal way is called a foreground job. When you start a background job, you get your shell prompt back immediately, enabling you to enter other commands.

The most obvious use for background jobs is programs that can take a long time to run, such as sort or gunzip on large files. For example, assume you just got an enormous compressed file loaded into your directory from magnetic tape. Today, the gzip utility is the de-facto file compression utility. gzip often achieves 50% to 90% compression of its input files. The compressed files have names of the form

The characters <, >, |, and & are four examples of special characters that have particular meanings to the shell. The wildcards we saw earlier in this chapter (*, ?, and [...]) are also special characters.

Table 1-6 gives indications of the meanings of all special characters within shell command lines only. Other characters have special meanings in specific situations, such as the regular expressions and string-handling operators we'll see in Chapter 3 and Chapter 4.

There are two ways of entering either editing mode. First, you can set your editing mode by using the environment variable VISUAL. The Korn shell checks to see if this variable ends with vi or macs. An excellent way to set VISUAL is to put a line like the following in your .profile or environment file:

VISUAL=$(whence emacs)

or

VISUAL=$(whence vi)

As you will find out in Chapter 3 and Chapter 4, the whence built-in command takes the name of another command as its argument and writes the command's full pathname on the standard output; the form $( command ) returns the standard output generated by command as a string value. Thus, the line above finds out the full pathname of your favorite editor and stores it in the environment variable VISUAL. The advantage of this code is that it is portable to other systems, which may have the executables for editors stored in different directories.

The second way of selecting an editing mode is to set the option explicitly with the set -o command:

set -o emacs

or

set -o vi

vi users may wish to add:

set -o viraw

along with set -o vi. This enables TAB completion in recent versions of ksh93. The additional overhead, particularly on single-user systems, is nominal and, in any case, is no worse than that of emacs-mode. (Starting with ksh93n, the viraw option is automatically enabled when you use vi-mode.)

You will find that the vi and emacs editing modes are good at emulating the basic commands of these editors, but not advanced features; their main purpose is to let you transfer "finger habits" from your favorite editor to the shell.

All of the Korn shell's command history facilities depend on a file that stores commands as you type them in. This file is normally .sh_history in your home directory, but you can call it whatever you like by setting the environment variable HISTFILE (see Chapter 3). When you run one of the Korn shell's editing modes, you are actually running a mini-editor on your history file.

If you run more than one login session at a time (e.g., more than one xterm on an X Windows workstation), you may find it advantageous to maintain a separate history file for each login session. Put the following line in your .profile:

HISTFILE=~/.hist.$(tty | sed 's;.*/;;')

This creates a history file whose name ends with the last component of your terminal's device name. For example, your window's terminal device name might be /dev/pts/42. The sed command strips everything through the last slash, leaving just the 42. The history file then becomes ~/.hist.42. You can remove the history file at logout, as explained in Chapter 4. Or you can leave the files around, and your history will be there the next time you start a window on that same terminal device. (Preserving history between sessions is the point of the history file, after all.)

An attractive alternative is to use a single history file for all your windows. Each running instance of the Korn shell is smart enough to share its file with other running instances; from a second window, you can recall and edit the commands run in the first window.

Another environment variable, HISTSIZE, can be used to determine the maximum number of commands accessible from the history file. The default is 128 (i.e., the 128 most recent commands), which should be more than adequate.

If you are an Emacs user, you will find it most useful to think of emacs editing mode as a simplified Emacs with a single, one-line window. All of the basic commands are available for cursor motion, cut and paste, and search.

Like emacs-mode, vi-mode essentially creates a one-line editing window into the history file. Vi-mode is popular because vi is the most standard Unix editor. But the function for which vi was designed, writing C programs, has different editing requirements from those of command interpreters. As a result, although it is possible to do complex things in vi with relatively few keystrokes, the relatively simple things you need to do in the Korn shell sometimes take too many keystrokes.

Like vi, vi-mode has two modes of its own: input and control mode. The former is for typing commands (as in normal Korn shell use); the latter is for moving around the command line and the history file. When you are in input mode, you can type commands and hit ENTER to run them. In addition, you have minimal editing capabilities via control characters, which are summarized in Table 2-6.

Under normal circumstances, you just stay in input mode. But if you want to go back and make changes to your command line, or if you want to recall previous commands, you need to go into control mode. To do this, hit ESC.

hist is a shell built-in command that provides a superset of the C shell history mechanism. You can use it to examine the most recent commands you entered, to edit one or more commands with your favorite "real" editor, and to run old commands with changes without having to type the entire command in again. We'll look at each of these uses.

The -l option for hist lists previous commands. It takes arguments that refer to commands in the history file. Arguments can be numbers or alphanumeric strings; numbers refer to the commands in the history file, while strings refer to the most recent command beginning with the string. hist treats arguments in a rather complex way:

A few examples should make these options clearer. Let's say you logged in and entered these commands:

ls -l

more myfile

vi myfile

wc -l myfile

pr myfile | lp -h

If you type hist -l (or history) with no arguments, you will see the above list with command numbers, as in:

1       ls -l

2       more myfile

3       vi myfile

4       wc -l myfile

5       pr myfile | lp -h

To paraphrase the old adage, old finger habits die hard. In fact, that is the primary reason for the choices of vi and Emacs for the Korn shell's editing modes. If you are an experienced user of one of these editors, by all means use the corresponding Korn shell editing mode. If you are a vi wizard, you probably know how to navigate between any two points on a line in three keystrokes or less.

But if you're not, you should seriously consider adopting emacs-mode finger habits. Because it is based on control keys, just like the minimal editing support you may have already used with the Bourne or C shell, you will find emacs-mode easier to assimilate. Although the full Emacs is an extremely powerful editor, its command structure lends itself very well to small subsetting: there are several "mini-emacs" style editors floating around for Unix, MS-DOS, and other systems.

The same cannot be said for vi, because its command structure is really meant for use in a full-screen editor. vi is quite powerful too, in its way, but its power becomes evident only when it is used for purposes similar to that for which it was designed: editing source code in C and LISP. Doing complicated things in vi takes relatively few keystrokes. But doing simple things takes more keystrokes in vi-mode than it does in emacs-mode. Unfortunately, the ability to do simple things with a minimal number of keystrokes is most desired in a command interpreter, especially nowadays when users are spending more time within applications and less time working with the shell.

Both Korn shell editing modes have quite a few commands; you will undoubtedly develop finger habits that include just a few of them. If you use emacs-mode and you aren't familiar with the full Emacs, here is a subset that is easy to learn yet enables you to do just about anything:

If you want to customize your environment, it is most important to know about a file called .profile in your home (login) directory. This is a file of shell commands, also called a shell script, that the Korn shell reads and runs whenever you log in to your system.

If you use a large machine in an office or department, the odds are good that your system administrator has already set up a .profile file for you that contains a few standard things. This is one of the "hidden" files mentioned in Chapter 1; other common hidden files include .xinitrc (for the X Window System), .emacs (for the GNU Emacs editor), and .mailrc (for the Unix mail program).

Your .profile, together with the environment file that we discuss towards the end of this chapter, will be the source of practically all of the customizations we discuss here as well as in subsequent chapters. Therefore, it is very important for you to become comfortable with a text editor like vi or Emacs so that you can try whatever customization techniques strike your fancy.

Bear in mind, however, that if you add commands to your .profile, they will not take effect until you log out and log back in again, or type the command login. Of course, you need not immediately add customization commands to your .profile — you can always just test them by typing them in yourself. (Be sure you test your changes though: it is possible to set things up in your .profile such that you can't log back in! Test your changes before logging out, by logging in again, perhaps from a new window or virtual console.)

If you already have a .profile, it's likely to contain lines similar to some of these:

PATH=/sbin:/usr/sbin:/usr/bin:/etc:/usr/ucb:/local/bin:

stty stop ^S intr ^C erase ^?

EDITOR=/usr/local/bin/emacs

SHELL=/bin/ksh

export EDITOR

These commands set up a basic environment for you, so you probably shouldn't change them until you learn about what they do — which you will by the end of this chapter. When you edit your

Perhaps the easiest and most popular type of customization is the alias, which is a synonym for a command or command string. This is one of several Korn shell features that were appropriated from the C shell. You define an alias by entering (or adding to your .profile) a line with the following form:

alias new=original

            

(Notice that there are no spaces on either side of the equal sign (=); this is required syntax.) The alias command defines new to be an alias for original; whenever you type new, the Korn shell substitutes original internally. (You cannot use any of the shell's special characters, such as *, $, =, and so on, in alias names.)

There are a few basic ways to use an alias. The first, and simplest, is as a more mnemonic name for an existing command. Many commonly used Unix commands have names that are poor mnemonics and therefore are excellent candidates for aliasing; the classic example is:

alias search=grep

grep, the Unix file-searching utility, derives its name from the command "g/re/p" in the original ed text editor, which does essentially the same thing as grep. (The regular expression matching code was carved out of ed to make a separate program.) This acronym may mean something to a computer scientist but probably not to the office administrator who has to find Fred in a list of phone numbers. If you have to find Fred, and you have the word search defined as an alias for grep, you can type:

search Fred phonelist

Another popular alias eschews exit in favor of a more widely used command for ending a login session:

alias logout=exit

If you are a C shell user, you may be used to having a .logout file of commands that the shell executes just before you log out. The Korn shell doesn't have this feature as such, but you can mimic it quite easily using an alias:

While aliases let you create convenient names for commands, they don't really let you change the shell's behavior. Options are one way of doing this. A shell option is a setting that is either "on" or "off." While several options relate to arcane shell features that are of interest only to programmers, those that we cover here are of interest to all users.

The basic commands that relate to options are set -o optionnames and set +o optionnames, where optionnames is a list of option names separated by whitespace. The use of plus (+) and minus (-) signs is counterintuitive: the - turns the named option on, while the + turns it off. The reason for this incongruity is that the dash (-) is the conventional Unix way of specifying options to a command, while the use of + is an afterthought.

Most options also have one-letter abbreviations that can be used in lieu of the set -o command; for example, set -o noglob can be abbreviated set -f. These abbreviations are carry-overs from the Bourne shell. Like several other "extra" Korn shell features, they exist to ensure upward compatibility; otherwise, their use is not encouraged.

Table 3-1 lists the options that are useful to general Unix users. All of them are off by default except as noted.

There are several characteristics of your environment that you may want to customize but that cannot be expressed as an on/off choice. Characteristics of this type are specified in shell variables. Shell variables can specify everything from your prompt string to how often the shell checks for new mail.

Like an alias, a shell variable is a name that has a value associated with it. The Korn shell keeps track of several built-in shell variables; shell programmers can add their own. By convention, built-in variables have names in all capital letters. The syntax for defining variables is somewhat similar to the syntax for aliases:

               varname=value

            

There must be no space on either side of the equal sign, and if the value is more than one word, it must be surrounded by quotes. To use the value of a variable in a command, precede its name by a dollar sign ($).

You can delete a variable with the command unset varname. Normally, this isn't useful, since all variables that don't exist are assumed to be null, i.e., equal to the empty string "". But if you use the option nounset (see Table 3-1), which causes the shell to indicate an error when it encounters an undefined variable, you may be interested in unset.

The easiest way to check a variable's value is to use the print built-in command. All print does is print its arguments, but not until the shell has evaluated them. This includes — among other things that will be discussed later — taking the values of variables and expanding filename wildcards. So, if the variable fred has the value bob, typing the following causes the shell to simply print bob:

print "$fred"

If the variable is undefined, the shell prints a blank line. A more verbose way to do this is:

print "The value of \$varname is \"$varname\"."

Some of the variables discussed above are used by commands you may run — as opposed to the shell itself — so that they can determine certain aspects of your environment. The majority, however, are not even known outside the shell.

This dichotomy begs an important question: which shell "things" are known outside the shell, and which are only internal? This question is at the heart of many misunderstandings about the shell and shell programming. Before we answer, we'll ask it again in a more precise way: which shell "things" are known to subprocesses? Remember that whenever you enter a command, you are telling the shell to run that command in a subprocess; furthermore, some complex programs may start their own subprocesses.

The answer is actually fairly simple. Subprocesses inherit only environment variables. They are available automatically, without the subprocess having to take any explicit action. All the other "things" — shell options, aliases, and functions — must be made explicitly available. The environment file is how you do this. Furthermore, only interactive shells process the environment file. The next two sections describe environment variables and the environment file, respectively.

You should feel free to try any of the techniques presented in this chapter. The best strategy is to test something out by typing it into the shell during your login session; if you decide you want to make it a permanent part of your environment, add it to your .profile.

A nice, painless way to add to your .profile without going into a text editor makes use of the print command and one of the Korn shell's editing modes. If you type a customization command in and later decide to add it to your .profile, you can recall it via CTRL-P or CTRL-R (in emacs-mode) or j, -, or / (vi-mode). Let's say the line is:

PS1="($LOGNAME !)->"

After you recall it, edit it so that it is preceded by a print command, surrounded by single quotes, and followed by an I/O redirector that (as you will see in Chapter 7) appends the output to ~/.profile:

print 'PS1="($LOGNAME !)->"' >> ~/.profile

Remember that the single quotes are important because they prevent the shell from trying to interpret things like dollar signs, double quotes, and exclamation points.

You should also feel free to snoop around other peoples' .profile files for customization ideas. A quick way to examine everyone's .profile is as follows: let's assume that all login directories are under /home. Then you can type:

cat /home/*/.profile > ~/other_profiles

and examine other people's .profile files with a text editor at your leisure (assuming you have read permission on them). If other users have environment files, the file you just created will show what they are, and you can examine them as well.

Finally, be sure that no one else but you has write permission on your .profile and environment files.

A script, or file that contains shell commands, is a shell program. Your .profile and environment files, discussed in Chapter 3, are shell scripts.

You can create a script using the text editor of your choice. Once you have created one, there are a number of ways to run it. One, which we have already covered, is to type . scriptname (i.e., the command is a dot). This causes the commands in the script to be read and run as if you typed them in.

Two more ways are to type ksh script or ksh < script. These explicitly invoke the Korn shell on the script, requiring that you (and your users) be aware that they are scripts.

The final way to run a script is simply to type its name and hit ENTER, just as if you were invoking a built-in command. This, of course, is the most convenient way. This method makes the script look just like any other Unix command, and in fact several "regular" commands are implemented as shell scripts (i.e., not as programs originally written in C or some other language), including spell, man on some systems, and various commands for system administrators. The resulting lack of distinction between "user command files" and "built-in commands" is one factor in Unix's extensibility and, hence, its favored status among programmers.

You can run a script by typing its name only if . (the current directory) is part of your command search path, i.e., is included in your PATH variable (as discussed in Chapter 3). If . isn't on your path, you must type ./ scriptname, which is really the same thing as typing the script's relative pathname (see Chapter 1).

Before you can invoke the shell script by name, you must also give it "execute" permission. If you are familiar with the Unix filesystem, you know that files have three types of permissions (read, write, and execute) and that those permissions apply to three categories of user (the file's owner, a group of users, and everyone else). Normally, when you create a file with a text editor, the file is set up with read and write permission for you and read-only permission for everyone else.

A major piece of the Korn shell's programming functionality relates to shell variables. We've already seen the basics of variables. To recap briefly: they are named places to store data, usually in the form of character strings, and their values can be obtained by preceding their names with dollar signs ($). Certain variables, called environment variables, are conventionally named in all capital letters, and their values are made known (with the export statement) to subprocesses.

This section presents the basics for shell variables. Discussion of certain advanced features is delayed until later in the chapter, after covering regular expressions.

If you are a programmer, you already know that just about every major programming language uses variables in some way; in fact, an important way of characterizing differences between languages is comparing their facilities for variables.

The chief difference between the Korn shell's variable schema and those of conventional languages is that the Korn shell's schema places heavy emphasis on character strings. (Thus it has more in common with a special-purpose language like SNOBOL than a general-purpose one like Pascal.) This is also true of the Bourne shell and the C shell, but the Korn shell goes beyond them by having additional mechanisms for handling integers and double-precision floating point numbers explicitly, as well as simple arrays.

$ fred=bob

$ print "$fred"

bob

ksh93 introduces a new feature, called compound variables. They are similar in nature to a Pascal or Ada record or a C struct, and they allow you to group related items together under the same name. Here are some examples:

now="May 20 2001 19:44:57"        Assign current date to variable now

now.hour=19                       Set the hour

now.minute=44                     Set the minute

...

Note the use of the period in the variable's name. Here, now is called the parent variable, and it must exist (i.e., have a value) before you can assign a value to an individual component (such as hour or minute). To access a compound variable, you must enclose the variable's name in curly braces. If you don't, the period ends the shell's scan for the variable's name:

$ print ${now.hour}

19

$ print $now.hour

May 20 2001 19:44:57.hour

person="John Q. Public"

person.firstname=John

person.initial=Q.

person.lastname=Public

person=(firstname=John initial=Q. lastname=Public)

$ print $person                                

                  Simple print

( lastname=Public initial=Q. firstname=John )

$ print -r "$person"                           

                  Print in full glory

(

        lastname=Public

        initial=Q.

        firstname=John

)

$ print ${person.initial}                      

                  Print just the middle initial

Q.

Most of the time, as we've seen so far, you manipulate variables directly, by name (x=1, for example). The Korn shell allows you to manipulate variables indirectly, using something called a nameref. You create a nameref using typeset -n, or the more convenient predefined alias, nameref. Here is a simple example:

$ name="bill"                       

               Set initial value

$ nameref firstname=name            

               Set  up the nameref

$ print $firstname                  

               Actually references variable name

bill

$ firstname="arnold"                

               Now change the indirect reference

$ print $name                       

               Shazzam! Original variable is changed

arnold

To find out the name of the real variable being referenced by the nameref, use ${! variable }:

$ print ${!firstname}

name

At first glance, this doesn't seem to be very useful. The power of namerefs comes into play when you pass a variable's name to a function, and you want that function to be able to update the value of that variable. The following example illustrates how it works:

$ date                                     

               Current day and time

Wed May 23 17:49:44 IDT 2001

$ function getday {                        

               Define a function

>     typeset -n day=$1                    

               Set up the nameref

>     day=$(date | awk '{ print $1 }')     

               Actually change it

> }

$ today=now                                

               Set initial value

$ getday today                             

               Run the function

$ print $today                             

               Display new value

Wed

The default output of date(1) looks like this:

$ date

Wed Nov 14 11:52:38 IST 2001

The getday function uses awk to print the first field, which is the day of the week. The result of this operation, which is done inside command substitution (described later in this chapter), is assigned to the local variable

The curly-brace syntax allows for the shell's string operators. String operators allow you to manipulate values of variables in various useful ways without having to write full-blown programs or resort to external Unix utilities. You can do a lot with string-handling operators even if you haven't yet mastered the programming features we'll see in later chapters.

In particular, string operators let you do the following:

From the discussion so far, we've seen two ways of getting values into variables: by assignment statements and by the user supplying them as command-line arguments (positional parameters). There is another way: command substitution, which allows you to use the standard output of a command as if it were the value of a variable. You will soon see how powerful this feature is.

The syntax of command substitution is:

$(Unix command)

The command inside the parenthesis is run, and anything the command writes to standard output (and to standard error) is returned as the value of the expression. These constructs can be nested, i.e., the Unix command can contain command substitutions.

Here are some simple examples:

We conclude this chapter with a couple of functions that you may find handy in your everyday Unix use. They solve the problem presented by Task 4-7.

We start by implementing a significant subset of their capabilities and finish the implementation in Chapter 6. (For ease of development and explanation, our implementation ignores some things that a more bullet-proof version should handle. For example, spaces in filenames will cause things to break.)

If you don't know what a stack is, think of a spring-loaded dish receptacle in a cafeteria. When you place dishes on the receptacle, the spring compresses so that the top stays at roughly the same level. The dish most recently placed on the stack is the first to be taken when someone wants food; thus, the stack is known as a "last-in, first-out" or LIFO structure. (Victims of a recession or company takeovers will also recognize this mechanism in the context of corporate layoff policies.) Putting something onto a stack is known in computer science parlance as pushing, and taking something off the top is called popping.

A stack is very handy for remembering directories, as we will see; it can "hold your place" up to an arbitrary number of times. The cd - form of the cd command does this, but only to one level. For example: if you are in firstdir and then you change to seconddir, you can type cd - to go back. But if you start out in firstdir, then change to seconddir, and then go to thirddir, you can use cd - only to go back to seconddir. If you type cd - again, you will be back in

The simplest type of flow control construct is the conditional, embodied in the Korn shell's if statement. You use a conditional when you want to choose whether or not to do something, or to choose among a small number of things to do, according to the truth or falsehood of conditions. Conditions test values of shell variables, characteristics of files, whether or not commands run successfully, and other factors. The shell has a large set of built-in tests that are relevant to the task of shell programming.

The if construct has the following syntax:

if condition

then

    statements

[elif condition

    then statements ...]

[else

    statements]

fi

The simplest form (without the elif and else parts, a.k.a. clauses) executes the statements only if the condition is true. If you add an else clause, you get the ability to execute one set of statements if a condition is true or another set of statements if the condition is false. You can use as many elif (a contraction of "else if") clauses as you wish; they introduce more conditions and thus more choices for which set of statements to execute. If you use one or more elifs, you can think of the else clause as the "if all else fails" part.

The most obvious enhancement we could make to the previous script is the ability to report on multiple files instead of just one. Tests like -e and -d only take single arguments, so we need a way of calling the code once for each file given on the command line.

The way to do this — indeed, the way to do many things with the Korn shell — is with a looping construct. The simplest and most widely applicable of the shell's looping constructs is the for loop. We'll use for to enhance fileinfo soon.

The for loop allows you to repeat a section of code a fixed number of times. During each time through the code (known as an iteration), a special variable called a loop variable is set to a different value; this way each iteration can do something slightly different.

The for loop is somewhat, but not entirely, similar to its counterparts in conventional languages like C and Pascal. The chief difference is that the shell's for loop doesn't let you specify a number of times to iterate or a range of values over which to iterate; instead, it only lets you give a fixed list of values. In other words, with the normal for loop, you can't do anything like this Pascal-type code, which executes statements 10 times:

for x := 1 to 10 do

begin

    statements ...

end

(You need the arithmetic for loop, which we'll see in Chapter 6, to do that.)

However, the for loop is ideal for working with arguments on the command line and with sets of files (e.g., all files in a given directory). We'll look at an example of each of these. But first, here is the syntax for the for construct:

for name [in list]

do

    statements that can use $name ...

done

The list is a list of names. (If in list is omitted, the list defaults to "$@", i.e., the quoted list of command-line arguments, but we always supply the in list for the sake of clarity.) In our solutions to the following task, we show two simple ways to specify lists.

The next flow control construct to cover is case. While the case statement in Pascal and the similar switch statement in C can be used to test simple values like integers and characters, the Korn shell's case construct lets you test strings against patterns that can contain wildcard characters. Like its conventional language counterparts, case lets you express a series of if-then-else type statements in a concise way.

The syntax of case is as follows:

case expression in

    pattern1 )

        statements ;;

               pattern2 )

        statements ;;

    ...

esac

Any of the patterns can actually be several patterns separated by "OR bar" characters (|, which is the same as the pipe symbol, but in this context means "or"). If expression matches one of the patterns, its corresponding statements are executed. If there are several patterns separated by OR bars, the expression can match any of them in order for the associated statements to be run. The patterns are checked in order until a match is found; if none is found, nothing happens.

This rather ungainly syntax should become clearer with an example. An obvious choice is to revisit our solution to Task 4-2, the front-end for the C compiler. Earlier in this chapter, we wrote some code that processed input files according to their suffixes (.c, .s, or .o for C, assembly, or object code, respectively).

We can improve upon this solution in two ways. First, we can use for to allow multiple files to be processed at one time; second, we can use case to streamline the code:

for filename in "$@"; do

    case $filename in

        *.c )

            objname=${filename%.c}.o

            ccom "$filename" "$objname" ;;

        *.s )

            objname=${filename%.s}.o

            as "$filename" "$objname" ;;

        *.o ) ;;

        *   )

            print "error: $filename is not a source or object file."

            exit 1 ;;

    esac

done

Almost all of the flow-control constructs we have seen so far are also available in the Bourne shell, and the C shell has equivalents with different syntax. Our next construct, select, is unique to the Korn shell; moreover, it has no analogue in conventional programming languages.

select allows you to generate simple menus easily. It has concise syntax, but it does quite a lot of work. The syntax is:

select name [in list]

do

    statements that can use $name ...

done

This is the same syntax as the regular for loop except for the keyword select. And like for, you can omit in list, and it will default to "$@", i.e., the list of quoted command-line arguments.

Here is what select does:

Once again, an example should help make this process clearer. Assume you need to write the code for Task 5-4, but your life is not as simple. You don't have terminals hardwired to your computer; instead, your users communicate through a terminal server, or they log in remotely, via telnet or ssh. This means, among other things, that the tty number does not determine the type of terminal.

Therefore, you have no choice but to prompt the user for his or her terminal type at login time. To do this, you can put the following code in /etc/profile (assume you have the same choice of terminal types):

The remaining two flow control constructs the Korn shell provides are while and until. These are similar; both allow a section of code to be run repetitively while (or until) a certain condition holds true. They also resemble analogous constructs in Pascal (while/do and repeat/until) and C (while and do/while).

while and until are actually most useful when combined with features we will see in the next chapter, such as arithmetic, input/output of variables, and command-line processing. Yet we can show a useful example even with the machinery we have covered so far.

The syntax for while is:

while condition

do

    statements ...

done

For until, just substitute until for while in the above example. As with if, the condition is really a list of statements that are run; the exit status of the last one is used as the value of the condition. You can use a conditional with [[ and ]] here, just as you can with if.

The result is that you can convert any until into a while simply by negating the condition. The only place where until might be better is something like this:

until command; do

    statements ...

done

The meaning of this is essentially, "Do statements until command runs correctly." This is occasionally useful, such as when waiting for the occurrence of a particular event. However, we use while throughout the rest of this book.

Task 5-5 is a good candidate for while

We have already seen many examples of the positional parameters (variables called 1, 2, 3, etc.) that the shell uses to store the command-line arguments to a shell script or function when it runs. We have also seen related variables like * and @ (for the string(s) of all arguments) and # (for the number of arguments).

Indeed, these variables hold all the information on the user's command line. But consider what happens when options are involved. Typical Unix commands have the form command [-options] args, meaning that there can be zero or more options. If a shell script processes the command fred bob pete, then $1 is ``bob'' and $2 is ``pete''. But if the command is fred -o bob pete, then $1 is -o, $2 is ``bob'', and $3 is ``pete''.

You might think you could write code like this to handle it:

if [[ $1 == -o ]]; then

    code that processes the -o option

    1=$2

    2=$3

fi

normal processing of $1 and $2...

            

But this code has several problems. First, assignments like 1=$2 are illegal because positional parameters are read-only. Even if they were legal, another problem is that this kind of code imposes limitations on how many arguments the script can handle — which is very unwise. Furthermore, if this command had several possible options, the code to handle all of them would get very messy very quickly.

1=$2

2=$3

...

1=$4

2=$5

...

The expression $(($OPTIND - 1)) in the last example gives a clue as to how the shell can do integer arithmetic. As you might guess, the shell interprets words surrounded by $(( and )) as arithmetic expressions. Variables in arithmetic expressions do not need to be preceded by dollar signs. It is OK to supply the dollar sign, except when assigning a value to a variable.

Arithmetic expressions are evaluated inside double quotes, like variables and command substitutions. We're finally in a position to state the definitive rule about quoting strings: When in doubt, enclose a string in single quotes, unless it contains any expression involving a dollar sign, in which case you should use double quotes.

For example, the date(1) command on modern versions of Unix accepts arguments that tell it how to format its output. The argument +%j tells it to print the day of the year, i.e., the number of days since December 31st of the previous year.

We can use +%j to print a little holiday anticipation message:

print "Only $(( (365-$(date +%j)) / 7 )) weeks until the New Year!"

We'll show where this fits in the overall scheme of command-line processing in Chapter 7.

The arithmetic expression feature is built in to the Korn shell's syntax, and it was available in the Bourne shell (most versions) only through the external command expr(1). Thus it is yet another example of a desirable feature provided by an external command (i.e., a syntactic kludge) being better integrated into the shell. [[...]] and getopts are also examples of this design trend.

While expr and ksh88 were limited to integer arithmetic, ksh93 supports floating-point arithmetic. As we'll see shortly, you can do just about any calculation in the Korn shell that you could do in C or most other programming languages.

The for loop as described in Chapter 5 has been in Unix shells since the Version 7 Bourne Shell. As mentioned, you can't do Pascal or C-style looping for a fixed number of iterations with that for loop. ksh93 introduced the arithmetic for loop to remedy this situation and to bring the shell closer to a traditional (some would say "real") programming language.

The syntax resembles the shell's arithmetic facilities that we have just seen. It is almost identical to the syntax of the C for loop, except for the extra set of parentheses:

for ((init; condition; increment))

do

    statements ...

done

Here, init represents something to be done once, at the start of the loop. The condition is tested, and as long as it's true, the shell executes statements. Before going back to the top of the loop to test the condition again, the shell executes increment.

Any of init, condition, and increment may be omitted. A missing condition is treated as true; i.e., the loop body always executes. (The so-called "infinite loop" requires you to use some other method to leave the loop.) We'll use the arithmetic for loop for Task 6-2, which is our next, rather simple task.

Here's the code; the explanation follows:

sum=0

count=$#

for ((i = 1; i <= count; i++))

do

    let "sum += $1"

    shift

done

print $sum

The first line initializes the variable sum to 0. sum accumulates the sum of the arguments. The second line is mostly for readability; count indicates how many arguments there are. The third line starts the loop itself. The variable i is the loop control variable. The init clause sets it to 1, the condition clause tests it against the limit count, and the increment clause adds 1 to it each time around the loop. One thing you'll notice right away is that inside the for loop header, there's no need for the

So far we have seen three types of variables: character strings, integers, and floating-point numbers. The fourth type of variable that the Korn shell supports is an array. As you may know, an array is like a list of things; you can refer to specific elements in an array with indices, so that a[i] refers to the ith element of array a. The Korn shell provides two kinds of arrays: indexed arrays, and associative arrays.

nicknames[2]=bob

nicknames[3]=ed

set -A aname val1 val2 val3 ...

The final Korn shell feature that relates to the kinds of values that variables can hold is the typeset command. If you are a programmer, you might guess that typeset is used to specify the type of a variable (integer, string, etc.); you'd be partially right.

typeset is a rather ad hoc collection of things that you can do to variables that restrict the kinds of values they can take. Operations are specified by options to typeset; the basic syntax is:

typeset option varname[=value]

Here, option is an option letter preceded with a hyphen or plus sign. Options can be combined and multiple varnames can be used. If you leave out varname, the shell prints a list of variables for which the given option is turned on.

The available options break down into two basic categories:

In Chapter 1 you learned about the shell's basic I/O redirectors, <, >, and |. Although these are enough to get you through 95% of your Unix life, you should know that the Korn shell supports a total of 20 I/O redirectors. Table 7-1 lists them, including the three we've already seen. Although some of the rest are useful, others are mainly for systems programmers. We will wait until the next chapter to discuss the last three, which, along with >| and <<<, are not present in most Bourne shell versions.

Now we'll zoom back in to the string I/O level and examine the print, printf, and read statements, which give the shell I/O capabilities that are more analogous to those of conventional programming languages.

We've seen how the shell processes input lines: it deals with single quotes (' '), double quotes (" "), and backslashes (\), and it separates parameter, command and arithmetic expansions into words, according to delimiters in the variable IFS. This is a subset of the things the shell does when processing command lines.

This section completes the discussion, in sometimes excruciating detail. We first examine two additional kinds of substitutions or expansions that the shell performs that may not be universally available. Then we present the full story of the order that the shell processes the command line. Covered next is the use of quoting, which prevents many or all of the substitution steps from occurring. Finally, we cover the eval command, which can be used for additional programmatic control of command line evaluations.

$ ls

cpp-args.c  cpp-lex.c  cpp-out.c  cpp-parse.c

$ print cpp-{args,l{e,o}x,parse}.c

cpp-args.c cpp-lex.c cpp-lox.c cpp-parse.c

Unix gives all processes numbers, called process IDs, when they are created. You will notice that, when you run a command in the background by appending & to it, the shell responds with a line that looks like this:

$ fred &

[1]     2349

In this example, 2349 is the process ID for the fred process. The [1] is a job number assigned by the shell (not the operating system). What's the difference? Job numbers refer to background processes that are currently running under your shell, while process IDs refer to all processes currently running on the entire system, for all users. The term job basically refers to a command line that was invoked from your login shell.

If you start up additional background jobs while the first one is still running, the shell numbers them 2, 3, etc. For example:

$ bob &

[2]     2367

$ dave | george &

[3]     2382

Clearly, 1, 2, and 3 are easier to remember than 2349, 2367, and 2382!

The shell includes job numbers in messages it prints when a background job completes, like this:

[1] +  Done                     fred &

We'll explain what the plus sign means soon. If the job exits with nonzero status (see Chapter 5), the shell includes the exit status in parentheses:

[1] +  Done(1)                  fred &

The shell prints other types of messages when certain abnormal things happen to background jobs; we'll see these later in this chapter.

Why should you care about process IDs or job numbers? Actually, you could probably get along fine in your Unix life without ever referring to process IDs (unless you use a windowing workstation — as we'll see soon). Job numbers are more important, however: you can use them with the shell commands for job control.

You already know the most obvious way to control a job: you can create one in the background with &. Once a job is running in the background, you can let it run to completion, bring it into the foreground, or send it a message called a signal.

[1]   Running                  fred &

[2] - Running                  bob &

[3] + Running                  dave | george &

[1]   2349      Running                  fred &

[2] - 2367      Running                  bob &

[3] + 2382      Running                  dave | george &

We said earlier that typing CTRL-Z to suspend a job is similar to typing CTRL-C to stop a job, except that you can resume the job later. They are actually similar in a deeper way: both are particular cases of the act of sending a signal to a process.

A signal is a message that one process sends to another when some abnormal event takes place or when it wants the other process to do something. Most often, a process sends a signal to a subprocess it created. You're undoubtedly already comfortable with the idea that one process can communicate with another through an I/O pipeline; think of a signal as another way for processes to communicate with each other. (In fact, any textbook on operating systems will tell you that both are examples of the general concept of interprocess communication, or IPC.)

Depending on the version of Unix, there are two or three dozen types of signals, including a few that can be used for whatever purpose a programmer wishes. Signals have numbers (from 1 to the number of signals the system supports) and names; we'll use the latter. You can get a list of all the signals on your system by typing kill -l. Bear in mind, when you write shell code involving signals, that signal names are more portable to other versions of Unix than signal numbers.

We've been discussing how signals affect the casual user; now let's talk a bit about how shell programmers can use them. We won't go into too much depth about this, because it's really the domain of systems programmers.

We mentioned earlier that programs in general can be set up to "trap" specific signals and process them in their own way. The trap built-in command lets you do this from within a shell script. trap is most important for "bullet-proofing" large shell programs so that they react appropriately to abnormal events — just as programs in any language should guard against invalid input. It's also important for certain systems programming tasks, as we'll see in the next chapter.

The syntax of trap is:

trap cmd sig1 sig2 ...

            

That is, when any of sig1, sig2, etc., are received, run cmd, then resume execution. After cmd finishes, the script resumes execution just after the command that was interrupted.

Of course, cmd can be a script or function. The sigs can be specified by name or by number. You can also invoke trap without arguments, in which case the shell prints a list of any traps that have been set, using symbolic names for the signals. If you use trap -p, the shell prints the trap settings in a way that can be saved and reread later by a different invocation of the shell.

The shell scans the text of cmd twice. The first time is while it is preparing to run the trap command; all the substitutions as outlined in Chapter 7 are performed before executing the trap command. The second time is when the shell actually executes the trap. For this reason, it is best to use single quotes around the cmd in the text of the shell program. When the shell executes the trap's command, $? is always the exit status of the last command run before the trap started. This is important for diagnostics.

Here's a simple example that shows how trap works. Suppose we have a shell script called

We've spent the last several pages on almost microscopic details of process behavior. Rather than continue our descent into the murky depths, we'll revert to a higher-level view of processes.

Earlier in this chapter, we covered ways of controlling multiple simultaneous jobs within an interactive login session; now we consider multiple process control within shell programs. When two (or more) processes are explicitly programmed to run simultaneously and possibly communicate with each other, we call them coroutines.

This is actually nothing new: a pipeline is an example of coroutines. The shell's pipeline construct encapsulates a fairly sophisticated set of rules about how processes interact with each other. If we take a closer look at these rules, we'll be better able to understand other ways of handling coroutines — most of which turn out to be simpler than pipelines.

When you invoke a simple pipeline, say ls | more, the shell invokes a series of Unix primitive operations, a.k.a. system calls. In effect, the shell tells Unix to do the following things; in case you're interested, we include in parentheses the actual system call used at each step:

You can probably imagine how the above steps change when the pipeline involves more than two processes.

Coroutines clearly represent the most complex relationship between processes that the Korn shell defines. To conclude this chapter, we will look at a much simpler type of interprocess relationship: that of a shell subprocess with its parent shell. We saw in Chapter 3 that whenever you run a shell script, you actually invoke another copy of the shell that is a subprocess of the main, or parent, shell process. Now let's look at them in more detail.

What sort of functionality do you need to debug a program? At the most empirical level, you need a way of determining what is causing your program to behave badly and where the problem is in the code. You usually start with an obvious what (such as an error message, inappropriate output, infinite loop, etc.), try to work backwards until you find a what that is closer to the actual problem (e.g., a variable with a bad value, a bad option to a command), and eventually arrive at the exact where in your program. Then you can worry about how to fix it.

Notice that these steps represent a process of starting with obvious information and ending up with often obscure facts gleaned through deduction and intuition. Debugging aids make it easier to deduce and intuit by providing relevant information easily or even automatically, preferably without modifying your code.

The simplest debugging aid (for any language) is the output statement, print in the shell's case. Indeed, old-time programmers debugged their Fortran code by inserting WRITE cards into their decks. You can debug by putting lots of print statements in your code (and removing them later), but you will have to spend lots of time narrowing down not only what exact information you want but also where you need to see it. You will also probably have to wade through lots and lots of output to find the information that you really want.

Commercially available debuggers give you much more functionality than the shell's set options and fake signals. The most advanced have fabulous graphical user interfaces, incremental compilers, symbolic evaluators, and other such amenities. But just about all modern debuggers — even the more modest ones — have features that enable you to "peek" into a program while it's running, to examine it in detail and in terms of its source language. Specifically, most debuggers let you do these things:

Our debugger, called kshdb, has these features and a few more. Although it's a basic tool, without too many bells and whistles, it is not a toy. This book's web site, http://www.oreilly.com/catalog/korn2/, has a link for a downloadable copy of all the book's example programs, including kshdb. If you don't have access to the Internet, you can type or scan the code in. Either way, you can use kshdb to debug your own shell scripts, and you should feel free to enhance it. This is version 2.0 of the debugger. It includes some changes suggested to us by Steve Alston, and the watchpoints feature is brand new. We'll suggest some enhancements at the end of this chapter.

As a prelude to system-wide customization, we want to emphasize something about the Korn shell that doesn't apply to most other shells: you can install it as if it were the standard Bourne shell, i.e., as /bin/sh. Just save the real Bourne shell as another filename, such as /bin/bsh, in case anyone actually needs it for anything (which is doubtful), then rename (or link) your Korn shell as /bin/sh.

Many installations have done this with absolutely no ill effects. Not only does this make the Korn shell your system's standard login shell, but it also makes most existing Bourne shell scripts run faster, and it has security advantages that we'll see later in this chapter.

As we will see in Appendix A, the Korn shell is backward-compatible with the Bourne shell except that it doesn't support ^ as a synonym for the pipe character |. Unless you have an ancient Unix system, or you have some very, very old shell scripts, you needn't worry about this.

But if you want to be absolutely sure, simply search through all shell scripts in all directories in your PATH. An easy way to do this is to use the file(1) command, which we saw in Chapter 5 and Chapter 9. file prints "executable shell script" when given the name of one. (The exact message varies from system to system; make sure that yours prints this message when given the name of a shell script. If not, just substitute the message your file command prints for "shell script" in the following example.) Here is a script that looks for ^ in shell scripts in every directory in your PATH:

dirs=$(print -- $PATH |

	sed -e 's/^:/.:/' -e 's/::/:.:/' -e s'/:$/:./' -e 's/:/ /g')

for d in $dirs

do

	print "checking $d:"

	cd "$d"

	scripts=$(file * | grep 'shell script' | cut -d: -f1)

	grep -l '\^' $scripts /dev/null

done

The first statement of this script pulls $PATH apart into separate directories, including handling the several cases of empty separators which signify the current directory. The

Like the Bourne shell, the Korn shell uses the file /etc/profile for system-wide customization. When a user logs in, the shell reads and runs /etc/profile before running the user's .profile.

We don't cover all the possible commands you might want to put in /etc/profile. But the Korn shell has a few unique features that are particularly relevant to system-wide customization; we discuss them here.

We'll start with two built-in commands that you can use in /etc/profile to tailor your users' environments and constrain their use of system resources. Users can also use these commands in their .profile, or at any other time, to override the default settings.

As we saw in Chapter 2, you have your choice of either emacs or vi editing modes when editing your command line. Besides the commands available in each mode, you can customize the behavior of the editing modes to suit your needs or environment.

Appendix A discusses a number of third party shells based on the Bourne and Korn shell design. Those shells generally provide command-line editing, as well as the ability to customize the editor via a special built-in command, a special start-up file, or both.

The Korn shell's approach is different. It is based on a paradigm where you program the behavior you want from the shell. This is accomplished via a fake trap, named KEYBD. If it exists, the trap set for KEYBD is evaluated when ksh processes normal command-line input characters. Within the code executed for the trap, two special variables contain the text of the command line and the text being entered that caused the trap. Additional special variables allow you to distinguish between emacs- and vi-modes and indicate the current position on the input line. These variables are listed in Table 10-2.

Unix security is a problem of legendary notoriety. Just about every aspect of a Unix system has some security issue associated with it, and it's usually the system administrator's job to worry about this issue.

We first present a list of "tips" for writing shell scripts that have a better chance of avoiding security problems. Next we cover the restricted shell, which attempts to put a straitjacket around the user's environment. Then we present the idea of a "trojan horse," and why such things should be avoided. Finally we discuss the Korn shell's privileged mode, which is used with shell scripts that run as if the user were root.

The Korn shell is almost completely backward-compatible with the Bourne shell. The only significant feature of the latter that the Korn shell doesn't support is ^ (caret) as a synonym for the pipe (|) character. This is an archaic feature that the Bourne shell includes for its own backward compatibility with earlier shells. No modern Unix version has any shell code that uses ^ as a pipe.

To describe the differences between the Bourne shell and the Korn shell, we'll go through each chapter of this book and enumerate the features discussed in the chapter that the Bourne shell does not support.

Perhaps the most obvious shell with which to compare ksh93 is ksh88, the 1988 version of the Korn shell. This section briefly describes those features of ksh93 that are either different or nonexistent in ksh88. As with the presentation for the Bourne shell, the topics are covered in the same order as they're presented in the rest of the book.

There have been many attempts to standardize Unix. Hardware companies' monolithic attempts at market domination, fragile industry coalitions, marketing failures, and other such efforts are the stuff of history — and the stuff of frustration.

Only one standardization effort has not been tied to commercial interests: the Portable Operating System Interface, known as POSIX. This effort started in 1981 with the /usr/group (now UniForum) Standards Committee, which produced the /usr/group Standard three years later. The list of contributors grew. Eventually, the effort to create a formal standard moved under the umbrella of the Institute of Electrical and Electronic Engineers (IEEE) and the International Organization for Standardization (ISO).

The first POSIX standard was published in 1988 and revised in 1996. This one, called IEEE Standard 1003.1, covered low-level issues at the system call level. IEEE Standard 1003.2, covering the shell, utility programs, and user interface issues, was ratified in September 1992 after a six-year effort. In September 2001, a joint revision of both standards was approved. The new standard, covering all the material in the two earlier separate documents, is known as IEEE Standard 1003.1-2001.

The POSIX standards were never meant to be rigid and absolute. The committee members certainly weren't about to put guns to the heads of operating system implementers and force them to adhere. Instead, the standards are designed to be flexible enough to allow for both coexistence of similar available software, so that existing code isn't in danger of obsolescence, and the addition of new features, so that vendors have the incentive to innovate. In other words, they are supposed to be the kind of third-party standards that vendors might actually be interested in following.

As a result, most Unix vendors currently comply with both standards. The Korn shell is no exception; it is intended to be 100% POSIX-compliant. It pays to be familiar with what's in the standard if you want to write code that is portable to different systems.

The Desk Top Korn Shell (dtksh) is a standard part of the Common Desktop Environment (CDE), available on commercial Unix systems such as Solaris, HP-UX, and AIX. It is based on a somewhat older version of ksh93. It evolved from the earlier program wksh, the Windowing Korn shell, released by Unix System Laboratories in late 1992. It's a full Korn shell, compatible with the version that this book describes, that has extensions for graphical user interface (GUI) programming in the X Window System environment. It is typically found in /usr/dt/bin/dtksh.

dtksh supports the OSF/Motif graphical Toolkit by making its routines available as built-in commands. This allows programmers to combine the Korn shell's strength as a Unix systems programming environment with the power and abstraction of the Toolkit. The result is a unified environment for quick and easy development of graphics-based software.

There are various GUI development tools that allow you to construct user interfaces with a graphics-based editor rather than with programming language code. But such tools are typically huge, expensive, and complex. dtksh, on the other hand, is inexpensive and unbeatable for the smoothness of its integration with Unix — it's the only such tool that you can use as your login shell! (Well, almost; see the next section.) It is a definite option for systems programmers who use X-based workstations and need a rapid prototyping tool.

To give you the flavor of dtksh code, here is a script that implements the canonical "Hello World" program by displaying a small box with an "OK" button. It is from the article Graphical Desktop Korn Shell, in the July, 1998 issue of Linux Journal, by George Kraft IV. (See http://www.linuxjournal.com/article.php?sid=2643.) This code should hold no surprises for X and Motif programmers:

#!/usr/dt/bin/dtksh



XtInitialize TOPLEVEL dtHello DtHello "$@"



XmCreateMessageDialog HELLO $TOPLEVEL hello \

            dialogTitle:"DtHello" \

            messageString:"$(print "Hello\nWorld")"

XmMessageBoxGetChild HELP $HELLO \

        DIALOG_HELP_BUTTON

XtUnmanageChild $HELP

XmMessageBoxGetChild CANCEL $HELLO \

        DIALOG_CANCEL_BUTTON

XtUnmanageChild $CANCEL

XtAddCallback $HELLO okCallback exit

XtManageChild $HELLO

XtMainLoop

Back in 1996, while a computer science graduate student at Princeton University, Dr. Jeffrey L. Korn wrote tksh. This is an integration of ksh93 with Tcl/Tk. The following quote (from Dr. Korn's research web page) summarizes it well:

The tksh home page is still at Princeton: http://www.cs.princeton.edu/~jlk/tksh/. It has links to papers and documentation that can be downloaded and printed. However, the link to tksh executables is out-of-date. The tksh source is available from AT&T Research as part of the ast-open package, which also contains ksh93 and reimplementations of many other Unix tools. See Appendix C for more information.

The following example script, from the USENIX paper on tksh, is called watchdir:

# Tksh Demo

# Jeff Korn

#

# This script keeps track of visited directories and shows the files

# in the current directory.   You can double-click on files and 

# directories.  The script should be used interactively, so to run:

#   $ tksh

#   $ . scripts/watchdir



function winsetup

{

    pack $(frame .f)

    frame .f.dirname -relief raised -bd 1

    pack .f.dirname -side top -fill x

    pack $(frame .f.ls) $(frame .f.dirs) -side left 

    label .f.dirname.label -text "Current directory: "

    label .f.dirname.pwd -textvariable PWD

    pack .f.dirname.label .f.dirname.pwd -side left



    scrollbar .f.ls.scroll -command ".f.ls.list yview"

    listbox .f.ls.list -yscroll ".f.ls.scroll set" -width 20 -setgrid 1

    pack $(label .f.ls.label -text "Directory Contents") -side top

    pack .f.ls.list .f.ls.scroll -side left -fill y -expand 1



    scrollbar .f.dirs.scroll -command ".f.dirs.list yview"

    listbox .f.dirs.list -yscroll ".f.dirs.scroll set" -width 20 -setgrid 1

    pack $(label .f.dirs.label -text "Visited Directories") -side top

    pack .f.dirs.list .f.dirs.scroll -side left -fill y -expand 1

    bind .f.dirs.list "<Double-1>" 'cd $(selection get)'

    bind .f.ls.list "<Double-1>" 'tkfileselect $(selection get)'

}



function tkfileselect

{

    [[ -d "$1" ]] && tkcd "$1"

    [[ -f "$1" ]] && ${EDITOR-${VISUAL-emacs}} "$1"

}



function tkcd

{

    cd $1 > /dev/null || return

    .f.ls.list delete 0 end

    set -o markdirs

    .f.ls.list insert end .. *

    [[ ${VisitedDir["$PWD"]} == "" ]] && .f.dirs.list insert end "$PWD"

    VisitedDir["$PWD"]=1

}



typeset -A VisitedDir

winsetup > /dev/null

alias cd=tkcd

tkcd .

Many of the Open Source Unix-like systems, such as GNU/Linux, come with the Public Domain Korn Shell, pdksh. pdksh is available as source code; start at its home page: http://www.cs.mun.ca/~michael/pdksh/. It comes with instructions for building and installing on various Unix platforms.

pdksh was originally written by Eric Gisin, who based it on Charles Forsyth's public-domain clone of the Version 7 Bourne shell. It is mostly compatible with the 1988 Korn shell and POSIX, with some extensions of its own.

Its emacs editing mode is actually more powerful than that of the 1988 Korn shell. Like the full Emacs editor, you can customize the keystrokes that invoke editing commands (known as key bindings in Emacs terminology). Editing commands have full names that you can associate with keystrokes by using the bind command.

For example, if you want to set up CTRL-U to do the same thing as CTRL-P (i.e., go back to the previous command in the history file), you could put this command in your .profile:

bind '^U'=up-history

You can even set up two-character escape sequences, which (for example) allow you to use ANSI arrow keys as well as control characters, and you can define macros, i.e., abbreviations for sequences of editing commands.

The public domain Korn shell's additional features include alternation wildcards (borrowed from the C shell) and user-definable tilde notation, in which you can set up ~ as an abbreviation for anything, not just user names. There are also a few subtle differences in integer expression evaluation and aliasing.

pdksh lacks the following features of the official version:

bash is probably the most popular "third-party" shell that is available for Unix and other systems. In particular, it is the default shell on GNU/Linux systems. (It also comes on the "freeware" CD with Solaris 8.) You can get it from the Internet, via anonymous FTP to ftp.gnu.org in the directory /pub/gnu/bash. You can also order it from its maker at the address listed here.

The Free Software Foundation

59 Temple Place - Suite 330

Boston, MA  02111-1307

(617) 542-2652

(617) 542-5942 (fax)

gnu@gnu.org

            http://www.gnu.org

         

Bash was written by Brian Fox and Chet Ramey. Chet Ramey currently maintains it. Its name is in line with the FSF's penchant for bad puns: it stands for Bourne-Again Shell. Although bash is easily available and you don't have to pay for it (other than the cost of media, phone calls, etc.), it's not really public domain software. While public domain software doesn't have licensing restrictions, the FSF's software does. But those restrictions are diametrically opposed to those in a commercial license: instead of agreeing not to distribute the software further, you agree not to prevent it from being distributed further! In other words, you enjoy unrestricted use of the software as long as you agree not to inhibit others from doing the same. Richard Stallman, the founder of the FSF, invented this intriguing and admirable concept.

These days, the ideals of the Free Software and Open Source movements, the GNU project, and the quality of GNU software are all well known. The most popular GNU system is GNU/Linux, which uses the Linux kernel and GNU utilities to make a complete, fully functional, Unix- and POSIX-compatible computing environment.

bash is fully compatible with the 1992 POSIX standard. It has several of the most important Korn shell features and the C shell features that the Korn shell has appropriated, including aliases, functions, tilde notation, emacs and vi editing modes, arithmetic expressions, job control, etc.

zsh is a powerful interactive shell and scripting language with many features found in ksh, bash, and tcsh, as well as several unique features. zsh has most of the features of ksh88 but few of ksh93. It is freely available and should compile and run on just about any modern version of Unix. Ports for other operating systems are also available. The zsh home page is http://www.zsh.org. The current version is 4.0.2.

In this section we cover:

grep foo **/*.c

print **/core

print *(.)

print *(/)

print **/core(.)

print /tmp/**/*(U)

print /tmp/**/*(U=)

The proliferation of the Korn shell has not stopped at the boundaries of Unix-dom. Many programmers who got their initial experience on Unix systems and subsequently crossed over into the PC world wished for a nice Unix-like environment (especially when faced with the horrors of the MS-DOS command line!), so it's not surprising that several Unix shell-style interfaces to small-computer operating systems have appeared, Korn shell emulations among them.

In the past several years, not just shell clones have appeared, but entire Unix "environments." Two of them use shells that we've already discussed. Two others provide their own shell reimplementations. Providing lists of major and minor differences is counterproductive. Instead, this section describes each environment in turn (in alphabetical order), along with contact and Internet download information.

Here is a list of the options you can use when invoking the Korn shell. In addition to these, any set option can be used on the command line; see the table on options later in this appendix. Login shells are usually invoked with the options -i (interactive), -s (read from standard input), and -m (enable job control).

Here is a summary of all built-in commands and keywords.

A number of aliases are predefined, i.e. automatically built-in to ksh at compile time. They are listed in the table below. Note that some of them are defined with a trailing space character. This enables alias expansion on the word following the alias on the command line.

Here is a summary of all built-in shell variables:

These are the operators that are used with the [[...]] construct. They can be logically combined with && ("and") and || ("or") and grouped with parenthesis. When used with filenames of the form /dev/fd/ N, they test the corresponding attribute of open file descriptor N.

These are options that can be turned on with the set -o command. All are initially off except where noted. Abbreviations, where listed, are options to set that can be used instead of the full set -o command (e.g., set -a is an abbreviation for set -o allexport). For the most part, the abbreviations are actually backward-compatible Bourne shell options. To disable an option, use set +o long-name or set +X, where long-name and X are the long form or the single character form of the option, respectively.

These are the options to the typeset command. Use + option to turn an option off, e.g., typeset +x foo to stop exporting the variable foo.

Starting with ksh93m, the built-in arithmetic facility understands a large percentage of the C language's expressions. This makes the shell more attractive as a full-blown programming language. The following features are available:

Here is a complete list of all emacs editing mode commands. Some of these, such as "ESC [ A," represent ANSI standard terminal arrow key sequences; they were added for ksh93h.

Here is a complete list of all vi-mode control commands. As for the emacs mode commands, the sequences such as "[ A" are for ANSI standard terminal arrow keys and were added for ksh93h.

The getopts command is extremely capable. With it, you can make your shell scripts accept long options, specify that arguments are optional or numeric, and provide descriptions of the arguments and values such that the -?, --man, --html and --nroff options work the same for your program as they do for the ksh93 built-in commands.

The price for this power is the complexity of the option description "language." Based on a description provided by Dr. Glenn Fowler of AT&T Research, we describe how the facilities evolved, how they work, and summarize how to use them in your own programs. We use the the extended getopts command in the solution to Task B-1.

The first step is to describe the options. This is done with a comment at the top of the script:

# usage: phaser4 [ options ] files

#   -k, --kill                  use kill setting (default)

#   -l n, --level n             set phaser level (default = 2)

#   -s, --stun                  use stun-only setting

#   -t [lf], --tricorder [lf]   tricorder mode, opt. scan for life form lf

Now the fun begins. This outline of capabilities follows the order in which features were added to getopts.

The starting place for all things related to the Korn shell is http://www.kornshell.com, maintained by David Korn, with links grouped by the following topics:

http://www.research.att.com/sw/download is the starting point for actually downloading the ksh software. The software is covered by an Open Source-style license. The current version of the license is at http://www.research.att.com/sw/license/ast-open.html. This license is reproduced in Appendix D. You should read and understand it first; if its terms aren't acceptable to you, you should not download the software source code or binaries from the AT&T web site.

The software on the AT&T web site is available in different "packages," most of which have names prefixed with "ast," which stands for "Advanced Software Tools." The source packages come as gziped tar files, using the .tgz file name suffix. Choose one or more of the following packages to download:

Building any of the packages from source code is pretty straightforward. The full details, with a FAQ and notes, are given on the AT&T web site. Here is a walk-through of the steps. We show the steps for the ast-open package, but they're identical for the other source code packages.